Objectives

Objectives

Learn how to effecitvely use visualization for

• exploring and understanding data
• communicating and explaining insights

Objectives

Learn how to effecitvely use visualization for

• exploring and understanding data
• communicating and explaining insights

Learn how to use data technologies for

• acquiring data
• cleaning data
• organizing data

Topics

Topics

Data visualization

• some history of visualization
• learning the basic graph types
• how to create basic graphs in R
• human perception, and how it affects visualization
• using understanding of perception to guide evaluation and design
• dynamic and interactive visualizations

Topics

Data visualization

• some history of visualization
• learning the basic graph types
• how to create basic graphs in R
• human perception, and how it affects visualization
• using understanding of perception to guide evaluation and design
• dynamic and interactive visualizations

Data technologies

• basic data types
• reshaping and transforming data
• aggregating and summarizing data
• merging several data sets
• regular expressions for cleaning data
• harvesting data from the web

Recommended Text Books

Recommended Text Books

Kieran Healy (2018) Data Visualization: A practical introduction, Princeton

Recommended Text Books

Kieran Healy (2018) Data Visualization: A practical introduction, Princeton

Paul Murrell (2009). Introduction to Data Technologies, Chapman & Hall/CRC.

Recommended Text Books

Kieran Healy (2018) Data Visualization: A practical introduction, Princeton

Paul Murrell (2009). Introduction to Data Technologies, Chapman & Hall/CRC.

Hadley Wickham, Mine Çetinkaya-Rundel, and Garrett Grolemund (2023), R for Data Science (2nd Edition), O'Reilly.

Prerequisites

Prerequisites

An introductory statistics course.

Prerequisites

An introductory statistics course.

A regression course.

Assessment

Assessment

Quizzes

• Short quizzes will be posted on ICON after most lectures.

Assessment

Quizzes

• Short quizzes will be posted on ICON after most lectures.

Homework

• Homework assignments will be due approximately once a week.
• You will typically submit your work by pushing it to your GitLab repository by 5:00 PM on the due date.
• Your homework solutions should be written as reports, using proper sentences and paragraphs to present your results.

Assessment

Quizzes

• Short quizzes will be posted on ICON after most lectures.

Homework

• Homework assignments will be due approximately once a week.
• You will typically submit your work by pushing it to your GitLab repository by 5:00 PM on the due date.
• Your homework solutions should be written as reports, using proper sentences and paragraphs to present your results.

Project

• You will do a project developing a visual analysis of a data set of your choosing.
• You can work on your own or in a group of up to three students.
• Your project should represent about 10 hours of work for each student.
• A one page proposal for your project is due on Monday, March 18.
• A final report on your project is due on Friday, May 3.
• Your project may be shared with the class through the class web page.

Tools

Tools

We will be using

• R for computing and graphics
• R Markdown for creating reproducible reports.
• git and the UI GitLab service for revision control and submitting work.

Tools

We will be using

• R for computing and graphics
• R Markdown for creating reproducible reports.
• git and the UI GitLab service for revision control and submitting work.

You will need an editor or IDE; you can use

• RStudio for editing and more
• any other editor or IDE

Tools

We will be using

• R for computing and graphics
• R Markdown for creating reproducible reports.
• git and the UI GitLab service for revision control and submitting work.

You will need an editor or IDE; you can use

• RStudio for editing and more
• any other editor or IDE

To access these tools you can

• use the UI IDAS RStudio Notebook Server,
• use the CLAS Linux systems via the FastX remote desktop,
• or install your own on your computer

First Steps: Do This Today!

First Steps: Do This Today!

Visit the UI GitLab site at https://research-git.uiowa.edu and log in with your HawkID.

First Steps: Do This Today!

Visit the UI GitLab site at https://research-git.uiowa.edu and log in with your HawkID.

Make sure you can access the UI IDAS Rstudio Notebook Server with your HawkID and password.

• The server is available at

  https://notebooks.hpc.uiowa.edu/spring2024-stat-4580-0001/hub/home.

• If you cannot log into the RStudio server, please let your TA or me know immediately.

First Steps: Do This Today!

Visit the UI GitLab site at https://research-git.uiowa.edu and log in with your HawkID.

Make sure you can access the UI IDAS Rstudio Notebook Server with your HawkID and password.

• The server is available at

  https://notebooks.hpc.uiowa.edu/spring2024-stat-4580-0001/hub/home.

• If you cannot log into the RStudio server, please let your TA or me know immediately.

Make sure you are able to log into the CLAS Linux systems with your HawkID and password.

• The easiest way is to use the FastX client at https://fastx.divms.uiowa.edu.
• If you cannot log into the CLAS workstations, please let your TA or me know immediately.

First Steps: Do This Today!

Make sure you have access to R and try someting like this:

with(faithful,
     plot(eruptions, waiting,
          xlab = "Eruption time (min)",
          ylab = "Waiting time to next eruption (min)"))Copy Code

Getting Set Up

Log into the UI GitLab site at https://research-git.uiowa.edu to get your GitLab account activated.

Getting Set Up

Log into the UI GitLab site at https://research-git.uiowa.edu to get your GitLab account activated.

Decide where you want to work:

• UI IDAS RStudio Notebook Server
• FastX for accessing the CLAS Linux systems via the web interface or the desktop client.
• Your own computer.

Getting Set Up

Log into the UI GitLab site at https://research-git.uiowa.edu to get your GitLab account activated.

Decide where you want to work:

• UI IDAS RStudio Notebook Server
• FastX for accessing the CLAS Linux systems via the web interface or the desktop client.
• Your own computer.

Setup needed for IDAS RStudio Server:

• If you are registered then you should have an account now. If you add the course late you should have an account within a day.
• Introduce yourself to Git.

Getting Set Up

Getting Set Up

Setting up your own computer: (A good resource for help with this is https://happygitwithr.com):

Getting Set Up

Setting up your own computer: (A good resource for help with this is https://happygitwithr.com):

• Install the current version of R.
  • You might have older versions from other courses (e.g. from Anaconda).
  • You will need to add packages as we go along.

Getting Set Up

Setting up your own computer: (A good resource for help with this is https://happygitwithr.com):

• Install the current version of R.
  • You might have older versions from other courses (e.g. from Anaconda).
  • You will need to add packages as we go along.

• Install RStudio if you want to use it (highly recommended).

Getting Set Up

Setting up your own computer: (A good resource for help with this is https://happygitwithr.com):

• Install the current version of R.
  • You might have older versions from other courses (e.g. from Anaconda).
  • You will need to add packages as we go along.

• Install RStudio if you want to use it (highly recommended).

• Install Git.

Getting Set Up

Setting up your own computer: (A good resource for help with this is https://happygitwithr.com):

• Install the current version of R.
  • You might have older versions from other courses (e.g. from Anaconda).
  • You will need to add packages as we go along.

• Install RStudio if you want to use it (highly recommended).

• Install Git.

• Introduce yourself to Git.

Life Expectancy in the Americas in 2007

We will look at:

Life Expectancy in the Americas in 2007

We will look at:

• How to create these views using code that makes them easily reproducible.

Life Expectancy in the Americas in 2007

We will look at:

• How to create these views using code that makes them easily reproducible.

• How to assess their advantages and disadvantages as visual representations of the data

Life Expectancy in the Americas in 2007

We will look at:

• How to create these views using code that makes them easily reproducible.

• How to assess their advantages and disadvantages as visual representations of the data

A data set with more variables for more countries and years is available in the gapminder R package.

Life Expectancy in the Americas in 2007

We will look at:

• How to create these views using code that makes them easily reproducible.

• How to assess their advantages and disadvantages as visual representations of the data

A data set with more variables for more countries and years is available in the gapminder R package.

Data preparation steps:

Life Expectancy in the Americas in 2007

We will look at:

• How to create these views using code that makes them easily reproducible.

• How to assess their advantages and disadvantages as visual representations of the data

A data set with more variables for more countries and years is available in the gapminder R package.

Data preparation steps:

• Filter the larger data set down to the countries and year we want.

Life Expectancy in the Americas in 2007

We will look at:

• How to create these views using code that makes them easily reproducible.

• How to assess their advantages and disadvantages as visual representations of the data

A data set with more variables for more countries and years is available in the gapminder R package.

Data preparation steps:

• Filter the larger data set down to the countries and year we want.

• Select the country name and life expectancy variables.

Yearly Snowfall in Iowa City

Data preparation steps:

Yearly Snowfall in Iowa City

Data preparation steps:

• Read in the CSV file.

Yearly Snowfall in Iowa City

Data preparation steps:

• Read in the CSV file.

• Reshape the data to have columns date, TMAX, TMIN, SNOW and PRCP.

Yearly Snowfall in Iowa City

Data preparation steps:

• Read in the CSV file.

• Reshape the data to have columns date, TMAX, TMIN, SNOW and PRCP.

• Filter out bogus dates created by the original format.

Yearly Snowfall in Iowa City

Data preparation steps:

• Read in the CSV file.

• Reshape the data to have columns date, TMAX, TMIN, SNOW and PRCP.

• Filter out bogus dates created by the original format.

• Convert units to more standard (American) ones (e.g. milimeters to inches).

• ...

Internet Adoption Across the World

Data preparation:

Internet Adoption Across the World

Data preparation:

• Read in the data.

Internet Adoption Across the World

Data preparation:

• Read in the data.

• Filter down to the countries we want.

Internet Adoption Across the World

Data preparation:

• Read in the data.

• Filter down to the countries we want.

• Reshape to have columns country, year, and users.

• ...

Iowa Wind Turbines

There are two data sets:

Iowa Wind Turbines

There are two data sets:

• Shape information for drawing the map.

Iowa Wind Turbines

There are two data sets:

• Shape information for drawing the map.

• Data on individual wind turbines.

Iowa Wind Turbines

There are two data sets:

• Shape information for drawing the map.

• Data on individual wind turbines.

Data preparation:

Iowa Wind Turbines

There are two data sets:

• Shape information for drawing the map.

• Data on individual wind turbines.

Data preparation:

• Read in the data.

Iowa Population in 2010

Again there are two data sets:

Iowa Population in 2010

Again there are two data sets:

• Shape data for drawing the map.

Iowa Population in 2010

Again there are two data sets:

• Shape data for drawing the map.

• County population data from the 2010 census.

Iowa Population in 2010

Again there are two data sets:

• Shape data for drawing the map.

• County population data from the 2010 census.

Data preparation:

Reproducible Reports and Analyses

Preparing a report on a data analysis project usually involves

Reproducible Reports and Analyses

Preparing a report on a data analysis project usually involves

• reading the data

Reproducible Reports and Analyses

Preparing a report on a data analysis project usually involves

• reading the data

• wrangling the data into usable form

Reproducible Reports and Analyses

Preparing a report on a data analysis project usually involves

• reading the data

• wrangling the data into usable form

• visualizing, summarizing, and modeling

Reproducible Reports and Analyses

Preparing a report on a data analysis project usually involves

• reading the data

• wrangling the data into usable form

• visualizing, summarizing, and modeling

• writing a report that includes your results

Reproducible Reports and Analyses

Preparing a report on a data analysis project usually involves

• reading the data

• wrangling the data into usable form

• visualizing, summarizing, and modeling

• writing a report that includes your results

To make your work reproducible for someone else, or for you when the data changes, it is best to use code for the entire workflow.

Tools for Reproducibility

R Markdown files contain report text along with code to produce numerical and graphical results.

Tools for Reproducibility

R Markdown files contain report text along with code to produce numerical and graphical results.

Tools are available to

Tools for Reproducibility

R Markdown files contain report text along with code to produce numerical and graphical results.

Tools are available to

• convert an R Markdown file into a PDF or HTML report;

Tools for Reproducibility

R Markdown files contain report text along with code to produce numerical and graphical results.

Tools are available to

• convert an R Markdown file into a PDF or HTML report;

• extract the code used to produce the computational and graphical results.

Tools for Reproducibility

R Markdown files contain report text along with code to produce numerical and graphical results.

Tools are available to

• convert an R Markdown file into a PDF or HTML report;

• extract the code used to produce the computational and graphical results.

These slides were generated from the R Markdown file intro.Rmd.

Tools for Reproducibility

R Markdown files contain report text along with code to produce numerical and graphical results.

Tools are available to

• convert an R Markdown file into a PDF or HTML report;

• extract the code used to produce the computational and graphical results.

These slides were generated from the R Markdown file intro.Rmd.

You will be creating R Markdown files like this for your homework and project.

Tools for Reproducibility

R Markdown files contain report text along with code to produce numerical and graphical results.

Tools are available to

• convert an R Markdown file into a PDF or HTML report;

• extract the code used to produce the computational and graphical results.

These slides were generated from the R Markdown file intro.Rmd.

You will be creating R Markdown files like this for your homework and project.

Some R Markdown tutorials:

Tools for Reproducibility

R Markdown files contain report text along with code to produce numerical and graphical results.

Tools are available to

• convert an R Markdown file into a PDF or HTML report;

• extract the code used to produce the computational and graphical results.

These slides were generated from the R Markdown file intro.Rmd.

You will be creating R Markdown files like this for your homework and project.

Some R Markdown tutorials:

• R Markdown: The Definitive Guide by Yihui Xie is a book-length presentation.

Code for the Iowa City Snowfall Example

Read the data:

library(tidyverse)
library(lubridate)
if (! file.exists(here("ic_noaa.csv.gz")))
    download.file("http://www.stat.uiowa.edu/~luke/data/ic_noaa.csv.gz",
                  here("ic_noaa.csv.gz"))
ic_data_raw <- as_tibble(read.csv(here("ic_noaa.csv.gz"), head = TRUE))Copy Code

Code for the Iowa City Snowfall Example

Read the data:

library(tidyverse)
library(lubridate)
if (! file.exists(here("ic_noaa.csv.gz")))
    download.file("http://www.stat.uiowa.edu/~luke/data/ic_noaa.csv.gz",
                  here("ic_noaa.csv.gz"))
ic_data_raw <- as_tibble(read.csv(here("ic_noaa.csv.gz"), head = TRUE))Copy Code

Reshape from (very) wide to (too) long:

ic_data <- select(ic_data_raw, year, month, element, starts_with("VALUE"))
ic_data <- pivot_longer(ic_data,
                        names_to = "day",
                        values_to = "value", c(VALUE1 : VALUE31))Copy Code

Code for the Iowa City Snowfall Example

Reshape from too long to tidy with one row per day, keeping only the primary variables:

corevars <- c("TMAX", "TMIN", "PRCP", "SNOW", "SNWD")
ic_data <- filter(ic_data, element %in% corevars)
ic_data <- pivot_wider(ic_data, names_from = "element", values_from = "value")Copy Code

Code for the Iowa City Snowfall Example

Reshape from too long to tidy with one row per day, keeping only the primary variables:

corevars <- c("TMAX", "TMIN", "PRCP", "SNOW", "SNWD")
ic_data <- filter(ic_data, element %in% corevars)
ic_data <- pivot_wider(ic_data, names_from = "element", values_from = "value")Copy Code

Add a date variable for plotting and to help get rid of bogus days:

ic_data <- mutate(ic_data, date = lubridate::make_date(year, month, day))
ic_data <- filter(ic_data, ! is.na(date))Copy Code

Code for the Iowa City Snowfall Example

Add a Month factor with abbreviated levels:

ic_data <- mutate(ic_data,
                  Month = lubridate::month(month, label = TRUE, abbr = TRUE))Copy Code

Code for the Iowa City Snowfall Example

Add a Month factor with abbreviated levels:

ic_data <- mutate(ic_data,
                  Month = lubridate::month(month, label = TRUE, abbr = TRUE))Copy Code

Associate January through June with the winter starting in the previous year:

ic_data <- mutate(ic_data, wyear = ifelse(month <= 6, year - 1, year))Copy Code

Code for the Internet Example

Reshape to longer format:

internet <- select(internet_raw, country = Country.Name, matches("X.+"))
internet <- pivot_longer(internet, -country,
                         names_to = "year", values_to = "users")
internet <- mutate(internet, year = as.integer(sub("X", "", year)))Copy Code

Code for the Internet Example

Get ordering by 2017 levels:

intr17 <- filter(internet_short, year == 2017)
levs <- arrange(intr17, users)$countryCopy Code

Code for the Internet Example

Adjust x and y scales:

p_inet <- p_inet +
    scale_x_continuous(expand = c(0, 0), name = NULL) +
    scale_y_discrete(name = NULL, position = "right")Copy Code