Warmish Welcome

Fantasy sports are a passion of mine because they reside at the intersection of two of my favorite things in the world: sports and data. Over the last 4 or so years, I have built data sets suitable for modeling fantasy football and basketball outcomes. I tend to favor daily fantasy, but weather data is valuable for season long fantasy football as well.

In fantasy football, weather data are valuable for a variety of reasons. Conventional wisdom states temperature, wind, and precipation will affect throwing the football, the kicking game, holding on to the football, etc. After reading through this post, you will have the necessary data to explore these claims.

# package management
library(tidyverse)
library(rvest)
library(janitor)
library(glue)

Exploring NFLWeather.com

Once you navigate to NFL Weather, you can see the weeks indexed with small blue boxes towards the top, directly in the middle of the page. Clicking on the ‘1’ takes you to the weather for week 1 of the 2020 season. Notice the url for this page: http://nflweather.com/en/week/2020/week-1/. Simply replace ‘2020’ in the url with ‘2019’ will direct you to the week 1 forecast for the 2019 season. As you might have guessed, swapping ‘week-1’ with ‘week-8’ will launch the forecast for week 8. Already, we can see that the forecast for each week of a given season can be accessed by switching the year and week parts of the url.

Let’s start with accessing the weather data for week 1 of the 2020 season. Using rvest, we can quickly scan the html for tables and parse appropriately using the read_html() and html_table() functions. These commands parse the html from the url given and return the tables in a list. Using glimpse(), we see that a list of 1 data frame that has some blank rows and null columns is returned.

url <- 'http://nflweather.com/en/week/2020/week-1/'

url %>% 
  read_html() %>% 
  html_table() %>%
  glimpse()

## List of 1
##  $ :'data.frame':    16 obs. of  13 variables:
##   ..$                  : logi [1:16] NA NA NA NA NA NA ...
##   ..$ Away             : chr [1:16] "Texans" "Eagles" "Dolphins" "Packers" ...
##   ..$ Game             : logi [1:16] NA NA NA NA NA NA ...
##   ..$ Game             : chr [1:16] "@" "@" "@" "@" ...
##   ..$ Game             : logi [1:16] NA NA NA NA NA NA ...
##   ..$ Home             : chr [1:16] "Chiefs" "Washington" "Patriots" "Vikings" ...
##   ..$ Time (ET)        : chr [1:16] "Final: 20 - 34" "Final: 17 - 27" "Final: 11 - 21" "Final: 43 - 34" ...
##   ..$ TV               : chr [1:16] "NBC" "FOX" "CBS" "FOX" ...
##   ..$                  : logi [1:16] NA NA NA NA NA NA ...
##   ..$ Forecast         : chr [1:16] "58f Overcast" "76f Partly Cloudy" "73f Clear" "DOME" ...
##   ..$ Extended Forecast: chr [1:16] "Overcast. Rain in the morning and afternoon." "Partly Cloudy. Clear throughout the day." "Clear. Partly cloudy throughout the day." "Clear. Clear throughout the day." ...
##   ..$ Wind             : chr [1:16] "6m NNE" "4m S" "6m S" "2m SW" ...
##   ..$                  : chr [1:16] "Details" "Details" "Details" "Details" ...

url %>% 
  read_html() %>% 
  html_table() %>%
  .[[1]] %>%  # select the first and only element in the list 
  clean_names() %>%
  kable(format = 'html') %>% 
  kable_styling()

url %>% 
  read_html() %>% 
  html_table() %>%
  .[[1]] %>% 
  clean_names() %>%
  select(away, home, forecast, wind) %>% 
  kable(format = 'html') %>% 
  kable_styling()

url %>% 
  read_html() %>% 
  html_table() %>%
  .[[1]] %>% 
  clean_names() %>%
  select(away, home, forecast, wind) %>%
  mutate(season = str_sub(url, start= 31, end = 34),
         week = str_sub(gsub('.*\\-', '', url), start = -3, end = -2)) %>% 
  kable(format = 'html') %>% 
  kable_styling()

url %>% 
  read_html() %>% 
  html_table() %>%
  .[[1]] %>% 
  clean_names() %>%
  select(away, home, forecast, wind) %>%
  mutate(season = str_sub(url, start= 31, end = 34),
         week = str_sub(gsub('.*\\-', '', url), start = -3, end = -2),
         wind = as.numeric(gsub( "m.*$", "",wind)),
         temperature = ifelse(forecast == 'DOME', 71, gsub( "f.*$", "", forecast)),
         weather = gsub(".*? ", "", forecast)) %>%
  kable(format = 'html') %>% 
  kable_styling()

url %>% 
  read_html() %>% 
  html_table() %>%
  .[[1]] %>% 
  clean_names() %>%
  select(away, home, forecast, wind) %>%
  mutate(season = str_sub(url, start= 31, end = 34),
         week = str_sub(gsub('.*\\-', '', url), start = -3, end = -2),
         wind = as.numeric(gsub( "m.*$", "",wind)),
         temperature = ifelse(forecast == 'DOME', 71, gsub( "f.*$", "", forecast)),
         weather = gsub(".*? ", "", forecast)) %>%
  pivot_longer(cols = c('away', 'home'), values_to = 'team') %>% 
  select(-name, -forecast) %>% 
  select(team, season, week, temperature, wind, weather) %>% 
  kable(format = 'html') %>% 
  kable_styling()

Mapping the Function

The final objective here is to use this code as a function. Functions can be used over and over without rewriting code. In this example, we can use and reuse our code for existing urls, grabbing data from a particular week of a particular season - or we can run for all weeks from multiple seasons. For this blog post, we’re going to grab weather data for weeks 1 through 17 from 2018 and 2019.

We actually need two functions. One for generating the web page urls and another for scraping / wrangling the data. For generating the urls, we establish the sequence of weeks and years. Then, with the help of tidyr and the crossing() function, we can generate all combinations of weeks and years. The result is a tibble with two columns containing a row for weeks 1-17 for each season 2018 and 2019. The function that generates the urls simply concatenates the base url with the year and week provided. Using pmap, we can iterate over each row of the weather_weeks_and_years tibble, passing the year and week from each row into our generate_urls function. The result is a list of 85 urls or web pages that we can now scrape data from.

# set up weeks and years
weeks <- c(1:17)
years <- c(2018:2019)
weather_weeks_and_years <- crossing(year = years, week = weeks) # generate all unique combinations

# function to generate urls
generate_urls <- function(year, week) {
  full_url <- glue::glue("http://nflweather.com/en/week/{year}/week-{week}/")
}

# pass weeks and years through function
url_list <- pmap(weather_weeks_and_years, generate_urls)

head(url_list)

## [[1]]
## http://nflweather.com/en/week/2018/week-1/
## 
## [[2]]
## http://nflweather.com/en/week/2018/week-2/
## 
## [[3]]
## http://nflweather.com/en/week/2018/week-3/
## 
## [[4]]
## http://nflweather.com/en/week/2018/week-4/
## 
## [[5]]
## http://nflweather.com/en/week/2018/week-5/
## 
## [[6]]
## http://nflweather.com/en/week/2018/week-6/

Below, we create a function that accepts a url, scrapes, wrangles, and returns the data. The code is the same code written above, used repeatedly for each url. We execute the function for each url within url_list by using map_df from the purrr package. map_df accepts an input list and applies a function to each element while returning a data frame. It might take a minute or two to run. Upon completion, you should have a solid weather dataset to start digging into.

# function to scrape weather
scrape_weather_data <- function(webpage_url) {
  webpage_url %>% 
    read_html() %>% 
    html_table() %>%
    .[[1]] %>% 
    clean_names() %>%
    select(away, home, forecast, wind) %>%
    mutate(season = str_sub(webpage_url, start= 31, end = 34),
           week = str_sub(gsub('.*\\-', '', webpage_url), start = -3, end = -2),
           wind = as.numeric(gsub( "m.*$", "",wind)),
           temperature = ifelse(forecast == 'DOME', 71, gsub( "f.*$", "", forecast)),
           weather = gsub(".*? ", "", forecast)) %>%
    pivot_longer(cols = c('away', 'home'), values_to = 'team') %>% 
    select(-name, -forecast) %>% 
    select(team, season, week, temperature, wind, weather)
}

# pass urls through the function
weather_data <- map_df(url_list, scrape_weather_data)

weather_data %>% 
  sample_n(10) %>% 
  kable(format = 'html') %>% 
  kable_styling()

Finally

That was a lot. From regular expressions, to creating functions, and using purrr - there were quite a few different tasks described in this post. In the wild, most data are messy and need to be gathered / cleaned using a wide variety of tricks and trades. Good bye.