AE08 - Web Scraping

Helping you become an informed voter

Purpose

Who pays for California’s ballot measures? Every proposition has a page on the California Secretary of State’s website listing the committees that support and oppose it, and how much money each one reported raising. In this AE you will decide whether you should scrape that site, then get one proposition’s tables into R and clean them into a tidy data frame. On Thursday you will turn that work into a function and use it on multiple props from that year.

By the end of Part 1 you should be able to:

• Use a checklist and decide, with reasons, whether a site can be scraped
• Read an HTML page into R and pull out its tables with read_html() and html_table()
• Clean a messy scraped table into a tidy data frame
• Check scraped numbers against the totals printed on the page

By the end of Part 2 you should be able to:

• Write a function with an argument that scrapes one page
• Use map() and list_rbind() to apply it to many pages, with a pause between requests
• Diagnose a page that breaks the loop, and check the combined result

WarningA lot of new functions today

We are throwing a lot of new functions at you today. Make a point of understanding every line you run. Add comments to the code that explain each line in words that make sense to you. Use help files, function vignettes, or an LLM to help you learn what each line is doing.

Instructions

• Work through the parts in order, and compare your numbers with a neighbor.
• Starter code is provided. Wherever you see _______, fill in the blank.
• Commit Part 1 by EOD Tuesday, and then a separate commit for part 2 by EOD Thursday.

NoteA note on the videos

The Data Science in a Box videos were made a few years ago and use older function names. The old versions still run, so you will see them in the videos and in examples online. In this class we use the current versions:

• html_node() in the videos has been superseded by html_element()
• html_nodes() in the videos has been superseded by html_elements()
• map_dfr() in the videos has been superseded by map() followed by list_rbind() in this class

The videos and slides mostly pull things out of a page with CSS selectors, and html_table() is only named in a list of functions. Today’s pages are plain tables, so we use html_table(), which needs no selectors.

The data

The data come from the California Secretary of State: 2024 Ballot Measure Contribution Totals. That index page links to one page for each proposition. Today you will work with one of them, Proposition 33 (rent control).

Each proposition page has up to two tables: committees in support, and committees in opposition. Each row is one committee, with its ID, its name, and the total contributions it reported.

Setup

library(tidyverse)
library(rvest)

Part 1: One page

1. Before you scrape

Run the checklist on the SOS site, then make a decision.

1. robots.txt: Does the site ask robots to stay out of the pages you want?
2. Terms of service: Do the site’s rules prohibit scraping, say nothing, or allow reuse?
3. What data: Is it about people? Is it sensitive? Is it a public record?
4. Load: How many requests will you make, and how will you pace them?
5. Official alternative: Is there a download or an API instead?

2. Read the page

read_html() downloads the page’s HTML into R. Each time you run it you send one request to the state’s server, so run it once and then work from the saved page object.

url <- 

(page <- _______(url))

3. Pull out the tables

• html_table() finds every table on the page and returns a list with one data frame per table.
• Use it on page and save the result as tables.
• How many tables did you get, and how does that compare to the page in your browser?

tables <- page |> _______()
length(tables)

If you open your browser’s developer tools you will see more than two table elements. The extra ones belong to the search box and are added by JavaScript after the page loads. rvest only sees the HTML the server sends, not what JavaScript adds later.

4. Clean the support table

Look at the first table, tables[[1]]. tables is a list, and double square brackets pull one data frame out of it: tables[[1]] is the first and tables[[2]] is the second.

_______

• What is wrong with this table as a data frame?
• Fix the problems and save the result as support.
  • Name the columns committee_id, committee_name and total
  • Drop the rows that correspond to headers and subheaders
  • Convert total to a number
  • Add a position variable with the value “support”.

support <- _______ |>
  rename(_______ = 1, _______ = 2, _______ = 3) |>
  slice(_______) |>
  mutate(total = _______(total),
         position = "_______")

support # confirm your modifications work

5. Clean the oppose table

Do the same thing for the oppose table.

6. Stack and check

Stack the two data frames into one with bind_rows() and save it as prop33.

(prop33 <- _______(support, _______))

Scraped numbers can be wrong without any error message, so check them against the page. Calculate the number of committees are on each side, and each side’s total.

prop33 |>
  group_by(_______) |>
  _______(committees = n(), total = _______)

Now look at the data itself.

1. Which side raised more, and by how much?
2. One oppose committee reports $0. Which one?
3. One support committee is named “Kevin De Leon Believing in a Better California Ballot Measure Committee - Yes on Propositions 3, 32 and 33”. What does that name tell you about its total, and what would go wrong if you added up money across propositions?

7. Compare with an LLM (if time permits)

WarningYour turn

Ask your LLM: “Write R code that pulls the tables from the 2024 Proposition 33 page at [paste the URL] and cleans them into one tidy data frame with the columns position, committee_id, committee_name and total (as a number).”

Compare what it gave you to what you wrote in steps 2 to 6, and critique it.

1. Does it run? Do its totals match the page ($48,482,311 support and $124,920,206 oppose)?
2. Go line by line. What did it do that you did not, and what did you do that it did not?
3. List everything it added that this page does not need.
4. Which version would you rather have to explain and maintain, and why?

Part 2: A function and iteration

8. A function that cleans one table

In steps 4 and 5 you copied and pasted the same three steps and changed only the table. That is the cue to write a function. Write clean_table(): it takes one raw table and returns the cleaned table.

What is the one thing that changed between your support and oppose code?

clean_table <- _______(x) {
  x |>
    rename(committee_id = 1, committee_name = 2, total = 3) |>
    slice(-(1:3)) |>
    mutate(total = parse_number(total))
}

clean_table(_______)

Is the result the same as support, without the position column?

9. A function that scrapes one page

So far we have assumed that the first table is support and the second is oppose. Not every page has two tables, so look for something on the page that says which table is which. Each table sits under a heading, and we can ask each table for the heading above it.

page |>
  html_elements("table") |>
  html_element(xpath = "preceding-sibling::h2[1]") |>
  _______()

xpath is another way to point at part of a page, like a CSS selector. This one starts at each table and goes to the closest h2 heading above it.

Now write scrape_prop(url). It pauses for a second, reads the page, finds the tables, cleans each one with clean_table(), labels each one by its heading, stacks them, and adds a column for the proposition.

scrape_prop <- function(_______) {
  Sys.sleep(1) # be polite: pause before each request

  page <- read_html(url)
  tables <- page |> html_elements("table")

  positions <- tables |>
    html_element(xpath = "preceding-sibling::h2[1]") |>
    html_text()

  tables |>
    html_table() |>
    map(_______) |>
    set_names(positions) |>
    list_rbind(names_to = "position") |>
    mutate(
      position = if_else(str_detect(position, "Support"), "support", "oppose"),
      prop = str_extract(url, "proposition-\\d+")
    )
}

Test it on Proposition 33, and then on Proposition 6.

(prop33_fromfun <- scrape_prop(url))

prop6_url <- "https://www.sos.ca.gov/campaign-lobbying/helpful-resources/measure-contributions/2024-ballot-measure-contribution-totals/proposition-6-aca-8-resolution-chapter-133-statutes-2024-wilson-slavery"
(prop6_fromfun <- _______(prop6_url))

How do the results compare to prop33? What is different about Proposition 6, and why does nothing break?

10. Get the links

Instead of pasting each address, pull the links from the index page, like we did in Deck 22.

index_page <- read_html("https://www.sos.ca.gov/campaign-lobbying/helpful-resources/measure-contributions/2024-ballot-measure-contribution-totals")

prop_urls <- index_page |>
  html_elements("a") |>
  _______("href") |>
  str_subset("proposition-\\d+") |>
  unique()

prop_urls

How many addresses did you get?

We’ll look at only three pages, for props 1, 6, and 33. That keeps the load on the server small while the whole class runs this at once. Keep the addresses that have proposition- followed by 1, 6, or 33. The | means “or”.

three_urls <- prop_urls |>
  str_subset("proposition-(_______)-")

three_urls

11. Iterate, politely

Map the function over the three addresses and stack the results.

props <- _______(three_urls, scrape_prop) |>
  list_rbind()

What happened?

Test that address by hand. How many tables does it have?

read_html(three_urls[_______]) |>
  html_elements("table")

Open that address in your browser. Where does it go? The real Proposition 1 page has the same address with helpful-resources in the path instead of cal-access-resources. Fix the addresses and run it again.

prop_urls <- prop_urls |>
  _______("cal-access-resources", "helpful-resources")

three_urls <- prop_urls |>
  str_subset("proposition-(1|6|33)-")

props <- map(three_urls, scrape_prop) |>
  list_rbind()

props

Why was it better that the failure was loud?

12. Check

Count the committees and total the money for each proposition and position.

props |>
  group_by(prop, position) |>
  summarize(committees = n(), total = _______(total), .groups = "drop")

1. Do the counts and totals for Proposition 33 match Part 1?
2. Proposition 6 has no oppose row. Does that mean opponents raised $0?
3. How many committees does Proposition 1 have on each side?

On HW 4 you will use these two functions on a whole year of propositions.