AE08 - Web Scraping (key)

Suggested answers

Author

Dr. D

Published

October 30, 2026

Important

These are suggested answers. This document is a reference, not an exhaustive key.

Purpose

Who pays for California’s ballot measures? Every proposition has a page on the California Secretary of State’s website listing the committees that support and oppose it, and how much money each one reported raising. In this AE you will decide whether you should scrape that site, then get one proposition’s tables into R and clean them into a tidy data frame. On Thursday you will turn that work into a function and use it on multiple 2024 propositions.

By the end of Part 1 you should be able to:

  • Use a checklist and decide, with reasons, whether a site can be scraped
  • Read an HTML page into R and pull out its tables with read_html() and html_table()
  • Clean a messy scraped table into a tidy data frame
  • Check scraped numbers against the totals printed on the page

By the end of Part 2 you should be able to:

  • Write a function with an argument that scrapes one page
  • Use map() and list_rbind() to apply it to many pages, with a pause between requests
  • Diagnose a page that breaks the loop, and check the combined result
WarningA lot of new functions today

We are throwing a lot of new functions at you today. Make a point of understanding every line you run. Add comments to the code that explain each line in words that make sense to you. Use help files, function vignettes, or an LLM to help you learn what each line is doing.

Instructions

  • Work through the parts in order, and compare your numbers with a neighbor.
  • You may use an LLM. Annotate any LLM-generated code line by line with a comment that says what the line does. If you cannot explain a line, do not submit it.
  • Tuesday’s work is not graded on its own. This AE is turned in after Thursday’s class.
NoteA note on the videos

The Data Science in a Box videos were made a few years ago and use older function names. The old versions still run, so you will see them in the videos and in examples online. In this class we use the current versions:

  • html_node() in the videos has been superseded by html_element()
  • html_nodes() in the videos has been superseded by html_elements()
  • map_dfr() in the videos has been superseded by map() followed by list_rbind() in this class

The videos and slides mostly pull things out of a page with CSS selectors, and html_table() is only named in a list of functions. Today’s pages are plain tables, so we use html_table(), which needs no selectors.

The data

The data come from the California Secretary of State: 2024 Ballot Measure Contribution Totals. That index page links to one page for each proposition. Today you will work with one of them, Proposition 33 (rent control).

Each proposition page has up to two tables: committees in support, and committees in opposition. Each row is one committee, with its ID, its name, and the total contributions it reported.

Setup

library(tidyverse)
library(rvest)

Part 1: One page

1. Before you scrape

Run the checklist on the SOS site, then make a decision.

robots.txt. Open https://www.sos.ca.gov/robots.txt in your browser. What does it say about the pages we want?

The page loads but the file is empty, so it does not ask robots to stay away from anything. An empty file is not the same as permission. It only tells us the site has not restricted any paths, so we still have to check the other items.

Terms of service. Read the CA.gov Conditions of Use. Does it prohibit scraping?

No. It does not mention scraping or automated access. It says that information on the site is, in general, considered public domain, and it warns that some items, such as photographs, may be copyrighted. We are not using any photographs.

What data? Is it personal, sensitive, or public record?

Committee IDs, committee names, and the dollar totals each committee reported. These are campaign finance filings that the state publishes. The SOS says it follows the Public Records Act, which gives the public a right to access state government records (see its privacy policy). The pages report totals for committees, not individual donors.

Load. How many requests will we make, and how will we keep them polite?

Today it is one page, later it will be multiple with a pause between requests. The SOS logs visitor IP addresses, so we only request the pages we need, and we will not request them faster than a person could click.

Is there an official alternative? Is there a download or an API?

Yes. Cal-Access publishes raw data downloads of the underlying filings. That is far more than we need for this AE so scraping the pages is reasonable here. If we needed the individual filings, we would use the download.

Decision. Should we scrape this site? Write one or two sentences with reasons.

Yes. robots.txt does not restrict the pages, the Conditions of Use do not prohibit it, the data are public records about committees rather than people, and the load is small and paced.

One boundary. The committee names on these pages link to Cal-Access, a different website (cal-access.sos.ca.gov) from the one we scrape (www.sos.ca.gov). Each site has its own robots.txt, and Cal-Access’s says Disallow: / for every robot. So we scrape the SOS pages and we do not follow those links.

2. Read the page

read_html() downloads the page’s HTML into R. Each time you run it you send one request to the state’s server, so run it once and then work from the saved page object.

url <- "https://www.sos.ca.gov/campaign-lobbying/helpful-resources/measure-contributions/2024-ballot-measure-contribution-totals/proposition-33-expands-local-governments-authority-enact-rent-control-residential-property-initiative-statute"

(page <- read_html(url))
{html_document}
<html lang="en">
[1] <head>\n<script src="/Play-at-cleauy-Station-All-thoughts-about-the-Su" a ...
[2] <body>\n\n<div class="ccm-page ccm-page-id-19826 page-type-left-sidebar p ...

3. Pull out the tables

  • html_table() finds every table on the page and returns a list with one data frame per table.
  • Use it on page and save the result as tables.
  • How many tables did you get, and how does that compare to the page in your browser?
tables <- page |> html_table()
length(tables)
[1] 2

How many tables did you get, and how does that compare to the page in your browser?

Two: support and oppose, in the order they appear on the page.

If you open your browser’s developer tools you will see more than two table elements. The extra ones belong to the search box and are added by JavaScript after the page loads. rvest only sees the HTML the server sends, not what JavaScript adds later.

4. Clean the support table

Look at the first table, tables[[1]]. tables is a list, and double square brackets pull one data frame out of it: tables[[1]] is the first and tables[[2]] is the second.

tables[[1]]
# A tibble: 5 × 3
  X1                                                                 X2    X3   
  <chr>                                                              <chr> <chr>
1 Total amount of reported contributions to this measure: $48,482,3… Tota… Tota…
2 Individual Ballot Measure Committees Formed for this Measure       Indi… Indi…
3 Committee ID**                                                     Comm… Tota…
4 1297793                                                            Kevi… $841…
5 1458912                                                            Rent… $47,…

What is wrong with this table as a data frame?

Fix the problems and save the result as support.

  • Name the columns committee_id, committee_name and total
  • Drop the rows that correspond to headers and subheaders
  • Convert total to a number
  • Add a position variable with the value “support”.
support <- tables[[1]] |>
  rename(committee_id = 1, committee_name = 2, total = 3) |>
  slice(-(1:3)) |>
  mutate(total = parse_number(total), 
         position = "support")

support # confirm your modifications work
# A tibble: 2 × 4
  committee_id committee_name                                     total position
  <chr>        <chr>                                              <dbl> <chr>   
1 1297793      Kevin De Leon Believing in a Better California B… 8.42e5 support 
2 1458912      Renters and Homeowners for Rent Control Yes on 3… 4.76e7 support

5. Clean the oppose table

Do the same thing for the oppose table.

oppose <- tables[[2]] |>
  rename(committee_id = 1, committee_name = 2, total = 3) |>
  slice(-(1:3)) |>
  mutate(total = parse_number(total), 
         position = "oppose")

oppose
# A tibble: 5 × 4
  committee_id committee_name                                     total position
  <chr>        <chr>                                              <dbl> <chr>   
1 1469388      California Affordable Rental Housing Coalition, … 0      oppose  
2 1400190      Homeownership for Families Sponsored by Californ… 2.70e7 oppose  
3 1405723      Issues PAC of Apartment Association of Greater L… 1.08e6 oppose  
4 1421884      No on 33, Californians for Responsible Housing: … 8.98e7 oppose  
5 1462983      No on Prop 33, Californians to Protect Affordabl… 7.00e6 oppose

6. Stack and check

Stack the two data frames into one with bind_rows() and save it as prop33.

(prop33 <- bind_rows(support, oppose))
# A tibble: 7 × 4
  committee_id committee_name                                     total position
  <chr>        <chr>                                              <dbl> <chr>   
1 1297793      Kevin De Leon Believing in a Better California B… 8.42e5 support 
2 1458912      Renters and Homeowners for Rent Control Yes on 3… 4.76e7 support 
3 1469388      California Affordable Rental Housing Coalition, … 0      oppose  
4 1400190      Homeownership for Families Sponsored by Californ… 2.70e7 oppose  
5 1405723      Issues PAC of Apartment Association of Greater L… 1.08e6 oppose  
6 1421884      No on 33, Californians for Responsible Housing: … 8.98e7 oppose  
7 1462983      No on Prop 33, Californians to Protect Affordabl… 7.00e6 oppose

Scraped numbers can be wrong without any error message, so check them against the page. Calculate the number of committees are on each side, and each side’s total.

prop33 |>
  group_by(position) |>
  summarize(committees = n(), total = sum(total))
# A tibble: 2 × 3
  position committees     total
  <chr>         <int>     <dbl>
1 oppose            5 124920206
2 support           2  48482311

Support has 2 committees and $48,482,311. Oppose has 5 committees and $124,920,206. Both match the totals printed at the top of each table on the page.

Now look at the data itself.

  1. Which side raised more, and by how much?

Opposition raised about 2.6 times as much as support: $124.9 million against $48.5 million, a difference of about $76.4 million.

  1. One oppose committee reports $0. Which one?
prop33 |>
  filter(total == 0)
# A tibble: 1 × 4
  committee_id committee_name                                     total position
  <chr>        <chr>                                              <dbl> <chr>   
1 1469388      California Affordable Rental Housing Coalition, N…     0 oppose

The California Affordable Rental Housing Coalition, committee ID 1469388.

  1. One support committee is named “Kevin De Leon Believing in a Better California Ballot Measure Committee - Yes on Propositions 3, 32 and 33”. What does that name tell you about its total, and what would go wrong if you added up money across propositions?

That committee also supports Propositions 3 and 32, and it reports one total for all three. We cannot tell how much of the $841,741 belongs to Proposition 33. If we added up committee totals across all the propositions, this money would be counted more than once.

7. Compare with an LLM (if time permits)

WarningYour turn

Ask your LLM: “Write R code that pulls the tables from the 2024 Proposition 33 page at [paste the URL] and cleans them into one tidy data frame with the columns position, committee_id, committee_name and total (as a number).”

Compare what it gave you to what you wrote in steps 2 to 6, and critique it.

  1. Does it run? Do its totals match the page ($48,482,311 support and $124,920,206 oppose)?
  2. Go line by line. What did it do that you did not, and what did you do that it did not?
  3. List everything it added that this page does not need.
  4. Which version would you rather have to explain and maintain, and why?

The shortest version is the one you wrote: read the page, pull the tables, clean each one, and stack them. Common extras to look for are browser automation to handle JavaScript, custom headers or a fake user agent, status checks on the request, tryCatch(), and code that installs packages. Also look for guesses about the data, such as column names that do not exist or steps that assume a different table layout. This page is plain HTML with plain tables, so none of that is needed.

Part 2: A function and iteration

8. A function that cleans one table

In steps 4 and 5 you copied and pasted the same three steps and changed only the table. That is the cue to write a function. Write clean_table(): it takes one raw table and returns the cleaned table.

What is the one thing that changed between your support and oppose code?

It was the table: tables[[1]] or tables[[2]]. That is the argument. Inside the function we call it x.

clean_table <- function(x) {
  x |>
    rename(committee_id = 1, committee_name = 2, total = 3) |>
    slice(-(1:3)) |>
    mutate(total = parse_number(total))
}

clean_table(tables[[1]])
# A tibble: 2 × 3
  committee_id committee_name                                              total
  <chr>        <chr>                                                       <dbl>
1 1297793      Kevin De Leon Believing in a Better California Ballot Mea… 8.42e5
2 1458912      Renters and Homeowners for Rent Control Yes on 33, Sponso… 4.76e7

The result is the same as support from step 4, without the position column.

9. A function that scrapes one page

So far we have assumed that the first table is support and the second is oppose. That is true for Proposition 33, but not every page has two tables. Look for something on the page that says which table is which. Each table sits under a heading, and we can ask each table for the heading above it.

page |>
  html_elements("table") |>
  html_element(xpath = "preceding-sibling::h2[1]") |>
  html_text()
[1] "In Support of this measure: "    "In Opposition to this measure: "

xpath is another way to point at part of a page, like a CSS selector. This one starts at each table and goes to the closest h2 heading above it.

Now write scrape_prop(url). It pauses for a second, reads the page, finds the tables, cleans each one with clean_table(), labels each one by its heading, stacks them, and adds a column for the proposition.

scrape_prop <- function(url) {
  Sys.sleep(1) # be polite: pause before each request

  page <- read_html(url)
  tables <- page |> html_elements("table")

  positions <- tables |>
    html_element(xpath = "preceding-sibling::h2[1]") |>
    html_text()

  tables |>
    html_table() |>
    map(clean_table) |>
    set_names(positions) |>
    list_rbind(names_to = "position") |>
    mutate(
      position = if_else(str_detect(position, "Support"), "support", "oppose"),
      prop = str_extract(url, "proposition-\\d+")
    )
}

Test it on Proposition 33, and then on Proposition 6.

(prop33_fromfun <- scrape_prop(url))
# A tibble: 7 × 5
  position committee_id committee_name                               total prop 
  <chr>    <chr>        <chr>                                        <dbl> <chr>
1 support  1297793      Kevin De Leon Believing in a Better Califo… 8.42e5 prop…
2 support  1458912      Renters and Homeowners for Rent Control Ye… 4.76e7 prop…
3 oppose   1469388      California Affordable Rental Housing Coali… 0      prop…
4 oppose   1400190      Homeownership for Families Sponsored by Ca… 2.70e7 prop…
5 oppose   1405723      Issues PAC of Apartment Association of Gre… 1.08e6 prop…
6 oppose   1421884      No on 33, Californians for Responsible Hou… 8.98e7 prop…
7 oppose   1462983      No on Prop 33, Californians to Protect Aff… 7.00e6 prop…
prop6_url <- "https://www.sos.ca.gov/campaign-lobbying/helpful-resources/measure-contributions/2024-ballot-measure-contribution-totals/proposition-6-aca-8-resolution-chapter-133-statutes-2024-wilson-slavery"
(prop6_fromfun <- scrape_prop(prop6_url))
# A tibble: 4 × 5
  position committee_id committee_name                               total prop 
  <chr>    <chr>        <chr>                                        <dbl> <chr>
1 support  1474463      All of Us or None Action Network (NonProfi… 175000 prop…
2 support  1476058      Quest for Democracy Action Network, Inc. (… 378299 prop…
3 support  1471645      Yes on Prop 6, End Slavery in California, … 364492 prop…
4 support  1471998      Yes on Proposition 6, Ending Modern Day Sl… 675741 prop…

Proposition 33 gives the same 7 committees as prop33, plus a prop column. Proposition 6 gives 4 committees, all support. Its page has no oppose table, so there are no oppose rows, and nothing breaks. If we had used tables[[2]], this would have failed with a “subscript out of bounds” error.

11. Iterate, politely

Map the function over the three addresses and stack the results.

props <- map(three_urls, scrape_prop) |>
  list_rbind()
Error in `map()`:
ℹ In index: 3.
Caused by error in `mutate()`:
ℹ In argument: `position = if_else(str_detect(position, "Support"),
  "support", "oppose")`.
Caused by error:
! object 'position' not found

What happened?

The first two pages worked and the third failed. The error names the position of the page that failed, 3, which is Proposition 1.

Test that address by hand. How many tables does it have?

read_html(three_urls[3]) |>
  html_elements("table")
{xml_nodeset (0)}

None. Open that address in your browser. It does not stay on a Proposition 1 page. The index links to a path that now redirects to a general page with no tables. The real Proposition 1 page has the same address with helpful-resources in the path instead of cal-access-resources. Fix the addresses and run it again.

prop_urls <- prop_urls |>
  str_replace("cal-access-resources", "helpful-resources")

three_urls <- prop_urls |>
  str_subset("proposition-(1|6|33)-")

props <- map(three_urls, scrape_prop) |>
  list_rbind()

props
# A tibble: 18 × 5
   position committee_id committee_name                              total prop 
   <chr>    <chr>        <chr>                                       <dbl> <chr>
 1 support  1474463      All of Us or None Action Network (NonProf… 1.75e5 prop…
 2 support  1476058      Quest for Democracy Action Network, Inc. … 3.78e5 prop…
 3 support  1471645      Yes on Prop 6, End Slavery in California,… 3.64e5 prop…
 4 support  1471998      Yes on Proposition 6, Ending Modern Day S… 6.76e5 prop…
 5 support  1297793      Kevin De Leon Believing in a Better Calif… 8.42e5 prop…
 6 support  1458912      Renters and Homeowners for Rent Control Y… 4.76e7 prop…
 7 oppose   1469388      California Affordable Rental Housing Coal… 0      prop…
 8 oppose   1400190      Homeownership for Families Sponsored by C… 2.70e7 prop…
 9 oppose   1405723      Issues PAC of Apartment Association of Gr… 1.08e6 prop…
10 oppose   1421884      No on 33, Californians for Responsible Ho… 8.98e7 prop…
11 oppose   1462983      No on Prop 33, Californians to Protect Af… 7.00e6 prop…
12 support  1466985      SF District Attorney Brooke Jenkins Ballo… 3.62e5 prop…
13 support  1466023      Statewide California Coalition Solutions … 0      prop…
14 support  1466153      Uber Innovation Ballot Measure Committee,… 4.50e6 prop…
15 support  1380675      Yes on Prop 1 - Governor Newsom's Ballot … 2.06e7 prop…
16 support  1463883      Yes on Proposition 1, Assemblymember Jacq… 1.15e5 prop…
17 support  1465319      Yes on Proposition 1 - Expanding Access t… 1.18e6 prop…
18 oppose   1465428      Californians Against Proposition 1         1   e3 prop…

This time the failure was loud. If the function had quietly returned an empty table instead, Proposition 1 would have been missing from the results with no warning. That is why the next step checks the result.

12. Check

Count the committees and total the money for each proposition and position.

props |>
  group_by(prop, position) |>
  summarize(committees = n(), total = sum(total), .groups = "drop")
# A tibble: 5 × 4
  prop           position committees     total
  <chr>          <chr>         <int>     <dbl>
1 proposition-1  oppose            1      1000
2 proposition-1  support           6  26723087
3 proposition-33 oppose            5 124920206
4 proposition-33 support           2  48482311
5 proposition-6  support           4   1593532

  1. Do the counts and totals for Proposition 33 match Part 1?

  2. Proposition 6 has no oppose row. Does that mean opponents raised $0?

  3. How many committees does Proposition 1 have on each side?

  4. Yes: 2 support committees with $48,482,311, and 5 oppose committees with $124,920,206.

  5. No. The page lists no committees opposing it, so there are no rows at all. A missing row is not the same as a $0 row. Also check the Proposition 6 support total against the page: the committees add up to $1,593,532, but the page states $1,593,533.

  6. Read the counts off your table. The page has 6 support committees and 1 oppose committee.

On HW 4 you will use these two functions on a whole year of propositions.