Before You Scrape

wk10-d01-checklist

Dr. D

Chico State
DATA 385 - Fall 2026

October 27, 2026

Today

Before we write any scraping code, we decide whether we should. Today we build a “before you scrape” checklist, test an LLM’s advice against it, and run IMDb through it. Then you use the checklist in AE08.

Think first, then pair up. Someone scraped 70,000 public OkCupid profiles and posted them online. The profiles were public. Was that okay? Yes, no, or it depends?

“Can you?” vs “Should you?”

  • Can you: is it technically possible? Almost always, yes.
  • Should you: is it allowed, is it harmless, and is it proportionate to what you need?

Public is not the same as okay. A checklist does not make the decision for you. It makes sure you looked before you scraped.

Build the checklist

Five things to check before scraping a site.

  1. robots.txt: Does the site ask robots to stay out of the pages you want?
  2. Terms of service: Do the site’s rules prohibit scraping, say nothing, or allow reuse?
  3. What data: Is it about people? Is it sensitive? Is it a public record?
  4. Load: How many requests will you make, and how will you pace them?
  5. Official alternative: Is there a download or an API instead?

1. robots.txt

Open www.csuchico.edu/robots.txt in your browser.

  • Find the rule for User-agent: *. The * means every robot, including yours.
  • Disallow: lines list the paths robots are asked to avoid. Disallow: / is the whole site.
  • Is a page you want listed under Disallow? An empty file asks robots to avoid nothing.
  • robots.txt is a request, not a lock.
  • allowing \(\neq\) permission. The terms of service still apply.

2. Terms of service

  • Look in the page footer or a “help” page for “Terms”, “Conditions of Use”, “Help”, or “Developers”.
  • Search the page for words like scrape, robot, automated and data mining.

3. What data is it?

  • Is it about people?
  • Is it sensitive?
  • Is it a public record?
  • Would the people in it expect it to be collected and combined?

Last week’s rules still apply. A scraped dataset about people is still personal data.

4. Load and 5. Alternatives

Load

  • Pause between requests.
  • Request only the pages you need.
  • Save what you scrape, so you are not re-scraping every time you render.

Official alternative

  • Is there a download or an API?
  • If the site offers one, it is usually easier and it is offered on purpose.

Ask before you scrape

Ask an LLM

Open an LLM (or collaborate with a neighbor) and send it this prompt.

Can I scrape IMDb's Top 250 page?

Check what it told you

  1. Did it cite real terms, or did it make some up?
  2. Did it hedge, or did it say it was fine? Can it grant permission that only IMDb can give?
  3. Could it know what IMDb’s terms say today? Or is it predicting what a helpful answer looks like?

Whatever it said, the next step is the same. Check the source.

Consider scraping IMDb

Run the checklist on IMDb’s Top 250 page.

https://www.imdb.com/chart/top/

What we found

  • robots.txt: Named search engines are mostly allowed. Every other robot gets Disallow: /. A comment at the top prohibits scraping without written permission.
  • Terms of service: The Help Center says “You may not use data mining, robots, screen scraping” and so on.
  • What data: Movie titles, years and ratings. Not personal data.
  • Load: 250 titles on one page. Light.
  • Official alternative: Yes. The Help Center points non-commercial users to IMDb’s datasets.

Sources:

As of 10/21/26.

Decision

Two items pass and two say no. We do not average them. A clear “no” in the site’s own rules ends it.

Important

We do not scrape IMDb.

Asking an LLM for help with scraping

Give it the URL and specify the part you want.

  • Is it a table? Try html_table() first. It needs no selectors.
  • Point at the piece. “The header that says…”, “the table under…”.
  • Say what you want back. A node name or CSS selector, what to look for to pull out rows, or a cleaning step for the html_text() output.
  • Constrain it. “rvest and tidyverse only, native pipe, html_elements(), no error handling, under 10 lines.”
  • Ask it to explain what each selector matches. Then run it and check the count against what you see on the page.
  • If it cannot open the page (blocked, or a login), right-click the thing you want, choose Inspect, and paste that element’s HTML. Public pages only.

Example prompts

Go to [URL]. What is the node name or CSS
selector for the header that says "[text]"?
On [URL], what should I look for to pull out
all the rows of the table under that header,
starting at the first data row? rvest only.
Here are 5 values from html_text(): [paste].
What pattern do they share? Tidyverse only:
how do I pull out just the number?
On [URL], how do I get the web address of
every link in the list of results, and not
the links in the menu? rvest only.