đ Introduction
APIs (Application Programming Interfaces) and Web Scraping are two common techniques for collecting data from the internet. APIs provide structured and authorized access to data, while web scraping extracts information directly from web pages by parsing their HTML content. R offers powerful packages for both approaches, enabling developers and data analysts to build automated data collection workflows.
Information
đ¯ Why Learn APIs and Web Scraping?
- Collect real-time data from online services.
- Automate data collection processes.
- Access public datasets and online resources.
- Build data-driven dashboards and applications.
- Support research, analytics, and machine learning projects.
đ APIs vs Web Scraping
| Feature | API | Web Scraping |
|---|---|---|
| Data Format | Structured (JSON, XML). | HTML pages. |
| Reliability | High. | Depends on website structure. |
| Speed | Usually faster. | Often slower. |
| Authentication | May require API keys. | Usually not required. |
| Recommended | Yes, when available. | When no suitable API exists. |
đĻ Installing Required Packages
Install Packages
install.packages("httr")
install.packages("jsonlite")
install.packages("rvest")
install.packages("xml2")
library(httr)
library(jsonlite)
library(rvest)
library(xml2)đ API Workflow
đĄ Sending an HTTP GET Request
The GET() function retrieves data from a web API.
GET Request
library(httr)
response <- GET(
"https://api.example.com/users"
)
status_code(
response
)đ Reading JSON Data
Many APIs return data in JSON format, which can be converted into R objects.
Parse JSON
library(jsonlite)
response <- GET(
"https://api.example.com/users"
)
data <- fromJSON(
content(
response,
"text"
)
)
head(data)đ Using API Keys
Some APIs require authentication using an API key.
Authenticated API Request
response <- GET(
"https://api.example.com/data",
add_headers(
Authorization =
"Bearer YOUR_API_KEY"
)
)đ¤ Sending POST Requests
POST requests send data to a server, often when creating new resources.
POST Request
response <- POST(
"https://api.example.com/users",
body = list(
name = "Alice",
age = 25
),
encode = "json"
)đ Working with API Responses
Inspect Response
status_code(
response
)
headers(
response
)
content(
response
)đ Common HTTP Methods
| Method | Purpose |
|---|---|
| GET | Retrieve data. |
| POST | Create new data. |
| PUT | Replace existing data. |
| PATCH | Update part of existing data. |
| DELETE | Remove data. |
đ¸ Introduction to Web Scraping
Web scraping extracts information directly from HTML pages using CSS selectors or XPath expressions.
Read a Web Page
library(rvest)
page <- read_html(
"https://example.com"
)đ Extracting Page Titles
Extract Title
page %>%
html_element(
"title"
) %>%
html_text()đ Extracting Text
Extract Paragraphs
page %>%
html_elements(
"p"
) %>%
html_text()đ Extracting Links
Extract Hyperlinks
page %>%
html_elements(
"a"
) %>%
html_attr(
"href"
)đ Extracting Tables
HTML tables can be converted directly into R data frames.
Read HTML Tables
tables <- page %>%
html_table()
tables[[1]]đ¯ Using CSS Selectors
CSS selectors identify specific HTML elements for extraction.
CSS Selector Example
page %>%
html_elements(
".article-title"
) %>%
html_text()đ Using XPath
XPath provides another way to locate HTML elements.
XPath Example
page %>%
html_elements(
xpath =
"//h2"
) %>%
html_text()đž Saving Scraped Data
Save to CSV
titles <- page %>%
html_elements(
"h2"
) %>%
html_text()
write.csv(
titles,
"titles.csv",
row.names = FALSE
)đ Real-World Example
A financial analyst collects daily stock information using a public financial API. When historical news headlines are unavailable through an API, the analyst scrapes article titles from publicly accessible news pages, respecting the website's terms of service and usage policies.
News Headline Extraction
library(rvest)
page <- read_html(
"https://example.com/news"
)
headlines <- page %>%
html_elements(
"h2"
) %>%
html_text()
print(
headlines
)đ Data Collection Workflow
đ Common Functions
| Function | Purpose |
|---|---|
| GET() | Sends an HTTP GET request. |
| POST() | Sends an HTTP POST request. |
| fromJSON() | Parses JSON data. |
| read_html() | Reads an HTML page. |
| html_element() | Selects a single HTML element. |
| html_elements() | Selects multiple HTML elements. |
| html_text() | Extracts text content. |
| html_table() | Extracts HTML tables. |
| html_attr() | Extracts HTML attributes. |
â ī¸ Common Mistakes
| Mistake | Explanation | Solution |
|---|---|---|
| Ignoring API documentation | Requests may fail due to incorrect parameters. | Read the API documentation carefully before making requests. |
| Hardcoding API keys | Credentials may be exposed. | Store API keys securely using environment variables or configuration files. |
| Scraping websites too aggressively | May overload servers or violate usage policies. | Respect rate limits, robots.txt guidance where appropriate, and the website's terms of service. |
| Assuming HTML structure never changes | Scrapers can stop working when websites are updated. | Write robust selectors and periodically maintain scraping code. |
đĄ Best Practices
- Use official APIs whenever they are available.
- Read API documentation before integrating with a service.
- Handle HTTP errors and unexpected responses gracefully.
- Keep API credentials secure and never expose them publicly.
- Respect website terms of service and request rate limits when scraping.
- Clean and validate collected data before analysis.
- Cache responses when appropriate to reduce unnecessary requests.
Best Practice
đ Summary
APIs and web scraping enable R to collect data from online sources for analysis and automation. In this chapter, you learned how to send HTTP requests using the httr package, parse JSON responses with jsonlite, authenticate using API keys, perform GET and POST requests, and inspect API responses. You also explored web scraping using rvest, including reading HTML pages, extracting text, links, tables, and attributes with CSS selectors and XPath expressions. Mastering these techniques allows you to build automated workflows that gather, process, and analyze data from the web efficiently and responsibly.