Python

🌸 Python Tutorial — BeautifulSoup (Web Scraping)

Introduction 🌟

BeautifulSoup is a Python library used for extracting data from HTML and XML pages. It is widely used in web scraping along with the requests module.

Note

💡 Parses HTML into a structured tree
💡 Easy searching (tags, classes, ids, attributes)
💡 Works with multiple parsers (default: html.parser)

1. Install BeautifulSoup 📦

install.sh

pip install beautifulsoup4 requests

2. Basic Usage — Fetch & Parse HTML 🌍

basic_parse.py

import requests
from bs4 import BeautifulSoup

url = "https://example.com"
res = requests.get(url)

soup = BeautifulSoup(res.text, "html.parser")
print(soup.title)

✔ BeautifulSoup turns the HTML text into a searchable object

3. Pretty Printing HTML 🌸

pretty_print.py

print(soup.prettify())

4. Finding Elements 🔍

Find by Tag

find_tag.py

element = soup.find("h1")
print(element.text)

Find All Matching Tags

find_all.py

links = soup.find_all("a")
for link in links:
    print(link.text, link["href"])

Find by Class

find_class.py

items = soup.find_all("div", class_="item")

Find by ID

find_id.py

title = soup.find(id="main-title")

Find with Attributes

find_attr.py

soup.find("img", {"src": "logo.png"})

5. Extracting Text 📝

text.py

paragraphs = soup.find_all("p")
for p in paragraphs:
    print(p.get_text())

6. Extracting Attributes 🎯

attributes.py

for link in soup.find_all("a"):
    print(link["href"])

safe_attr.py

link.get("href")  # safer than link["href"]

7. CSS Selectors (select, select_one) 🎨

css_selector.py

items = soup.select(".product .title")
for i in items:
    print(i.text)

✔ Uses CSS syntax: classes (.class), ids (#id), nesting (div > span)

8. Navigating the DOM Tree 🌴

Parent

parent.py

print(element.parent)

Children

children.py

for child in soup.div.children:
    print(child)

Next & Previous Sibling

siblings.py

title = soup.find("h1")
print(title.next_sibling)
print(title.previous_sibling)

9. Modifying HTML 🌐

modify_html.py

tag = soup.find("h1")
tag.string = "New Heading"
print(tag)

10. Removing Elements ❌

decompose.py

soup.find("script").decompose()

11. Parsing HTML from Files 📁

parse_file.py

with open("page.html") as f:
    soup = BeautifulSoup(f, "html.parser")

12. Web Scraping Example 🧩 (Products from a Page)

scrape_example.py

import requests
from bs4 import BeautifulSoup

url = "https://books.toscrape.com/"
res = requests.get(url)
soup = BeautifulSoup(res.text, "html.parser")

books = soup.select(".product_pod")

for book in books:
    title = book.h3.a["title"]
    price = book.select_one(".price_color").text
    print(title, "-", price)

13. Pagination Handling 🔁

pagination.py

page = 1

while True:
    url = f"https://example.com/products?page={page}"
    res = requests.get(url)
    soup = BeautifulSoup(res.text, "html.parser")

    items = soup.select(".item")
    if not items:
        break

    for i in items:
        print(i.text)

    page += 1

14. Avoiding Common Scraping Pitfalls ⚠️

✔ Always check website Terms of Service
✔ Use headers to mimic browsers
✔ Add delays between requests
✔ Handle exceptions & broken HTML
✔ Use proxies for large scrapes

15. Adding Fake Browser Headers 🥸

headers.py

headers = {
    "User-Agent": "Mozilla/5.0"
}

res = requests.get("https://example.com", headers=headers)

16. Using BeautifulSoup with lxml Parser ⚡

lxml.py

pip install lxml

use_lxml.py

soup = BeautifulSoup(html, "lxml")

✔ Faster & more robust parsing

17. Scraping Tables 🧮

scrape_table.py

rows = soup.select("table tr")

for row in rows:
    cols = [col.text.strip() for col in row.select("td")]
    print(cols)

18. Exporting Scraped Data to CSV 📄

export_csv.py

import csv

with open("data.csv", "w", newline="") as f:
    writer = csv.writer(f)
    writer.writerow(["Title", "Price"])

    for book in books:
        writer.writerow([book_title, book_price])

BeautifulSoup Cheat Sheet 📘

Task	Method
Find first tag	soup.find()
Find all tags	soup.find_all()
CSS selector	soup.select()
Get text	tag.get_text()
Get attribute	tag["href"]
Prettify	soup.prettify()

Best Practices 💡

✔ Always respect website’s robots.txt
✔ Avoid scraping dynamic JavaScript sites (use Selenium instead)
✔ Cache pages when scraping many times
✔ Use BeautifulSoup with requests, not urllib
✔ Clean extracted data (strip, replace)

Conclusion 🎉

>>“BeautifulSoup turns messy HTML into clean, searchable Python objects — perfect for web scraping and data extraction.” ✨

You now fully understand BeautifulSoup in Python! Want the next topic? Try Selenium Automation, Scrapy Framework, Async Web Scraping, or API-based scraping. Just tell me! 😊