πΈοΈ Python Tutorial β Selenium (Web Automation & Web Scraping)
Introduction π
Selenium is a browser automation tool used for web testing, automation, and scraping dynamic JavaScript websites. It controls real browsers like Chrome, Firefox, Edge, Safari through WebDriver.
Note
π‘ Best for dynamic sites (JS-rendered content)
π‘ Can fill forms, click buttons, extract data
π‘ Works with ChromeDriver, GeckoDriver, etc.
π‘ Can fill forms, click buttons, extract data
π‘ Works with ChromeDriver, GeckoDriver, etc.
1. Install Selenium π¦
install.sh
pip install selenium webdriver-managerβ webdriver-manager automatically downloads drivers
2. Launching a Browser (Chrome) π
launch_browser.py
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get("https://google.com")
print(driver.title)
driver.quit()β Opens Chrome
β Loads URL
β Closes browser
3. Selecting Elements π
Find by ID
find_id.py
element = driver.find_element("id", "username")Find by Name
find_name.py
element = driver.find_element("name", "q")Find by Class
find_class.py
driver.find_element("class name", "btn-login")Find by CSS Selector
find_css.py
driver.find_element("css selector", "div.header > a.link")Find by XPath
find_xpath.py
driver.find_element("xpath", "//button[@type='submit']")Note
π‘ XPath is the most powerful but can become long.
4. Sending Text (Typing) β¨οΈ
typing.py
search = driver.find_element("name", "q")
search.send_keys("Python Selenium Tutorial")5. Clicking Elements π±οΈ
click.py
button = driver.find_element("css selector", ".btn-search")
button.click()6. Extracting Text & Attributes π
extract.py
text = driver.find_element("id", "title").text
link = driver.find_element("tag name", "a").get_attribute("href")
print(text, link)7. Handling Waits (Important!) β³
Implicit Wait
implicit_wait.py
driver.implicitly_wait(5)Explicit Wait
explicit_wait.py
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(driver, 10)
element = wait.until(EC.presence_of_element_located((By.ID, "product")))Note
β Prefer explicit waits for consistency
8. Scrolling the Page π
scroll.py
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")9. Handling Forms π§Ύ
form.py
username = driver.find_element("id", "user")
password = driver.find_element("id", "pass")
username.send_keys("admin")
password.send_keys("secret")
driver.find_element("id", "login").click()10. Handling Alerts β οΈ
alerts.py
alert = driver.switch_to.alert
print(alert.text)
alert.accept()11. Handling Dropdowns π½
dropdown.py
from selenium.webdriver.support.ui import Select
select = Select(driver.find_element("id", "country"))
select.select_by_visible_text("India")12. Switching Windows & Tabs πͺ
switch_window.py
driver.find_element("css selector", "a.new-tab").click()
tabs = driver.window_handles
driver.switch_to.window(tabs[1])13. Taking Screenshots πΈ
screenshot.py
driver.save_screenshot("page.png")14. Running Headless Mode π€ (Without UI)
headless.py
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("--headless")
driver = webdriver.Chrome(ChromeDriverManager().install(), options=options)β Useful for automation & servers
15. Selenium for Web Scraping π§©
scrape_dynamic.py
from selenium import webdriver
from bs4 import BeautifulSoup
import time
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get("https://quotes.toscrape.com/js/")
time.sleep(3) # wait for JS
soup = BeautifulSoup(driver.page_source, "html.parser")
quotes = soup.select(".quote")
for q in quotes:
print(q.text.strip())
driver.quit()16. Selenium With Actions (Mouse/Keyboard) π±οΈβ¨οΈ
actions.py
from selenium.webdriver import ActionChains
actions = ActionChains(driver)
element = driver.find_element("id", "drag")
target = driver.find_element("id", "drop")
actions.drag_and_drop(element, target).perform()17. File Uploads & Downloads π€π₯
Upload
upload.py
file_input = driver.find_element("id", "file")
file_input.send_keys("/path/to/file.png")Download (Chrome)
download.py
options.add_experimental_option("prefs", {
"download.default_directory": "/my/download/path"
})18. Closing Browser Properly π
close.py
driver.quit()Selenium Locators Cheat Sheet π
| Locator | Example |
|---|---|
| ID | find_element("id", "username") |
| Name | find_element("name", "q") |
| Class | find_element("class name", "item") |
| Tag | find_element("tag name", "a") |
| CSS Selector | find_element("css selector", ".title") |
| XPath | find_element("xpath", "//div[@class='info']") |
Best Practices π‘
- β Use explicit waits instead of sleep()
- β Avoid XPath when CSS selector works
- β Use headless mode for automation
- β Keep ChromeDriver updated
- β Do not overload websites with rapid requests
Conclusion π
>>βSelenium lets you control browsers like a human β perfect for automation, testing, and scraping dynamic websites.β β¨
You now understand Selenium in Python! Want the next topic? Try Scrapy Framework, Async Selenium (undetected_chromedriver), FastAPI Automation, or Browser Automation Bots. Just tell me! π