APIs & Web Scraping

API ও ওয়েব স্ক্র্যাপিং — web-এর সাথে কথা বলা

Read: ~30 min Intermediate 5 practice problems Install locally

1. Why Talk to the Web?

A huge amount of modern software just moves data between the internet and a local process. Consume a weather API, pull stock prices, post to Slack, scrape a pricing page, push metrics to a dashboard. Python has two excellent HTTP clients (requests, httpx) and the standard HTML parser (BeautifulSoup). Install with pip install requests httpx beautifulsoup4 lxml.

আধুনিক software-এর একটি বিশাল অংশের কাজ — internet থেকে ডেটা আনা বা পাঠানো। Python-এ দুটি চমৎকার HTTP client (requests, httpx) ও প্রধান HTML parser (BeautifulSoup)।

2. requests — The Classic Client

requests_demo.py
import requests

# GET — fetch a JSON API
r = requests.get("https://api.github.com/users/torvalds", timeout=10)
r.raise_for_status()
data = r.json()
print(data["login"], "—", data["public_repos"], "repos")

# Query params
r = requests.get("https://httpbin.org/get", params={"q": "python", "n": 5})
print(r.url)

# POST JSON
r = requests.post("https://httpbin.org/post", json={"city": "Dhaka"})
print(r.json()["json"])

# Headers & auth
h = {"User-Agent": "ABCL-TECH-Demo/1.0",
     "Authorization": "Bearer <token>"}
# r = requests.get(url, headers=h)

3. httpx — Modern + Async

httpx has the same sync API as requests but also supports async/await — letting you hit hundreds of URLs concurrently.

httpx_async.py
import asyncio, httpx

async def fetch(client, url):
    r = await client.get(url, timeout=10)
    return url, r.status_code

async def main():
    urls = [
        "https://httpbin.org/status/200",
        "https://httpbin.org/status/404",
        "https://httpbin.org/status/500",
    ]
    async with httpx.AsyncClient() as client:
        results = await asyncio.gather(*[fetch(client, u) for u in urls])

    for url, code in results:
        print(code, url)

asyncio.run(main())

4. HTML Scraping with BeautifulSoup

scrape.py
from bs4 import BeautifulSoup

html = """
<html><body>
  <h1>ABCL TECH Courses</h1>
  <ul class="courses">
    <li><a href="/c">C Programming</a></li>
    <li><a href="/py">Python Programming</a></li>
    <li><a href="/ai">Introduction to AI</a></li>
  </ul>
</body></html>
"""

soup = BeautifulSoup(html, "html.parser")

print(soup.find("h1").text)

for a in soup.select("ul.courses li a"):
    print(a.text, "→", a["href"])

5. Ethics & Etiquette

  • Read robots.txt — the file at site.com/robots.txt tells bots what they may and may not crawl.
  • Check terms of service — some sites explicitly forbid scraping.
  • Rate-limit yourself — 1–2 requests per second is usually polite; use time.sleep or semaphores in async code.
  • Identify yourself — set a real User-Agent header with a contact URL.
  • Cache aggressively — do not re-fetch the same page over and over.
  • Prefer APIs over HTML — if an API exists, use it.
Don't be the reason a small site goes down. Scraping is a privilege, not a right.

6. Reliability — Retries, Timeouts, Errors

reliability.py
import time, requests

def get_with_retries(url, tries=3, backoff=0.5):
    for attempt in range(tries):
        try:
            r = requests.get(url, timeout=10)
            r.raise_for_status()
            return r.json()
        except requests.RequestException as e:
            wait = backoff * (2 ** attempt)
            print(f"attempt {attempt+1} failed ({e}), waiting {wait}s")
            time.sleep(wait)
    raise RuntimeError(f"all {tries} attempts failed for {url}")

# data = get_with_retries("https://api.github.com/users/torvalds")
print("function defined")

7. Vocabulary (শব্দভাণ্ডার)

TermMeaningবাংলায়
APIApplication Programming Interface.Application-এর সাথে কথা বলার interface।
EndpointA specific URL an API exposes.API-র expose করা একটি URL।
PayloadData in the body of a request/response.Request/response-এর body-তে থাকা data।
ScrapingExtracting data from HTML pages.HTML পেজ থেকে data extract করা।
Rate limitMax requests per time window.প্রতি সময়-window-এ সর্বোচ্চ request সংখ্যা।

8. Practice Problems

  1. Write a function that fetches GitHub user info and returns (login, followers).
    GitHub user-এর info এনে (login, followers) return করুন।
    ✨ Show Answer (উত্তর দেখুন)
    ans1.py
    import requests

def user_info(name):
    r = requests.get(f"https://api.github.com/users/{name}", timeout=10)
    r.raise_for_status()
    d = r.json()
    return d["login"], d["followers"]

print(user_info("torvalds"))
  2. POST {"name":"Asif"} as JSON to httpbin and print the echoed body.
    httpbin-এ JSON POST করে echo হওয়া body প্রিন্ট করুন।
    ✨ Show Answer (উত্তর দেখুন)
    ans2.py
    import requests
r = requests.post("https://httpbin.org/post", json={"name": "Asif"})
print(r.json()["json"])
  3. Using BeautifulSoup, extract all <a href=...> links from an HTML string.
    BeautifulSoup দিয়ে HTML থেকে সব <a href=...> link বের করুন।
    ✨ Show Answer (উত্তর দেখুন)
    ans3.py
    from bs4 import BeautifulSoup
html = "<a href='/a'>A</a><a href='/b'>B</a><a href='/c'>C</a>"
soup = BeautifulSoup(html, "html.parser")
print([a["href"] for a in soup.find_all("a")])
  4. Explain why async (httpx) can be 10× faster than sync requests when hitting 100 URLs.
    ১০০টি URL hit করতে async httpx sync requests-এর চেয়ে ১০x দ্রুত কেন হতে পারে?
    ✨ Show Answer (উত্তর দেখুন)

    Answer: The bottleneck is I/O — waiting for the network, not Python computation. requests serializes: it waits for each response before starting the next. httpx with asyncio.gather fires many requests concurrently on one thread; while one is waiting on the network, others are being initiated and parsed, so total wall-clock time is roughly the slowest single request instead of the sum of all.

    বাধা হলো I/O — network-এর জন্য অপেক্ষা, Python computation নয়। requests serial — প্রতিটি response-এর জন্য অপেক্ষা করে। httpx+asyncio.gather এক thread-এ অনেক request একসাথে চালায়; একটির network-wait চলাকালীন অন্যগুলো শুরু ও parse হয় — wall-clock time সবচেয়ে ধীর একটি request-এর সমান।

  5. List three etiquette rules for web scraping.
    Web scraping-এর ৩টি etiquette নিয়ম লিখুন।
    ✨ Show Answer (উত্তর দেখুন)

    1) Respect robots.txt and the site's terms of service. 2) Rate-limit yourself — 1–2 requests per second — and use exponential backoff on errors. 3) Send a real, descriptive User-Agent header with a contact URL, and cache responses so you never re-download the same page unnecessarily.

Summary — Module 38

requests is the easy way to talk to HTTP APIs synchronously. httpx gives you the same API plus async for high-concurrency use cases. BeautifulSoup parses HTML when no API exists. Scrape politely — timeouts, retries, rate limits, honest User-Agent, and cache responses.

requests — সহজ synchronous HTTP client। httpx — একই API + async, high-concurrency-র জন্য। BeautifulSoup — API না থাকলে HTML parse। Scraping polite রাখুন — timeout, retry, rate limit, সৎ User-Agent, response cache।

Next Module → Machine Learning Preview — scikit-learn।

Previous: Flask / FastAPI Next: ML Preview