API ও web scraping

Web scraping-এর legality jurisdiction, content type, ও use case-ভেদে ভিন্ন।

Global landmark cases:

• HiQ Labs vs LinkedIn (২০১৭, US): public profile scraping permitted; LinkedIn-এর "no scraping" ToS not enforceable for public data।
• Facebook vs Power Ventures: automated access without permission CFAA violation।
• Ryanair vs PR Aviation: EU — database right invoke।
• Van Buren vs US: CFAA "exceeds authorized access" narrow — scraping public data সাধারণত OK।

Bangladesh-specific concerns:

1. Digital Security Act 2018:
   • Section 22 — unauthorized access to data → up to 7 years।
   • Section 25 — fake/false data publish → criminal।
   • Vague language — risk-averse approach।
2. Personal Data Protection Bill (২০২৪ draft):
   • EU GDPR-এর আদলে।
   • Personal data scraping → consent দরকার।
   • Cross-border transfer regulated।
3. Copyright Act 2000:
   • News article republish → infringement।
   • Database compilation copyright।
   • Fair use exceptions narrow।
4. ICT Act 2006 (amended):
   • Section 56 — unauthorized access penalty।
   • Section 57 — defamation via digital — vague।

Ethical framework — beyond law:

• Public interest: journalism, research → defensible।
• Commercial repackaging: someone else's content monetize — gray area।
• Personal data: avoid even if technically allowed।
• Server burden: small site DDoS — irrespective of legality, irresponsible।
• Reciprocity: "would I want my site scraped this way?"

Safe scraping checklist:

1. robots.txt check।
2. ToS read — explicit "no scraping" → reconsider।
3. Identifiable User-Agent + contact email।
4. Rate limit: ১-৫ সে delay; off-peak hours।
5. Public data only — no login/paywall bypass।
6. No PII; aggregated metrics OK।
7. Cite source if publish।
8. Stop on cease-and-desist।
9. Lawyer consult-এর জন্য proceed-এ doubt।

API-first culture:

• Email site owner — API request — অনেকে provide করেন।
• Partnership অপশন — sometimes paid API access।
• Open data movement support — government data unlock-এ advocate।

Bangladesh real cases:

• News aggregator Bondhu — copyright concern news scrape।
• Job portal data scraping — student researchers ToS-এ atki।
• Real estate aggregator — bikroy + bproperty দু'টোর scraping issue।

মূল উপলব্ধি: Legality + ethics + practicality — তিনটি বিচার। Lawyer-এর কাছে ১ ঘণ্টার consultation = পরের ১ বছরের headache কম।

"Quick script" থেকে "production pipeline"-এর পার্থক্য rate limit handling-এ।

Rate limit-এর প্রকৃতি:

• API provider compute & cost protect করে।
• Common patterns: 60/min, 1000/hour, 10000/day।
• Burst limits + sustained limits — ভিন্ন।
• Headers: X-RateLimit-Remaining, X-RateLimit-Reset।
• HTTP 429 (Too Many Requests) — explicit signal।

Exponential backoff math:

delay(n) = base * (factor ^ n) + jitter
# 1st retry: 1s
# 2nd retry: 2s
# 3rd retry: 4s
# 4th retry: 8s
# ...
# +random jitter to avoid thundering herd

Why exponential — not linear:

• Linear: 1s, 2s, 3s — slow recovery।
• Exponential: 1s, 2s, 4s, 8s — quick relief in early failures, give server time।
• Distributed system theory — proven optimal।

Implementation patterns:

# 1. Library-based (preferred)
from urllib3.util.retry import Retry
retry = Retry(
    total=5,
    backoff_factor=1.5,
    status_forcelist=[429, 500, 502, 503, 504],
)

# 2. Manual with tenacity
from tenacity import retry, stop_after_attempt, wait_exponential

@retry(
    stop=stop_after_attempt(5),
    wait=wait_exponential(multiplier=1, min=2, max=60),
)
def fetch(url):
    r = requests.get(url, timeout=10)
    r.raise_for_status()
    return r.json()

# 3. Adaptive (read header)
def adaptive_fetch(url):
    while True:
        r = requests.get(url)
        if r.status_code == 429:
            reset = int(r.headers.get("X-RateLimit-Reset", 60))
            time.sleep(reset)
            continue
        return r

Concurrency-aware rate limiting:

• Multiple workers concurrently — global rate limit share করা দরকার।
• Token bucket algorithm — Redis-backed।
• Library: aiolimiter, ratelimit।

Common mistakes:

• No retry → single failure pipeline break।
• Infinite retry → DoS your own provider।
• No backoff → stampede।
• No jitter → coordinated retry storm।
• Retry POST blindly → duplicate operations।
• No timeout → hang forever।

Idempotency consideration:

• GET — safe to retry।
• PUT/DELETE — usually idempotent।
• POST — risk of duplicate। Use idempotency key।

Monitoring:

• Retry count metric।
• P99 latency।
• 429 count alert।
• Backoff time logged।

Real-world story:

• Bangladeshi e-commerce — competitor price scraping; aggressive concurrency → IP banned + lawsuit।
• News aggregator — exponential retry without max → infinite loop, $5000 cloud bill।
• Social media analytics — proper backoff → sustained 99% success over 2 years।

Best practice template:

1. Read API docs first।
2. Implement exponential backoff with jitter।
3. Cap max retries (5-10)।
4. Distinguish retryable (5xx, 429) vs not (4xx)।
5. Log every retry — debug-এ critical।
6. Circuit breaker — prolonged failure auto-stop।
7. Test with chaos — inject failures।

মূল উপলব্ধি: Rate limit handling distributed system reliability-এর core skill। ১ ঘণ্টার extra effort — পরের ১ বছরের 3am alert avoid।

Scraping tool selection — task-এর nature-এর উপর নির্ভর।

(১) BeautifulSoup + requests:

• Strength: simple, beginner-friendly, no setup।
• Weakness: static HTML only; JavaScript-rendered content miss।
• Use: small projects, well-structured static sites, learning।
• Speed: single-threaded, ~১০ pages/min।

(২) Scrapy:

• Strength: framework, async, built-in retry/throttle/pipeline।
• Weakness: learning curve; overkill small task-এ।
• Use: large crawl, structured pipeline, production।
• Speed: async, ~১০০-১০০০ pages/min।
• Features: spider, item, pipeline, middleware, autothrottle।

(৩) Playwright/Selenium:

• Strength: real browser, JavaScript-rendered, anti-bot bypass।
• Weakness: slow (full page render), resource-heavy।
• Use: SPA, login flows, screenshot, dynamic content।
• Speed: ~৫-২০ pages/min।

Modern hybrid — Scrapy + Playwright:

scrapy-playwright integration:
- Scrapy framework + Playwright for JS pages
- Selectively render — speed maximize

Decision tree:

• Static site, <100 pages → BeautifulSoup।
• Static site, >1000 pages → Scrapy।
• JS-heavy, login-required → Playwright।
• Both static + dynamic, large scale → Scrapy + Playwright।

Other tools to know:

• lxml: faster than BeautifulSoup for big HTML।
• requests-html: JS render via Pyppeteer।
• parsel: Scrapy-এর selector independent।
• httpx: requests-এর async successor।
• cloudscraper: Cloudflare bypass।

Anti-scraping measures (commonly faced):

1. User-Agent block — rotate UA।
2. IP rate limit — proxy rotation।
3. JavaScript challenge — Playwright।
4. CAPTCHA — solving service (ethical concern)।
5. Honeypot link — hidden, bot trap।
6. Session/cookie validation — maintain session।
7. Behavioral analysis — random delays, mouse movement।

Bangladesh examples:

• News scraping — most static, BeautifulSoup যথেষ্ট।
• E-commerce product list — JavaScript-heavy, Playwright।
• Job portal — login required, Playwright।
• Government PDF — pdfplumber + requests।

Maintenance reality:

• Scraper average lifetime ৩-৬ মাস — sites HTML change করে।
• Brittle — production-এ monitoring + alert essential।
• API থাকলে scraper-এর choice নেই — ১০x kemo lifetime।

মূল উপলব্ধি: "One tool fits all" নয়। Task scope, JS dependency, scale — বিচার করে select। Senior data engineer-রা সব tool-এ basic familiar — situation-এ pick।

একটি real-world data engineering scenario — multiple source orchestration।

Source-by-source plan:

(ক) Weather — OpenWeather API:

• ৬৪ district-এর coordinate list-এ daily call।
• Free tier: 60 calls/min — যথেষ্ট।
• Schedule: প্রতিদিন সকাল ৬টায়।
• Storage: weather_daily(district, date, temp_c, humidity, rainfall_mm)।

(খ) USD-BDT rate — Bangladesh Bank API / fallback:

• Primary: Bangladesh Bank-এর daily reference rate page (scrape if no API)।
• Fallback: ExchangeRate-API (free tier)।
• Schedule: প্রতিদিন সকাল ১১টা (BB publish-এর পর)।
• Storage: fx_daily(date, base_ccy, target_ccy, rate, source)।

(গ) E-commerce price — Scraping (with caution):

• robots.txt check; ToS read।
• Public product listings, no login।
• Scrapy + Playwright (JS-rendered)।
• Schedule: প্রতিদিন রাত ২টা (off-peak)।
• Identifiable User-Agent, 5-second delay।
• Storage: product_price(date, product_id, name, price_bdt, in_stock)।

Architecture:

┌─────────────┐    ┌────────────────┐    ┌─────────────┐
│ OpenWeather │ ──→│  Airflow DAG   │ ──→│  PostgreSQL │
│  (API)      │    │  daily 6am     │    │   weather   │
└─────────────┘    └────────────────┘    └─────────────┘

┌─────────────┐    ┌────────────────┐    ┌─────────────┐
│  BB Page    │ ──→│  Airflow DAG   │ ──→│  PostgreSQL │
│  (scrape)   │    │  daily 11am    │    │     fx      │
└─────────────┘    └────────────────┘    └─────────────┘

┌─────────────┐    ┌────────────────┐    ┌─────────────┐
│ E-commerce  │ ──→│  Scrapy spider │ ──→│  PostgreSQL │
│   (HTML)    │    │  daily 2am     │    │  product    │
└─────────────┘    └────────────────┘    └─────────────┘
                          ↓
                    ┌──────────────┐
                    │ dbt model    │
                    │ unified view │
                    └──────────────┘

Orchestration tool — Airflow DAG:

from airflow import DAG
from airflow.operators.python import PythonOperator

with DAG('data_acquisition', schedule='0 6 * * *') as dag:
    weather  = PythonOperator(task_id='weather',  python_callable=fetch_weather)
    fx_rate  = PythonOperator(task_id='fx',       python_callable=fetch_fx)
    products = PythonOperator(task_id='products', python_callable=run_scrapy)
    [weather, fx_rate, products]  # parallel

Error handling per source:

• Weather API: retry 3x; if persistent fail, alert ও skip district।
• FX: if BB unavailable, fallback API; if both fail, use yesterday + flag।
• Scraping: per-product retry; partial failure OK; site structure change → schema alert।

Data quality checks:

• Weather temp range (-5 to 50°C) — outlier flag।
• FX rate change > 5% day-over-day — alert (BB peg history)।
• Product count drop > 30% — site change suspect।
• Schema validation — column count, type।

Storage strategy:

• Raw layer: JSON/HTML as received — re-parse possible।
• Staging: parsed, validated।
• Mart: business-ready, joined।

Security:

• API keys → secret manager (AWS SSM, Vault)।
• Database creds → encrypted।
• Logging — exclude sensitive headers।

Observability:

• Airflow UI — DAG status।
• Slack alert — task failure।
• Grafana dashboard — row counts trend।
• Sentry — exception tracking।

Future scaling:

• ৬৪ → ৪০০ upazila — API call rate plan।
• Hourly → real-time streaming (Kafka)।
• Multiple e-commerce — distributed scraping।

মূল উপলব্ধি: Real data pipeline 80% engineering, 20% transformation। Source diversity manage করতে orchestration + monitoring + error handling — তিনটি equally important। SQL-pandas-API-scrape সব skill মিলে modern data engineer।