Headless Web Scraping

Extract from resistant pages (JS-rendered, Cloudflare, dynamic SPAs) via scrapling 3-tier fetcher + CSS extraction.

Use When

• JS rendering (SPA, React, Vue)
• Anti-bot (Cloudflare Turnstile, TLS fingerprint)
• Structured multi-element via CSS
• WebFetch / requests.get() empty or blocked
• Tabular/list/repeated DOM at scale

In

• Required: URL(s)
• Required: data to extract (CSS selectors, field names, target desc)
• Optional: fetcher tier override (default: auto)
• Optional: out format (default JSON; CSV, dict)
• Optional: rate limit sec (default 1)

Do

Step 1: Select tier

# Decision matrix:
# 1. Fetcher        — static HTML, no JS, no anti-bot (fastest)
# 2. StealthyFetcher — Cloudflare/Turnstile, TLS fingerprint checks
# 3. DynamicFetcher  — JS-rendered SPAs, click/scroll interactions

# Quick probe: try Fetcher first, escalate on failure
from scrapling import Fetcher

fetcher = Fetcher()
response = fetcher.get("https://example.com/target-page")

if response.status == 200 and response.get_all_text():
    print("Fetcher tier sufficient")
else:
    print("Escalate to StealthyFetcher or DynamicFetcher")

Signal	Recommended Tier
Static HTML, no protection	Fetcher
403/503, Cloudflare challenge page	StealthyFetcher
Page loads but content area is empty	DynamicFetcher
Need to click buttons or scroll	DynamicFetcher
altcha CAPTCHA present	None (cannot be automated)

→ 1 of 3 tiers. Modern sites → StealthyFetcher usual start.

If err: all 3 blocked → check altcha CAPTCHA (PoW, cannot bypass). Document limitation + manual extraction.

Step 2: Configure

from scrapling import Fetcher, StealthyFetcher, DynamicFetcher

# Tier 1: Fast HTTP with TLS fingerprint impersonation
fetcher = Fetcher()
fetcher.configure(
    timeout=30,
    retries=3,
    follow_redirects=True
)

# Tier 2: Headless Chromium with anti-detection
fetcher = StealthyFetcher()
fetcher.configure(
    headless=True,
    timeout=60,
    network_idle=True  # wait for all network requests to settle
)

# Tier 3: Full browser automation
fetcher = DynamicFetcher()
fetcher.configure(
    headless=True,
    timeout=90,
    network_idle=True,
    wait_selector="div.results"  # wait for specific element before extracting
)

→ Fetcher configured + ready. No err on init. Stealth/Dynamic → Chromium auto-managed first run.

If err:

• playwright / browser binary missing → python -m playwright install chromium
• configure() timeout → increase timeout or check network
• Import err → pip install scrapling

Step 3: Fetch + extract

# Fetch the page
response = fetcher.get("https://example.com/target-page")

# Single element extraction
title = response.find("h1.page-title")
if title:
    print(title.get_all_text())

# Multiple elements
items = response.find_all("div.result-item")
for item in items:
    name = item.find("span.name")
    price = item.find("span.price")
    print(f"{name.get_all_text()}: {price.get_all_text()}")

# Get attribute values
links = response.find_all("a.product-link")
urls = [link.get("href") for link in links]

# Get raw HTML content of an element
detail_html = response.find("div.description").html_content

API ref:

Method	Purpose
response.find("selector")	First matching element
response.find_all("selector")	All matching elements
element.get("attr")	Attribute value (href, src, data-*)
element.get_all_text()	All text content, recursively
element.html_content	Raw inner HTML

→ Extracted data matches visible content. Non-None elements, non-empty text on populated pages.

If err:

• find() → None → inspect response.html_content for actual HTML; selectors may differ
• Empty get_all_text() → shadow DOM / iframe → DynamicFetcher w/ wait_selector
• NO .css_first() → not scrapling API (other lib confusion)

Step 4: Handle failures + edge cases

import time

def scrape_with_fallback(url, selector):
    """Try each fetcher tier in order, with CAPTCHA detection."""
    tiers = [
        ("Fetcher", Fetcher),
        ("StealthyFetcher", StealthyFetcher),
        ("DynamicFetcher", DynamicFetcher),
    ]

    for tier_name, tier_class in tiers:
        fetcher = tier_class()
        fetcher.configure(headless=True, timeout=60)

        try:
            response = fetcher.get(url)
        except Exception as error:
            print(f"{tier_name} failed: {error}")
            continue

        # Detect CAPTCHA / challenge pages
        page_text = response.get_all_text().lower()
        if "altcha" in page_text or "proof of work" in page_text:
            print(f"altcha CAPTCHA detected -- cannot automate")
            return None

        if response.status == 403 or response.status == 503:
            print(f"{tier_name} blocked (HTTP {response.status}), escalating")
            continue

        result = response.find(selector)
        if result and result.get_all_text().strip():
            return result.get_all_text()

        print(f"{tier_name} returned empty content, escalating")

    print("All tiers exhausted. Manual extraction required.")
    return None

→ Returns text on success, None + diagnostic on fail. CAPTCHA detected + reported not retried.

If err:

• All 403 → site blocks all automation (WIPO, TMview, gov DBs). Document as manual access.
• Timeout → slow CDN → increase to 120s.
• Session/cookie errs → login required → add cookie handling / auth.

Step 5: Rate limit + ethical

import time
import urllib.robotparser

def check_robots_txt(base_url, target_path):
    """Check if scraping is allowed by robots.txt."""
    rp = urllib.robotparser.RobotFileParser()
    rp.set_url(f"{base_url}/robots.txt")
    rp.read()
    return rp.can_fetch("*", f"{base_url}{target_path}")

def scrape_urls(urls, selector, delay=1.0):
    """Scrape multiple URLs with rate limiting."""
    results = []
    fetcher = StealthyFetcher()
    fetcher.configure(headless=True, timeout=60)

    for url in urls:
        response = fetcher.get(url)
        data = response.find(selector)
        if data:
            results.append(data.get_all_text())

        time.sleep(delay)  # respect the server

    return results

Ethical checklist:

1. robots.txt first → respect Disallow
2. Min 1-sec delay
3. Descriptive User-Agent
4. No personal data w/o legal basis
5. Cache locally → avoid redundant reqs
6. 429 → stop immediately

→ Controlled rate. robots.txt checked pre-bulk. No 429.

If err:

• 429 → increase delay 3-5 sec, or stop + retry later
• robots.txt disallow → respect, do not override
• IP ban → stop immediately. If legit access (public, ToS-permit, robots-respect) must continue → see rotate-scraping-proxies for network-layer escalation

Check

• Correct tier (not over/under)
• configure() used (not deprecated constructor kwargs)
• Selectors match actual structure (verified vs source)
• .find() / .find_all() used (not .css_first())
• CAPTCHA detection (altcha reported, not retried)
• Rate limit for multi-URL
• robots.txt checked pre-bulk
• Extracted data non-empty + correct

Traps

• .css_first() instead .find(): scrapling uses .find()/.find_all(). .css_first() = diff lib → AttributeError.
• Start w/ DynamicFetcher: try Fetcher first. Dynamic 10-50× slower (full browser startup).
• Constructor kwargs: scrapling v0.4.x deprecated → always configure().
• Ignore altcha: no tier solves altcha PoW → detect early + fallback manual.
• No rate limit: even w/o 429 → IP ban / service degradation.
• Stable selectors: CSS changes frequently → validate before each campaign.

→

• rotate-scraping-proxies — network-layer escalation when client-side stealth exhausted
• use-graphql-api — GraphQL endpoint > scraping
• serialize-data-formats — JSON/CSV conversion
• deploy-searxng — self-hosted aggregator
• forage-solutions — broader info gathering

<!-- Keep under 500 lines. Extract large examples to references/EXAMPLES.md if needed. -->