Usage

Single article

from newsfetch import fetch

article = fetch("https://www.thehindu.com/news/national/...")
print(article.title)
print(article.text)
print(article.authors)
print(article.published_at)
print(article.image)
print(article.publisher)
print(article.confidence.overall)
print(article.content_source)

Strict mode (production pipelines)

from newsfetch import fetch, LowConfidenceExtractionError

try:
    article = fetch(url, strict=True)          # default ~0.6 title/content
    # article = fetch(url, min_confidence=0.8)
except LowConfidenceExtractionError as e:
    print(e.failed_fields, e.confidence.overall)

Offline extraction (no network)

Transport stays separate from extraction — pass HTML you already fetched:

from newsfetch import extract

html = open("page.html", "rb").read()
article = extract(html, url="https://example.com/story")

Configurable client

from newsfetch import NewsFetcher, Config

fetcher = NewsFetcher(Config(timeout=20, debug=True, cache=True))
article = fetcher.fetch(url)
print(article.trace.candidates[:5])  # when debug=True
fetcher.close()

Bulk scraping

from newsfetch import fetch_many, fetch_iter

results = fetch_many(
    urls,
    max_workers=20,
    proxies=[
        "http://user:pass@proxy1.example:8080",
        "http://user:pass@proxy2.example:8080",
    ],
    request_delay=0.05,
    on_progress=lambda done, total, url, article: print(f"{done}/{total}"),
)
# failures → None (order preserved)

for url, article in fetch_iter(urls, max_workers=16):
    if article:
        print(article.title)

Single proxy:

article = fetch(url, proxy="http://user:pass@proxy.example:8080")

Discovery (RSS / sitemaps)

from newsfetch import discover

for item in discover("https://www.bbc.com", limit=10):
    print(item["url"], item.get("title"), item.get("date"))

Legacy alias still works: from newsfetch.discovery import NewsSiteURLExtractor.

Async

from newsfetch import fetch_async, fetch_many_async

article = await fetch_async(url, proxy="http://proxy.example:8080")
articles = await fetch_many_async(urls, max_concurrency=50, proxies=PROXIES)

Requires pip install news-fetch[async].

Cache, robots, browser

from newsfetch import fetch

fetch(url, cache=True, respect_robots=True)
fetch(url, render=True)                 # needs news-fetch[browser]
fetch(url, browser_fallback=True)       # retry with Playwright if confidence is low

Custom strategy plugin

from newsfetch import NewsFetcher, CallableStrategy
from newsfetch.strategies.base import Candidate

def my_strategy(doc):
    return {"title": [Candidate("Custom Title", "plugin.custom", 0.99)]}

fetcher = NewsFetcher()
fetcher.register_strategy(CallableStrategy("custom", my_strategy))
article = fetcher.fetch(url)

CLI

news-fetch https://example.com/article
news-fetch get URL --json
news-fetch batch urls.txt -o articles.jsonl --workers 20
news-fetch discover https://www.theguardian.com --limit 10

See CLI for details.

Legacy API (deprecated)

from newsfetch.news import Newspaper  # emits DeprecationWarning

news = Newspaper(url="https://...")
print(news.headline, news.get_dict)

Prefer fetch() / NewsFetcher for new code.