Usage

Single article

from newsfetch import fetch

article = fetch("https://www.thehindu.com/news/national/...")
print(article.title)
print(article.text)
print(article.authors)
print(article.published_at)
print(article.image)
print(article.publisher)
print(article.confidence.overall)
print(article.content_source)

Markdown and HTML output

article.markdown is always filled. It is rendered from the article body's DOM node, so it keeps headings, nested lists, links, images with captions, blockquotes, fenced code and GFM tables. When the body only exists as text (for example JSON-LD articleBody with no matching node) it falls back to plain paragraphs. article.sources["markdown"] tells you which path was used.

article = fetch(url)
print(article.markdown)
# ## Key points
#
# Read the [full **report**](https://example.com/report.pdf) …
#
# - First point
# - Second point

article = fetch(url, include_html=True)
print(article.html)   # cleaned body HTML: no scripts, nav, share widgets or tracking attributes

article.html is None unless include_html=True.

Strict mode (production pipelines)

from newsfetch import fetch, LowConfidenceExtractionError

try:
    article = fetch(url, strict=True)          # default ~0.6 title/content
    # article = fetch(url, min_confidence=0.8)
except LowConfidenceExtractionError as e:
    print(e.failed_fields, e.confidence.overall)

Offline extraction (no network)

Transport stays separate from extraction — pass HTML you already fetched:

from newsfetch import extract

html = open("page.html", "rb").read()
article = extract(html, url="https://example.com/story")

Configurable client

from newsfetch import NewsFetcher, Config

fetcher = NewsFetcher(Config(timeout=20, debug=True, cache=True))
article = fetcher.fetch(url)
print(article.trace.candidates[:5])  # when debug=True
fetcher.close()

Bulk scraping

from newsfetch import fetch_many, fetch_iter

results = fetch_many(
    urls,
    max_workers=20,
    proxies=[
        "http://user:pass@proxy1.example:8080",
        "http://user:pass@proxy2.example:8080",
    ],
    request_delay=0.05,
    on_progress=lambda done, total, url, article: print(f"{done}/{total}"),
)
# failures → None (order preserved)

for url, article in fetch_iter(urls, max_workers=16):
    if article:
        print(article.title)

Per-URL errors with FetchResult

fetch_many / fetch_iter return None for failed URLs. To know why a URL failed, use fetch_results / iter_results, which return a FetchResult per URL:

from newsfetch import fetch_results, iter_results

for result in fetch_results(urls, max_workers=20):
    if result.ok:
        print(result.article.title)
    else:
        print(result.url, type(result.error).__name__, result.status, result.error)

# streaming, unordered:
for result in iter_results(urls, max_workers=16):
    ...

# result.unwrap() returns the article or raises the captured error
# result.to_dict() is JSON-ready

NewsFetcher has the same methods (fetch_result, fetch_results, iter_results) when you want to reuse one configured client. The existing fetch_many / fetch_iter / fetch_many_async are unchanged.

Proxies

Single proxy:

article = fetch(url, proxy="http://user:pass@proxy.example:8080")

Discovery (RSS / sitemaps)

from newsfetch import discover

for item in discover("https://www.bbc.com", limit=10):
    print(item["url"], item.get("title"), item.get("date"))

NewsFetcher.discover reuses the fetcher's HTTP client, so its user agent, proxies, retries and size limits apply to robots.txt, feeds and sitemaps too. Sitemap/feed XML is parsed with entity expansion, DTD loading and network access disabled.

Legacy alias still works: from newsfetch.discovery import NewsSiteURLExtractor.

Async

from newsfetch import fetch_async, fetch_many_async, fetch_results_async

article = await fetch_async(url, proxy="http://proxy.example:8080")
articles = await fetch_many_async(urls, max_concurrency=50, proxies=PROXIES)
results = await fetch_results_async(urls, max_concurrency=50)   # list[FetchResult]

Requires pip install news-fetch[async].

Cache, robots, browser, size limits

from newsfetch import fetch

fetch(url, cache=True, respect_robots=True)
fetch(url, render=True)                 # needs news-fetch[browser]
fetch(url, browser_fallback=True)       # retry with Playwright if confidence is low
fetch(url, max_bytes=2 * 1024 * 1024)   # cap response size (default 10 MiB, None / 0 = no limit)

robots.txt — respect_robots=True follows RFC 9309: * / $ wildcards, longest match wins (allow wins ties), product-token group matching, percent-encoding normalization and a 500 KiB file cap.

robots.txt response Result
2xx Rules applied
4xx No restrictions
5xx, 429, network error Unreachable → fetch refused (changed in 1.1.0)

The error message states the reason.

Size limit — response bodies are streamed and capped after decompression (sync and async), so oversized pages and compressed "bombs" raise FetchError.

Custom strategy plugin

from newsfetch import NewsFetcher, CallableStrategy
from newsfetch.strategies.base import Candidate

def my_strategy(doc):
    return {"title": [Candidate("Custom Title", "plugin.custom", 0.99)]}

fetcher = NewsFetcher()
fetcher.register_strategy(CallableStrategy("custom", my_strategy))
article = fetcher.fetch(url)

CLI

news-fetch https://example.com/article
news-fetch get URL --json
news-fetch get URL --markdown
news-fetch batch urls.txt -o articles.jsonl --workers 20
news-fetch discover https://www.theguardian.com --limit 10

See CLI for details.

Legacy API (deprecated)

from newsfetch.news import Newspaper  # emits DeprecationWarning

news = Newspaper(url="https://...")
print(news.headline, news.get_dict)

Prefer fetch() / NewsFetcher for new code.