Usage
Single article
from newsfetch import fetch
article = fetch("https://www.thehindu.com/news/national/...")
print(article.title)
print(article.text)
print(article.authors)
print(article.published_at)
print(article.image)
print(article.publisher)
print(article.confidence.overall)
print(article.content_source)
Markdown and HTML output
article.markdown is always filled. It is rendered from the article body's DOM node, so it keeps headings, nested lists, links, images with captions, blockquotes, fenced code and GFM tables. When the body only exists as text (for example JSON-LD articleBody with no matching node) it falls back to plain paragraphs. article.sources["markdown"] tells you which path was used.
article = fetch(url)
print(article.markdown)
# ## Key points
#
# Read the [full **report**](https://example.com/report.pdf) …
#
# - First point
# - Second point
article = fetch(url, include_html=True)
print(article.html) # cleaned body HTML: no scripts, nav, share widgets or tracking attributes
article.html is None unless include_html=True.
Strict mode (production pipelines)
from newsfetch import fetch, LowConfidenceExtractionError
try:
article = fetch(url, strict=True) # default ~0.6 title/content
# article = fetch(url, min_confidence=0.8)
except LowConfidenceExtractionError as e:
print(e.failed_fields, e.confidence.overall)
Offline extraction (no network)
Transport stays separate from extraction — pass HTML you already fetched:
from newsfetch import extract
html = open("page.html", "rb").read()
article = extract(html, url="https://example.com/story")
Configurable client
from newsfetch import NewsFetcher, Config
fetcher = NewsFetcher(Config(timeout=20, debug=True, cache=True))
article = fetcher.fetch(url)
print(article.trace.candidates[:5]) # when debug=True
fetcher.close()
Bulk scraping
from newsfetch import fetch_many, fetch_iter
results = fetch_many(
urls,
max_workers=20,
proxies=[
"http://user:pass@proxy1.example:8080",
"http://user:pass@proxy2.example:8080",
],
request_delay=0.05,
on_progress=lambda done, total, url, article: print(f"{done}/{total}"),
)
# failures → None (order preserved)
for url, article in fetch_iter(urls, max_workers=16):
if article:
print(article.title)
Per-URL errors with FetchResult
fetch_many / fetch_iter return None for failed URLs. To know why a URL failed, use fetch_results / iter_results, which return a FetchResult per URL:
from newsfetch import fetch_results, iter_results
for result in fetch_results(urls, max_workers=20):
if result.ok:
print(result.article.title)
else:
print(result.url, type(result.error).__name__, result.status, result.error)
# streaming, unordered:
for result in iter_results(urls, max_workers=16):
...
# result.unwrap() returns the article or raises the captured error
# result.to_dict() is JSON-ready
NewsFetcher has the same methods (fetch_result, fetch_results, iter_results) when you want to reuse one configured client. The existing fetch_many / fetch_iter / fetch_many_async are unchanged.
Proxies
Single proxy:
article = fetch(url, proxy="http://user:pass@proxy.example:8080")
Discovery (RSS / sitemaps)
from newsfetch import discover
for item in discover("https://www.bbc.com", limit=10):
print(item["url"], item.get("title"), item.get("date"))
NewsFetcher.discover reuses the fetcher's HTTP client, so its user agent, proxies, retries and size limits apply to robots.txt, feeds and sitemaps too. Sitemap/feed XML is parsed with entity expansion, DTD loading and network access disabled.
Legacy alias still works: from newsfetch.discovery import NewsSiteURLExtractor.
Async
from newsfetch import fetch_async, fetch_many_async, fetch_results_async
article = await fetch_async(url, proxy="http://proxy.example:8080")
articles = await fetch_many_async(urls, max_concurrency=50, proxies=PROXIES)
results = await fetch_results_async(urls, max_concurrency=50) # list[FetchResult]
Requires pip install news-fetch[async].
Cache, robots, browser, size limits
from newsfetch import fetch
fetch(url, cache=True, respect_robots=True)
fetch(url, render=True) # needs news-fetch[browser]
fetch(url, browser_fallback=True) # retry with Playwright if confidence is low
fetch(url, max_bytes=2 * 1024 * 1024) # cap response size (default 10 MiB, None / 0 = no limit)
robots.txt — respect_robots=True follows RFC 9309: * / $ wildcards, longest match wins (allow wins ties), product-token group matching, percent-encoding normalization and a 500 KiB file cap.
| robots.txt response | Result |
|---|---|
| 2xx | Rules applied |
| 4xx | No restrictions |
| 5xx, 429, network error | Unreachable → fetch refused (changed in 1.1.0) |
The error message states the reason.
Size limit — response bodies are streamed and capped after decompression (sync and async), so oversized pages and compressed "bombs" raise FetchError.
Custom strategy plugin
from newsfetch import NewsFetcher, CallableStrategy
from newsfetch.strategies.base import Candidate
def my_strategy(doc):
return {"title": [Candidate("Custom Title", "plugin.custom", 0.99)]}
fetcher = NewsFetcher()
fetcher.register_strategy(CallableStrategy("custom", my_strategy))
article = fetcher.fetch(url)
CLI
news-fetch https://example.com/article
news-fetch get URL --json
news-fetch get URL --markdown
news-fetch batch urls.txt -o articles.jsonl --workers 20
news-fetch discover https://www.theguardian.com --limit 10
See CLI for details.
Legacy API (deprecated)
from newsfetch.news import Newspaper # emits DeprecationWarning
news = Newspaper(url="https://...")
print(news.headline, news.get_dict)
Prefer fetch() / NewsFetcher for new code.