API Reference
Primary imports:
from newsfetch import (
fetch,
fetch_many,
fetch_iter,
fetch_results,
iter_results,
extract,
discover,
fetch_async,
fetch_many_async,
fetch_results_async,
NewsFetcher,
Config,
Article,
FetchResult,
Confidence,
LowConfidenceExtractionError,
)
fetch(url, **options) -> Article
Fetch and extract a single article.
| Parameter | Type | Description |
|---|---|---|
url |
str |
Article URL |
html |
str \| bytes \| None |
Skip network; extract from provided HTML |
debug |
bool |
Attach article.trace with candidates |
proxy |
str \| dict \| None |
Single proxy |
proxies |
str \| dict \| list \| None |
Proxy or rotating pool |
strict |
bool |
Raise if confidence below threshold |
min_confidence |
float |
Overall / field gate (with strict) |
**config_kwargs |
Any Config field |
Raises FetchError (including responses over max_bytes and robots.txt refusals), ExtractionError, LowConfidenceExtractionError.
extract(html, url=..., **options) -> Article
Same as fetch(..., html=html) — no network. Preferred for tests, caches, and custom HTTP clients.
fetch_many(urls, **options) -> list[Article | None]
Concurrent batch fetch. Returns a list the same length/order as urls; failures are None.
Notable options: max_workers, proxy / proxies, request_delay, on_progress(done, total, url, article).
fetch_iter(urls, **options) -> Iterator[tuple[str, Article | None]]
Stream (url, article_or_none) as work completes (unordered). Better for large jobs than materializing the full list. Stops submitting work when the consumer stops iterating early.
fetch_results(urls, **options) -> list[FetchResult]
Like fetch_many, but each item is a FetchResult carrying either the article or the exception, so you can see why a URL failed. Same order as urls. on_progress(done, total, url, result) receives the FetchResult.
iter_results(urls, **options) -> Iterator[FetchResult]
Streaming, unordered counterpart of fetch_results.
FetchResult
Outcome of fetching one URL. New in 1.1.0.
| Member | Description |
|---|---|
url |
Requested URL |
article |
Article \| None on success |
error |
Exception \| None on failure |
ok |
True when there is an article and no error |
status |
HTTP status for FetchError failures, when known |
unwrap() |
Return the article, or raise the captured error |
error_dict() |
{"type", "message", "status"} or None |
to_dict() |
{"url", "ok", "article", "error"} — JSON-ready |
discover(site_url, *, limit=50, client=None) -> list[dict]
Discover article URLs via robots.txt sitemaps, homepage feed autodiscovery, and common sitemap paths.
Each item: {"url", "title", "date"} (title/date may be None).
Pass client=HttpClient(...) to reuse a configured client (user agent, proxies, retries, size limits). XML is parsed with entity expansion, DTD loading and network access disabled.
fetch_async / fetch_many_async / fetch_results_async
Async counterparts. Require pip install news-fetch[async].
article = await fetch_async(url)
articles = await fetch_many_async(urls, max_concurrency=50, proxies=PROXIES)
results = await fetch_results_async(urls, max_concurrency=50) # list[FetchResult]
fetch_many_async runs on asyncio.TaskGroup.
NewsFetcher
from newsfetch import NewsFetcher, Config
with NewsFetcher(Config(cache=True, max_workers=16)) as fetcher:
article = fetcher.fetch(url)
results = fetcher.fetch_many(urls)
for r in fetcher.fetch_results(urls):
...
for u, a in fetcher.iter_fetch(urls):
...
fetcher.register_strategy(my_strategy)
| Method | Description |
|---|---|
fetch(url, html=None) |
Single article |
fetch_many(urls, max_workers=None, on_progress=None) |
Batch |
iter_fetch(urls, max_workers=None) |
Streaming batch |
fetch_result(url, html=None) |
Single URL → FetchResult |
fetch_results(urls, max_workers=None, on_progress=None) |
Batch → list[FetchResult] |
iter_results(urls, max_workers=None) |
Streaming batch → FetchResult |
discover(site_url, limit=50) |
Discovery (reuses the fetcher's client) |
register_strategy(strategy) |
Plugin hook |
close() |
Close HTTP session / cache |
Config
| Field | Default | Description |
|---|---|---|
timeout |
15.0 |
Request timeout (seconds) |
max_redirects |
10 |
Max redirects |
user_agent |
news-fetch/1.1 (...) |
User-Agent (also used for robots.txt and discovery) |
headers |
{} |
Extra headers |
verify_ssl |
True |
TLS verify |
debug |
False |
Extraction trace |
max_workers |
8 |
Bulk concurrency |
strip_tracking_params |
True |
Strip utm_/gclid/… from canonical |
min_text_length |
50 |
Soft content length note |
include_html |
False |
Fill article.html with cleaned body HTML |
proxy / proxies |
None |
Proxy / pool |
rotate_proxies |
True |
Round-robin pool |
request_delay |
0.0 |
Sleep before each request |
retries |
2 |
HTTP retries (incl. 429 Retry-After); 2 means 3 attempts total |
max_bytes |
10 MiB |
Max decompressed response size; None / 0 = unlimited |
strict |
False |
Confidence gate |
min_confidence |
0.0 |
Threshold when gating |
min_content_confidence |
None |
Field-specific gate |
min_title_confidence |
None |
Field-specific gate |
require_article_page |
False |
Require is_article |
respect_robots |
False |
Honor robots.txt (RFC 9309; unreachable = disallowed) |
cache |
False |
SQLite HTML cache |
cache_path |
.news-fetch-cache.sqlite |
Cache file |
cache_ttl |
86400 |
Cache TTL (seconds) |
render |
False |
Always use Playwright |
browser_fallback |
False |
Playwright if confidence low |
render_min_confidence |
0.55 |
Fallback trigger |
Article
| Attribute | Type | Description |
|---|---|---|
url |
str |
Request / final URL |
canonical_url |
str \| None |
Canonical URL |
title |
str \| None |
Headline |
description |
str \| None |
Meta / structured description |
text |
str \| None |
Article body |
markdown |
str \| None |
Body as Markdown (always filled when there is body text) |
html |
str \| None |
Cleaned body HTML (only with include_html=True) |
authors |
list[str] |
Authors |
published_at |
datetime \| None |
Publication time (tz-aware when possible); None if year, month or day is missing |
modified_at |
datetime \| None |
Modified time |
publisher |
str \| None |
Publisher / site name |
language |
str \| None |
BCP 47 primary subtag (en, fil, yue) |
image |
str \| None |
Primary image URL |
images |
list[str] |
Image list |
keywords |
list[str] |
Keywords |
section |
str \| None |
Section / category |
summary |
str \| None |
Lead-sentence summary |
word_count |
int |
Body word count |
reading_time_minutes |
int |
~200 wpm |
page_type |
str |
article, homepage, category, … |
is_article |
bool |
Page looks like an article |
sources |
dict[str, str] |
Field → source label |
confidence |
Confidence |
Field + overall scores |
extraction |
ExtractionReport |
Per-field provenance |
metadata |
dict |
Extra metadata |
trace |
ExtractionTrace \| None |
Debug candidates (debug=True) |
Convenience
title_source,date_source,content_source,image_source,authors_sourcetitle_confidence,content_confidence,date_confidence,author_confidence,image_confidenceto_dict(),to_json(indent=None)
Confidence
Floats in 0.0–1.0: title, content, date, authors, image, publisher, description, page_type, overall.
confidence.below(threshold, fields=[...]) → list of failing field names.
Exceptions
| Exception | When |
|---|---|
NewsFetchError |
Base |
FetchError |
HTTP / transport (url, status), oversized response, robots.txt refusal |
ParseError |
HTML parse failure |
ExtractionError |
No usable title/text |
LowConfidenceExtractionError |
Strict / threshold gate (failed_fields, confidence, article) |
DiscoveryError |
Discovery failure |
Plugins
from newsfetch import ExtractionStrategy, CallableStrategy
from newsfetch.strategies.base import Candidate
A strategy implements name and extract(document) -> dict[str, list[Candidate]].
Candidate(value, source, score, raw=None) feeds the evidence ranker.
Legacy
| Symbol | Status |
|---|---|
newsfetch.news.Newspaper |
Deprecated shim over NewsFetcher |
newsfetch.discovery.NewsSiteURLExtractor |
Compat alias for discovery |
See also: Confidence & evidence · CLI