API Reference

Primary imports:

from newsfetch import (
    fetch,
    fetch_many,
    fetch_iter,
    fetch_results,
    iter_results,
    extract,
    discover,
    fetch_async,
    fetch_many_async,
    fetch_results_async,
    NewsFetcher,
    Config,
    Article,
    FetchResult,
    Confidence,
    LowConfidenceExtractionError,
)

fetch(url, **options) -> Article

Fetch and extract a single article.

Parameter Type Description
url str Article URL
html str \| bytes \| None Skip network; extract from provided HTML
debug bool Attach article.trace with candidates
proxy str \| dict \| None Single proxy
proxies str \| dict \| list \| None Proxy or rotating pool
strict bool Raise if confidence below threshold
min_confidence float Overall / field gate (with strict)
**config_kwargs Any Config field

Raises FetchError (including responses over max_bytes and robots.txt refusals), ExtractionError, LowConfidenceExtractionError.


extract(html, url=..., **options) -> Article

Same as fetch(..., html=html) — no network. Preferred for tests, caches, and custom HTTP clients.


fetch_many(urls, **options) -> list[Article | None]

Concurrent batch fetch. Returns a list the same length/order as urls; failures are None.

Notable options: max_workers, proxy / proxies, request_delay, on_progress(done, total, url, article).


fetch_iter(urls, **options) -> Iterator[tuple[str, Article | None]]

Stream (url, article_or_none) as work completes (unordered). Better for large jobs than materializing the full list. Stops submitting work when the consumer stops iterating early.


fetch_results(urls, **options) -> list[FetchResult]

Like fetch_many, but each item is a FetchResult carrying either the article or the exception, so you can see why a URL failed. Same order as urls. on_progress(done, total, url, result) receives the FetchResult.


iter_results(urls, **options) -> Iterator[FetchResult]

Streaming, unordered counterpart of fetch_results.


FetchResult

Outcome of fetching one URL. New in 1.1.0.

Member Description
url Requested URL
article Article \| None on success
error Exception \| None on failure
ok True when there is an article and no error
status HTTP status for FetchError failures, when known
unwrap() Return the article, or raise the captured error
error_dict() {"type", "message", "status"} or None
to_dict() {"url", "ok", "article", "error"} — JSON-ready

discover(site_url, *, limit=50, client=None) -> list[dict]

Discover article URLs via robots.txt sitemaps, homepage feed autodiscovery, and common sitemap paths.

Each item: {"url", "title", "date"} (title/date may be None).

Pass client=HttpClient(...) to reuse a configured client (user agent, proxies, retries, size limits). XML is parsed with entity expansion, DTD loading and network access disabled.


fetch_async / fetch_many_async / fetch_results_async

Async counterparts. Require pip install news-fetch[async].

article = await fetch_async(url)
articles = await fetch_many_async(urls, max_concurrency=50, proxies=PROXIES)
results = await fetch_results_async(urls, max_concurrency=50)   # list[FetchResult]

fetch_many_async runs on asyncio.TaskGroup.


NewsFetcher

from newsfetch import NewsFetcher, Config

with NewsFetcher(Config(cache=True, max_workers=16)) as fetcher:
    article = fetcher.fetch(url)
    results = fetcher.fetch_many(urls)
    for r in fetcher.fetch_results(urls):
        ...
    for u, a in fetcher.iter_fetch(urls):
        ...
    fetcher.register_strategy(my_strategy)
Method Description
fetch(url, html=None) Single article
fetch_many(urls, max_workers=None, on_progress=None) Batch
iter_fetch(urls, max_workers=None) Streaming batch
fetch_result(url, html=None) Single URL → FetchResult
fetch_results(urls, max_workers=None, on_progress=None) Batch → list[FetchResult]
iter_results(urls, max_workers=None) Streaming batch → FetchResult
discover(site_url, limit=50) Discovery (reuses the fetcher's client)
register_strategy(strategy) Plugin hook
close() Close HTTP session / cache

Config

Field Default Description
timeout 15.0 Request timeout (seconds)
max_redirects 10 Max redirects
user_agent news-fetch/1.1 (...) User-Agent (also used for robots.txt and discovery)
headers {} Extra headers
verify_ssl True TLS verify
debug False Extraction trace
max_workers 8 Bulk concurrency
strip_tracking_params True Strip utm_/gclid/… from canonical
min_text_length 50 Soft content length note
include_html False Fill article.html with cleaned body HTML
proxy / proxies None Proxy / pool
rotate_proxies True Round-robin pool
request_delay 0.0 Sleep before each request
retries 2 HTTP retries (incl. 429 Retry-After); 2 means 3 attempts total
max_bytes 10 MiB Max decompressed response size; None / 0 = unlimited
strict False Confidence gate
min_confidence 0.0 Threshold when gating
min_content_confidence None Field-specific gate
min_title_confidence None Field-specific gate
require_article_page False Require is_article
respect_robots False Honor robots.txt (RFC 9309; unreachable = disallowed)
cache False SQLite HTML cache
cache_path .news-fetch-cache.sqlite Cache file
cache_ttl 86400 Cache TTL (seconds)
render False Always use Playwright
browser_fallback False Playwright if confidence low
render_min_confidence 0.55 Fallback trigger

Article

Attribute Type Description
url str Request / final URL
canonical_url str \| None Canonical URL
title str \| None Headline
description str \| None Meta / structured description
text str \| None Article body
markdown str \| None Body as Markdown (always filled when there is body text)
html str \| None Cleaned body HTML (only with include_html=True)
authors list[str] Authors
published_at datetime \| None Publication time (tz-aware when possible); None if year, month or day is missing
modified_at datetime \| None Modified time
publisher str \| None Publisher / site name
language str \| None BCP 47 primary subtag (en, fil, yue)
image str \| None Primary image URL
images list[str] Image list
keywords list[str] Keywords
section str \| None Section / category
summary str \| None Lead-sentence summary
word_count int Body word count
reading_time_minutes int ~200 wpm
page_type str article, homepage, category, …
is_article bool Page looks like an article
sources dict[str, str] Field → source label
confidence Confidence Field + overall scores
extraction ExtractionReport Per-field provenance
metadata dict Extra metadata
trace ExtractionTrace \| None Debug candidates (debug=True)

Convenience

  • title_source, date_source, content_source, image_source, authors_source
  • title_confidence, content_confidence, date_confidence, author_confidence, image_confidence
  • to_dict(), to_json(indent=None)

Confidence

Floats in 0.0–1.0: title, content, date, authors, image, publisher, description, page_type, overall.

confidence.below(threshold, fields=[...]) → list of failing field names.


Exceptions

Exception When
NewsFetchError Base
FetchError HTTP / transport (url, status), oversized response, robots.txt refusal
ParseError HTML parse failure
ExtractionError No usable title/text
LowConfidenceExtractionError Strict / threshold gate (failed_fields, confidence, article)
DiscoveryError Discovery failure

Plugins

from newsfetch import ExtractionStrategy, CallableStrategy
from newsfetch.strategies.base import Candidate

A strategy implements name and extract(document) -> dict[str, list[Candidate]].

Candidate(value, source, score, raw=None) feeds the evidence ranker.


Legacy

Symbol Status
newsfetch.news.Newspaper Deprecated shim over NewsFetcher
newsfetch.discovery.NewsSiteURLExtractor Compat alias for discovery

See also: Confidence & evidence · CLI