API Reference

Primary imports:

from newsfetch import (
    fetch,
    fetch_many,
    fetch_iter,
    extract,
    discover,
    fetch_async,
    fetch_many_async,
    NewsFetcher,
    Config,
    Article,
    Confidence,
    LowConfidenceExtractionError,
)

`fetch(url, **options) -> Article`

Fetch and extract a single article.

Parameter	Type	Description
`url`	`str`	Article URL
`html`	`str \\| bytes \\| None`	Skip network; extract from provided HTML
`debug`	`bool`	Attach `article.trace` with candidates
`proxy`	`str \\| dict \\| None`	Single proxy
`proxies`	`str \\| dict \\| list \\| None`	Proxy or rotating pool
`strict`	`bool`	Raise if confidence below threshold
`min_confidence`	`float`	Overall / field gate (with `strict`)
`**config_kwargs`		Any `Config` field

Raises FetchError, ExtractionError, LowConfidenceExtractionError.

`extract(html, url=..., **options) -> Article`

Same as fetch(..., html=html) — no network. Preferred for tests, caches, and custom HTTP clients.

`fetch_many(urls, **options) -> list[Article | None]`

Concurrent batch fetch. Returns a list the same length/order as urls; failures are None.

Notable options: max_workers, proxy / proxies, request_delay, on_progress(done, total, url, article).

`fetch_iter(urls, **options) -> Iterator[tuple[str, Article | None]]`

Stream (url, article_or_none) as work completes (unordered). Better for large jobs than materializing the full list.

`discover(site_url, *, limit=50) -> list[dict]`

Discover article URLs via robots.txt sitemaps, homepage feed autodiscovery, and common sitemap paths.

Each item: {"url", "title", "date"} (title/date may be None).

`fetch_async` / `fetch_many_async`

Async counterparts. Require pip install news-fetch[async].

article = await fetch_async(url)
articles = await fetch_many_async(urls, max_concurrency=50, proxies=PROXIES)

`NewsFetcher`

from newsfetch import NewsFetcher, Config

with NewsFetcher(Config(cache=True, max_workers=16)) as fetcher:
    article = fetcher.fetch(url)
    results = fetcher.fetch_many(urls)
    for u, a in fetcher.iter_fetch(urls):
        ...
    fetcher.register_strategy(my_strategy)

Method	Description
`fetch(url, html=None)`	Single article
`fetch_many(urls, max_workers=None, on_progress=None)`	Batch
`iter_fetch(urls, max_workers=None)`	Streaming batch
`discover(site_url, limit=50)`	Discovery
`register_strategy(strategy)`	Plugin hook
`close()`	Close HTTP session / cache

`Config`

Field	Default	Description
`timeout`	`15.0`	Request timeout (seconds)
`max_redirects`	`10`	Max redirects
`user_agent`	`news-fetch/1.0 (...)`	User-Agent
`headers`	`{}`	Extra headers
`verify_ssl`	`True`	TLS verify
`debug`	`False`	Extraction trace
`max_workers`	`8`	Bulk concurrency
`strip_tracking_params`	`True`	Strip utm_/gclid/… from canonical
`min_text_length`	`50`	Soft content length note
`proxy` / `proxies`	`None`	Proxy / pool
`rotate_proxies`	`True`	Round-robin pool
`request_delay`	`0.0`	Sleep before each request
`retries`	`2`	HTTP retries (incl. 429 Retry-After)
`strict`	`False`	Confidence gate
`min_confidence`	`0.0`	Threshold when gating
`min_content_confidence`	`None`	Field-specific gate
`min_title_confidence`	`None`	Field-specific gate
`require_article_page`	`False`	Require `is_article`
`respect_robots`	`False`	Honor robots.txt
`cache`	`False`	SQLite HTML cache
`cache_path`	`.news-fetch-cache.sqlite`	Cache file
`cache_ttl`	`86400`	Cache TTL (seconds)
`render`	`False`	Always use Playwright
`browser_fallback`	`False`	Playwright if confidence low
`render_min_confidence`	`0.55`	Fallback trigger

`Article`

Attribute	Type	Description
`url`	`str`	Request / final URL
`canonical_url`	`str \\| None`	Canonical URL
`title`	`str \\| None`	Headline
`description`	`str \\| None`	Meta / structured description
`text`	`str \\| None`	Article body
`authors`	`list[str]`	Authors
`published_at`	`datetime \\| None`	Publication time (tz-aware when possible)
`modified_at`	`datetime \\| None`	Modified time
`publisher`	`str \\| None`	Publisher / site name
`language`	`str \\| None`	Language code
`image`	`str \\| None`	Primary image URL
`images`	`list[str]`	Image list
`keywords`	`list[str]`	Keywords
`section`	`str \\| None`	Section / category
`summary`	`str \\| None`	Lead-sentence summary
`word_count`	`int`	Body word count
`reading_time_minutes`	`int`	~200 wpm
`page_type`	`str`	`article`, `homepage`, `category`, …
`is_article`	`bool`	Page looks like an article
`sources`	`dict[str, str]`	Field → source label
`confidence`	`Confidence`	Field + overall scores
`extraction`	`ExtractionReport`	Per-field provenance
`metadata`	`dict`	Extra metadata
`trace`	`ExtractionTrace \\| None`	Debug candidates (`debug=True`)

Convenience

title_source, date_source, content_source, image_source, authors_source
title_confidence, content_confidence, date_confidence, author_confidence, image_confidence
to_dict(), to_json(indent=None)

`Confidence`

Floats in 0.0–1.0: title, content, date, authors, image, publisher, description, page_type, overall.

confidence.below(threshold, fields=[...]) → list of failing field names.

Exceptions

Exception	When
`NewsFetchError`	Base
`FetchError`	HTTP / transport (`url`, `status`)
`ParseError`	HTML parse failure
`ExtractionError`	No usable title/text
`LowConfidenceExtractionError`	Strict / threshold gate (`failed_fields`, `confidence`, `article`)
`DiscoveryError`	Discovery failure

Plugins

from newsfetch import ExtractionStrategy, CallableStrategy
from newsfetch.strategies.base import Candidate

A strategy implements name and extract(document) -> dict[str, list[Candidate]].

Candidate(value, source, score, raw=None) feeds the evidence ranker.

Legacy

Symbol	Status
`newsfetch.news.Newspaper`	Deprecated shim over `NewsFetcher`
`newsfetch.discovery.NewsSiteURLExtractor`	Compat alias for discovery

API Reference

fetch(url, **options) -> Article

extract(html, url=..., **options) -> Article

fetch_many(urls, **options) -> list[Article | None]

fetch_iter(urls, **options) -> Iterator[tuple[str, Article | None]]

discover(site_url, *, limit=50) -> list[dict]

fetch_async / fetch_many_async

NewsFetcher

Config

Article

Confidence