Changelog
Documentation mirrors the library changelog. Full details: news-fetch CHANGELOG.
[1.1.0] - 2026-11-15
Upgrade notes
- Python 3.11 or newer is required.
- With
respect_robots=True, an unreachable robots.txt (5xx, 429, network error) now blocks the fetch. - Responses larger than 10 MiB (after decompression) raise
FetchError; raise or disablemax_bytesif needed. - Dates without a full year, month and day are no longer guessed;
published_atmay beNonewhere it used to hold a made-up date. - Language codes keep their full primary subtag (
fil,yue), not always two letters. - CLI
batcherror rows changed from"error": trueto an object. pip install news-fetch[dev]no longer exists; useuv sync(see Installation).
Added
Config.max_bytes(default 10 MiB): response bodies are streamed and capped after decompression (sync and async), protecting against oversized pages and gzip bombs.None/0disables it.- RFC 9309 robots.txt support:
*/$wildcards, longest-match precedence (allow wins ties), product-token group matching with merged groups, percent-encoding normalization, 500 KiB cap. NewRobotsRulesandRobotsCache.check()returning aRobotsDecision(allowed, reason). discover(..., client=HttpClient);NewsFetcher.discoverreuses the fetcher's client.FetchResult(url, article, error)with.ok,.status,.unwrap(),.to_dict(), returned by newfetch_results/iter_results,NewsFetcher.fetch_result/fetch_results/iter_resultsandfetch_results_async. Existingfetch_many/fetch_iter/fetch_many_asyncare unchanged.Article.markdown: article body as Markdown (headings, lists, links, images, blockquotes, code, GFM tables), always filled.sources["markdown"]says whether it came from the DOM node or plain text.Article.htmlis now filled with cleaned body HTML whenConfig(include_html=True).- CLI:
news-fetch get --markdown. HttpClient.get(..., max_bytes=, truncate=)per-call size overrides.- CI test workflow: Python 3.11–3.14 (+ macOS), lowest-dependency job, wheel smoke test, ruff; Dependabot for SHA-pinned actions.
Changed
- Python 3.11+ required (3.10 reached end of life in October 2026).
- Development uses uv with a committed
uv.lock; dev tools moved to a PEP 735[dependency-groups]table. Thedevinstall extra andrequirements*.txtare removed. Publishing builds withuv build. - CLI
batch: failed URLs are written as{"url": ..., "error": {"type", "message", "status"}}(was"error": true), aN ok, M failedsummary goes to stderr, and the exit code is 1 when no URL succeeded. fetch_many_asyncnow runs onasyncio.TaskGroup.- robots.txt (
respect_robots=True): 5xx, 429 and network errors now mean "unreachable" → fetch disallowed (previously allowed). 4xx still means no restrictions. Error messages state the reason. - robots.txt and discovery requests go through
HttpClient: configured user agent (no longerpython-requests), proxies, retries and size limits apply. - Sitemap/feed XML is parsed with entity expansion, DTD loading and network access disabled.
- User agent bumped to
news-fetch/1.1.
Fixed
<html lang>was never detected; language tags are reduced to the BCP 47 primary subtag (filno longer becomesfi).DiskCacheis now thread-safe (fetch_many(cache=True)raised sqliteInterfaceError).- Retries no longer multiply (
retries=2made 9 connection attempts; now 3). - Dates missing a year, month or day are rejected instead of being completed from today.
- Keyword extraction works for non-Latin scripts.
fetch_iter/iter_fetchstop fetching when the consumer stops early.- A missing
cssselectinstall now raises instead of silently matching nothing.
Removed
- Unused internals:
heuristic.iter_images_near,Extractor.extract_from_nodes.
[1.0.0]
Added
- Production-stable independent extraction engine.
- Evidence → rank → confidence (
article.confidence,article.extraction). - Page-type detection (
page_type,is_article). - Strict / threshold gates (
strict=True,LowConfidenceExtractionError). - Public API:
fetch,extract,fetch_many,fetch_iter,discover, async variants,NewsFetcher. - CLI:
news-fetch,batch(JSONL),discover. - Proxy / proxy pools, rotation,
request_delay, 429 Retry-After. - Strategy plugins (
register_strategy). - Optional
[async](httpx) and[browser](Playwright). - Disk cache (
cache=True) andrespect_robots=True. - Fixture tests and benchmark harness.
Changed
- Breaking vs 0.4.x: core no longer depends on newspaper4k / BeautifulSoup / Unidecode.
- Preferred API is
from newsfetch import fetch(notNewspaper). - Classifier: Production/Stable.
Deprecated
newsfetch.news.Newspaper— shim retained for one cycle.
[0.6.0] / [0.5.x]
Internal pre-1.0 milestones introducing the independent engine, confidence model, and bulk/proxy support. Folded into the 1.0.0 public release.
[0.4.1]
Last release of the newspaper4k-wrapper era (JSON-LD @graph fixes, meta description, discovery limits, typing). See GitHub history for details.
[0.4.0] and earlier
See GitHub commits.