Changelog

Documentation mirrors the library changelog. Full details: news-fetch CHANGELOG.

[1.1.0] - 2026-11-15

Upgrade notes

  • Python 3.11 or newer is required.
  • With respect_robots=True, an unreachable robots.txt (5xx, 429, network error) now blocks the fetch.
  • Responses larger than 10 MiB (after decompression) raise FetchError; raise or disable max_bytes if needed.
  • Dates without a full year, month and day are no longer guessed; published_at may be None where it used to hold a made-up date.
  • Language codes keep their full primary subtag (fil, yue), not always two letters.
  • CLI batch error rows changed from "error": true to an object.
  • pip install news-fetch[dev] no longer exists; use uv sync (see Installation).

Added

  • Config.max_bytes (default 10 MiB): response bodies are streamed and capped after decompression (sync and async), protecting against oversized pages and gzip bombs. None/0 disables it.
  • RFC 9309 robots.txt support: * / $ wildcards, longest-match precedence (allow wins ties), product-token group matching with merged groups, percent-encoding normalization, 500 KiB cap. New RobotsRules and RobotsCache.check() returning a RobotsDecision(allowed, reason).
  • discover(..., client=HttpClient); NewsFetcher.discover reuses the fetcher's client.
  • FetchResult(url, article, error) with .ok, .status, .unwrap(), .to_dict(), returned by new fetch_results / iter_results, NewsFetcher.fetch_result / fetch_results / iter_results and fetch_results_async. Existing fetch_many / fetch_iter / fetch_many_async are unchanged.
  • Article.markdown: article body as Markdown (headings, lists, links, images, blockquotes, code, GFM tables), always filled. sources["markdown"] says whether it came from the DOM node or plain text.
  • Article.html is now filled with cleaned body HTML when Config(include_html=True).
  • CLI: news-fetch get --markdown.
  • HttpClient.get(..., max_bytes=, truncate=) per-call size overrides.
  • CI test workflow: Python 3.11–3.14 (+ macOS), lowest-dependency job, wheel smoke test, ruff; Dependabot for SHA-pinned actions.

Changed

  • Python 3.11+ required (3.10 reached end of life in October 2026).
  • Development uses uv with a committed uv.lock; dev tools moved to a PEP 735 [dependency-groups] table. The dev install extra and requirements*.txt are removed. Publishing builds with uv build.
  • CLI batch: failed URLs are written as {"url": ..., "error": {"type", "message", "status"}} (was "error": true), a N ok, M failed summary goes to stderr, and the exit code is 1 when no URL succeeded.
  • fetch_many_async now runs on asyncio.TaskGroup.
  • robots.txt (respect_robots=True): 5xx, 429 and network errors now mean "unreachable" → fetch disallowed (previously allowed). 4xx still means no restrictions. Error messages state the reason.
  • robots.txt and discovery requests go through HttpClient: configured user agent (no longer python-requests), proxies, retries and size limits apply.
  • Sitemap/feed XML is parsed with entity expansion, DTD loading and network access disabled.
  • User agent bumped to news-fetch/1.1.

Fixed

  • <html lang> was never detected; language tags are reduced to the BCP 47 primary subtag (fil no longer becomes fi).
  • DiskCache is now thread-safe (fetch_many(cache=True) raised sqlite InterfaceError).
  • Retries no longer multiply (retries=2 made 9 connection attempts; now 3).
  • Dates missing a year, month or day are rejected instead of being completed from today.
  • Keyword extraction works for non-Latin scripts.
  • fetch_iter / iter_fetch stop fetching when the consumer stops early.
  • A missing cssselect install now raises instead of silently matching nothing.

Removed

  • Unused internals: heuristic.iter_images_near, Extractor.extract_from_nodes.

[1.0.0]

Added

  • Production-stable independent extraction engine.
  • Evidence → rank → confidence (article.confidence, article.extraction).
  • Page-type detection (page_type, is_article).
  • Strict / threshold gates (strict=True, LowConfidenceExtractionError).
  • Public API: fetch, extract, fetch_many, fetch_iter, discover, async variants, NewsFetcher.
  • CLI: news-fetch, batch (JSONL), discover.
  • Proxy / proxy pools, rotation, request_delay, 429 Retry-After.
  • Strategy plugins (register_strategy).
  • Optional [async] (httpx) and [browser] (Playwright).
  • Disk cache (cache=True) and respect_robots=True.
  • Fixture tests and benchmark harness.

Changed

  • Breaking vs 0.4.x: core no longer depends on newspaper4k / BeautifulSoup / Unidecode.
  • Preferred API is from newsfetch import fetch (not Newspaper).
  • Classifier: Production/Stable.

Deprecated

  • newsfetch.news.Newspaper — shim retained for one cycle.

[0.6.0] / [0.5.x]

Internal pre-1.0 milestones introducing the independent engine, confidence model, and bulk/proxy support. Folded into the 1.0.0 public release.

[0.4.1]

Last release of the newspaper4k-wrapper era (JSON-LD @graph fixes, meta description, discovery limits, typing). See GitHub history for details.

[0.4.0] and earlier

See GitHub commits.