FAQ
Is news-fetch a wrapper around newspaper4k?
No (as of 1.0). The core extraction engine is independent: JSON-LD, Open Graph, meta, semantic HTML, and DOM scoring live inside news-fetch. newspaper4k is no longer a dependency.
Do I need an API key?
No. Install with pip and call fetch(url).
Do I need to download NLTK data?
No. Keywords/summary use a small stdlib fallback. There is no NLTK requirement.
What’s the difference between description and summary?
description— page / structured metadata descriptionsummary— first sentences of the extracted body
Why is a field sometimes missing?
The engine only returns values supported by evidence on the page. Check article.sources and article.confidence to see what was found. Use debug=True to inspect candidates.
What does confidence mean?
A 0–1 score derived from strategy strength, cross-source agreement, and quality penalties. See Confidence & evidence.
Can I scrape thousands of URLs?
Yes — use fetch_many / fetch_iter or news-fetch batch with proxies, request_delay, and progress callbacks. One failure returns None / an error JSONL row without aborting the job.
Does it respect robots.txt?
Only when you opt in: fetch(url, respect_robots=True). Since 1.1.0 parsing follows RFC 9309, and an unreachable robots.txt (5xx, 429, network error) blocks the fetch; a 4xx means no restrictions. You are still responsible for site terms and polite rate limits.
How do I see why a URL failed in a batch?
Use fetch_results / iter_results (or fetch_results_async). Each FetchResult has .ok, .error, .status and .unwrap(). fetch_many / fetch_iter still return None for failures.
Can I get the article as Markdown?
Yes — article.markdown is always filled. For cleaned body HTML pass include_html=True and read article.html. From the CLI: news-fetch get URL --markdown.
Why did I get FetchError on a large page?
Responses over 10 MiB (after decompression) are rejected. Raise the limit with max_bytes=... or disable it with max_bytes=None.
Why is published_at None where 1.0 had a date?
1.1.0 no longer completes partial dates from today's date. If the page doesn't give a full year, month and day, the date is rejected.
Which Python versions are supported?
Python 3.11+ as of 1.1.0.
Does it bypass paywalls / CAPTCHAs?
No. Optional Playwright support is for JS-rendered HTML, not access-control bypass.
Does async require a separate install?
Yes: pip install news-fetch[async].
Does browser rendering require a separate install?
Yes: pip install news-fetch[browser] then playwright install chromium. Use render=True or browser_fallback=True.
Is the old Newspaper API gone?
It still works as a deprecated shim. New code should use from newsfetch import fetch.
Something isn’t working — where do I report it?
Open an issue on GitHub with the URL (if public) and what you expected vs. got. Prefer attaching debug=True provenance when possible.