FAQ
Is news-fetch a wrapper around newspaper4k?
No (as of 1.0). The core extraction engine is independent: JSON-LD, Open Graph, meta, semantic HTML, and DOM scoring live inside news-fetch. newspaper4k is no longer a dependency.
Do I need an API key?
No. Install with pip and call fetch(url).
Do I need to download NLTK data?
No. Keywords/summary use a small stdlib fallback. There is no NLTK requirement.
What’s the difference between description and summary?
description— page / structured metadata descriptionsummary— first sentences of the extracted body
Why is a field sometimes missing?
The engine only returns values supported by evidence on the page. Check article.sources and article.confidence to see what was found. Use debug=True to inspect candidates.
What does confidence mean?
A 0–1 score derived from strategy strength, cross-source agreement, and quality penalties. See Confidence & evidence.
Can I scrape thousands of URLs?
Yes — use fetch_many / fetch_iter or news-fetch batch with proxies, request_delay, and progress callbacks. One failure returns None / an error JSONL row without aborting the job.
Does it respect robots.txt?
Only when you opt in: fetch(url, respect_robots=True). You are still responsible for site terms and polite rate limits.
Does it bypass paywalls / CAPTCHAs?
No. Optional Playwright support is for JS-rendered HTML, not access-control bypass.
Does async require a separate install?
Yes: pip install news-fetch[async].
Does browser rendering require a separate install?
Yes: pip install news-fetch[browser] then playwright install chromium. Use render=True or browser_fallback=True.
Is the old Newspaper API gone?
It still works as a deprecated shim. New code should use from newsfetch import fetch.
Something isn’t working — where do I report it?
Open an issue on GitHub with the URL (if public) and what you expected vs. got. Prefer attaching debug=True provenance when possible.