FAQ

Is news-fetch a wrapper around newspaper4k?

No (as of 1.0). The core extraction engine is independent: JSON-LD, Open Graph, meta, semantic HTML, and DOM scoring live inside news-fetch. newspaper4k is no longer a dependency.

Do I need an API key?

No. Install with pip and call fetch(url).

Do I need to download NLTK data?

No. Keywords/summary use a small stdlib fallback. There is no NLTK requirement.

What’s the difference between description and summary?

  • description — page / structured metadata description
  • summary — first sentences of the extracted body

Why is a field sometimes missing?

The engine only returns values supported by evidence on the page. Check article.sources and article.confidence to see what was found. Use debug=True to inspect candidates.

What does confidence mean?

A 0–1 score derived from strategy strength, cross-source agreement, and quality penalties. See Confidence & evidence.

Can I scrape thousands of URLs?

Yes — use fetch_many / fetch_iter or news-fetch batch with proxies, request_delay, and progress callbacks. One failure returns None / an error JSONL row without aborting the job.

Does it respect robots.txt?

Only when you opt in: fetch(url, respect_robots=True). Since 1.1.0 parsing follows RFC 9309, and an unreachable robots.txt (5xx, 429, network error) blocks the fetch; a 4xx means no restrictions. You are still responsible for site terms and polite rate limits.

How do I see why a URL failed in a batch?

Use fetch_results / iter_results (or fetch_results_async). Each FetchResult has .ok, .error, .status and .unwrap(). fetch_many / fetch_iter still return None for failures.

Can I get the article as Markdown?

Yes — article.markdown is always filled. For cleaned body HTML pass include_html=True and read article.html. From the CLI: news-fetch get URL --markdown.

Why did I get FetchError on a large page?

Responses over 10 MiB (after decompression) are rejected. Raise the limit with max_bytes=... or disable it with max_bytes=None.

Why is published_at None where 1.0 had a date?

1.1.0 no longer completes partial dates from today's date. If the page doesn't give a full year, month and day, the date is rejected.

Which Python versions are supported?

Python 3.11+ as of 1.1.0.

Does it bypass paywalls / CAPTCHAs?

No. Optional Playwright support is for JS-rendered HTML, not access-control bypass.

Does async require a separate install?

Yes: pip install news-fetch[async].

Does browser rendering require a separate install?

Yes: pip install news-fetch[browser] then playwright install chromium. Use render=True or browser_fallback=True.

Is the old Newspaper API gone?

It still works as a deprecated shim. New code should use from newsfetch import fetch.

Something isn’t working — where do I report it?

Open an issue on GitHub with the URL (if public) and what you expected vs. got. Prefer attaching debug=True provenance when possible.