Confidence & evidence

news-fetch ranks evidence from multiple strategies instead of blindly falling through a single scraper.

Pipeline

Strategies → Evidence → Rank (+ agreement / penalties) → Confidence → Article

Built-in strategies: JSON-LD, Open Graph / Twitter, HTML meta, semantic HTML, DOM heuristics. Custom strategies can be registered via plugins.

Confidence

article = fetch(url)

print(article.confidence.overall)
print(article.confidence.title)
print(article.confidence.content)
print(article.confidence.date)
print(article.title_confidence)  # alias

Confidence is evidence-based: strong structured sources score higher; agreement across strategies boosts; weak titles / thin bodies / high link density penalize.

Provenance

print(article.sources)
# {'title': 'json-ld', 'text': 'json-ld.articleBody', 'published_at': 'article:published_time', ...}

print(article.extraction.content.strategy)
print(article.extraction.content.confidence)
print(article.extraction.content.signals)
print(article.content_source)

Page type

print(article.page_type)   # article | homepage | category | search | live_blog | ...
print(article.is_article)

Helps filter “HTTP 200 but not an article” pages in bulk jobs.

Strict mode

from newsfetch import fetch, LowConfidenceExtractionError

try:
    article = fetch(url, strict=True)
except LowConfidenceExtractionError as exc:
    print(exc.failed_fields)
    print(exc.confidence.to_dict())
    # exc.article is still available for inspection

Field-specific gates:

fetch(url, min_content_confidence=0.85, min_title_confidence=0.80)

Debug tracing

article = fetch(url, debug=True)
for c in article.trace.candidates:
    if c.field == "title":
        print(c.score, c.source, c.selected, c.value)