Method
The Pravda network (also called “Portal Kombat”) is a documented network of pro-Russian websites that republishes content in large volumes in dozens of languages. This site reads its Lithuania edition through the public RSS feed and news sitemap and compares each story with trusted Lithuania news. Being on the Pravda network does not make a story false: the network also republishes real news. This site flags; a person decides.
The pipeline, for every Pravda story
- Find candidates. Trusted articles published within ±72 h: the top 5 by character
n-gram similarity plus the top 5 by embedding similarity (
text-embedding-3-large, one model for all texts). - Same event? An LLM labels each candidate same event, same topic or unrelated. Only same-event articles are used as evidence. If none is found, the story is shown as “not found in trusted news (yet)”. That means a person should look. It is not a verdict.
- Compare. This site fetches the full text of the same-event articles, and the LLM lists what the Pravda story contradicts, adds, omits or reframes, quoting both sides exactly.
- Verify the quotes in code. Every quote must appear word for word in the article it cites (ignoring diacritics, quote styles and spacing). Items whose quotes cannot be found are dropped. An “omitted” item that cites a trusted report published more than 1 h after the Pravda story is also dropped: a site can't omit what nobody had reported yet.
- Score.
- Misinformation (facts) is counted by code from verified contradictions only: 0 → low, 1 → medium, 2 or more → high. Claims that are only “added” (not found in trusted news) may still be true. They are shown as “unsupported — check this” and do not raise the score.
- Manipulation (framing) is rated by the LLM from omissions and reframing, and is capped at low unless at least one verified reframed or omitted item exists.
- No verified items at all → both scores are “not sure”.
How well does it link stories? (hand-labelled gold set)
30 Pravda stories (20 that mention Lithuania, 10 at random) and every candidate either method proposed, labelled by hand: 206 pairs. 23 of the 30 stories have a same-event trusted article.
Finding the right article: how often the right trusted article is among the top candidates.
| Method | in top 1 | in top 3 |
|---|---|---|
| Character n-grams (title + summary) | 70% | 78% |
| Embeddings (text-embedding-3-large) | 74% | 87% |
Same-event check: precision = when the model says “same event”, how often it is right (a wrong “yes” puts the wrong evidence next to a story); recall = how many true same-event pairs it finds. “Blind” = the 10 stories labelled before seeing any model's answer: the most honest numbers.
| Model | precision | recall | blind precision | blind recall |
|---|---|---|---|---|
| Claude Haiku 4.5 | 94% | 69% | 87% | 77% |
| Gemini 3.5 Flash | 91% | 80% | 100% | 85% |
| GPT-5 mini | 89% | 84% | 92% | 85% |
| Claude Sonnet 5.5 | 83% | 81% | 85% | 85% |
| Nemotron 3.5 Super, thinking on | 89% | 80% | 88% | 81% |
| Nemotron 3.5 Super, thinking off | 84% | 91% | 85% | 85% |
All 206 pairs: 64 same-event. Blind subset: 62 pairs, 26 same-event. Small samples: one pair moves the blind numbers by about 4 points.
Script: experiments/03_gold_eval.py.
Live numbers (Lithuania)
Quote check: 5646 items kept, 55 dropped because a quote could not be found, 54 dropped because they cite a later report.
| Score (compared stories) | low | medium | high | not sure |
|---|---|---|---|---|
| Misinformation | 1448 (90%) | 87 (5%) | 7 (0%) | 61 (4%) |
| Manipulation | 710 (44%) | 676 (42%) | 156 (10%) | 61 (4%) |
Human check (blind)
One person labelled 40 flagged stories (186 items) without seeing which model produced them or its scores: each item is a real distortion, harmless, or unsure. Rule used: criticism or opinion is not propaganda; a distortion changes what a reader believes happened — the facts, the cause, who did it, or the scale.
| Sonnet 5.5 (20 stories) | Nemotron, thinking on (20 stories) | |
|---|---|---|
| Flagged stories with ≥1 real distortion | 85% (17/20) | 75% (15/20) |
| Item precision (unsure excluded) | 82% (54/66) | 58% (36/62) |
| — added | 86% | 64% |
| — omitted | 67% | 36% |
| — reframed | 84% | 67% |
| — contradicted | 100% (n=1) | 40% (n=5) |
| Wrong pair (not the same event) | 0% | 10% (2/20) |
Read a flag as “worth a look”, never as “this is propaganda”. One labeller, small samples (about ±10 points). 35% of items were “unsure”, mostly because the trusted article was headline-only (paywall or Google News link), so “only in Pravda” can be a gap in the site’s input. Earlier stories were checked with Sonnet; newer ones with Nemotron, which is noisier — “omitted” is its weakest item type.
Models that produced the stored results:
claude-haiku-4.5-eu+claude-sonnet-5.5-eu (2471), nemotron-3.5-super-vl+nemotron-3.5-super-vl (1)
(same-event check + comparison).
Events and agendas
An agenda is a belief the network pushes for months. An event is one thing that happened. The site keeps them apart.
- Events. Stories are read in time order. A story joins an open event when it is close in meaning to the event's centre (same embeddings as above, within 72 h), otherwise it opens a new one. An event needs at least 8 Pravda-network stories. No AI call. Trusted coverage of an event = the trusted articles already verified as the same event, story by story. Events
- The belief behind an event. One AI call per event reads its headlines and verified differences and writes the belief the coverage pushes, in a form that could recur through other events.
- Which agenda? The same call sees the known agendas and picks the one pushing the same belief, or none. The same topic is not enough.
- New agendas, automatically. An event that fits no agenda waits. When a second, different event pushes the same belief, the two become a new agenda. One event is news; the same slant twice is an agenda. Existing agenda names are never changed.
The first agendas of each country came from grouping all stories by meaning (k-means, AI-written names). Not yet measured: the event → agenda step was read by one person on 28 events, with no labelled check. An event means the network piled onto something, not that the stories are false. Only bursts are seen: a belief pushed in a slow drip does not form an event.
Data and limits
- Only public RSS feeds, news sitemaps and article pages. This site respects robots.txt and crawl delays. Feeds are polled every 15 minutes. All sources and their health: Sources.
- Some outlets only appear through Google News links, so the site matches on their title and summary, not the full text.
- Trusted coverage is not complete: “not found” can mean a story is real but unreported.
- Countries are added as country packs (a Pravda edition + that country's trusted outlets): a list of sources, not new code.
- Flagged stories as a feed: RSS.