Editor Runs 30 Interviews Through Claude: Quotes Hold Up, but the Model Never Says "Don't Watch"

Over a weekend, Andrej Karpathy put together HackerTube: a showcase of 6,454 videos that Hacker News users have been recommending to one another for years. The founder of Pressfeed and editor of the magazine "Signal" decided to build a different product on the same data: not a link to a video, but a Russian-language card that can be read in a minute and answers whether the video is worth opening. The data is released under the ODC-By license, and forks are welcome.

The task is familiar to anyone who has built an LLM pipeline: the model generates confident-sounding text, but only part of it can be verified automatically. The most useful part of this story is exactly where the line of what can be verified falls.

How the pipeline works

From the 6,454 videos, the author kept interviews and podcasts, which gave 435. After removing one-day news items and videos that merely mentioned the topic, 310 remained, and 30 of those were picked by hand. Everything runs locally in Python, with no queues or databases. yt-dlp fetches subtitles, which are cut into blocks of roughly one minute with timestamps. The blocks, together with Hacker News comments, go to Claude Sonnet, which returns JSON: who is speaking, three main ideas, a quote, a verdict. Then code checks the model's claims, and Python builds static pages for nginx.

What the code checked and what it couldn't

The main rule: the model cannot simply assert that the subject said something. The code searches for the English quote in the transcript. If it matches character for character, the status is "exact". If a very similar passage is found (ratio of at least 0.85), the status is "near". If nothing is found, the model gets a second attempt, and after that the quote is simply not shown. Of 30 quotes, 26 matched exactly and 4 nearly; none had to be discarded. Timestamps work the same way: only a block number that actually exists in the transcript is accepted.

But the rest of the text can't be checked this way, and manual proofreading turned up three types of problems.

First: an invented fact. The model attributed a personal bankruptcy to Chris Sacca, although in the transcript he says the opposite: he declined bankruptcy and paid off the debt himself. The words "debt", "millions" and "responsibility" appeared close together, and the model finished the story the way such stories usually end.

Second: timing. The conversation between Karpathy and Lex Fridman was described as released "when ChatGPT had just appeared", although the video was uploaded on October 29, 2022, a month before its launch.

Third, and the most common: attractive but unverifiable superlatives such as "the only long interview" or "not documented anywhere else". At least five across 30 cards. The author's takeaway is worth remembering: it's not strange stories that should arouse suspicion, but smooth generalizations. The story about Ken Thompson's confiscated chess computer sounded wild, but turned out to be true.

The model can't say "don't watch"

Every card has a verdict: watch in full, watch a segment, or the card is enough. On the first pass the model produced 0 "in full", 25 "segment" and 5 "card is enough", and all five went to videos shorter than five minutes. After tightening the prompt (a segment no longer than 25 minutes, "enough" for roughly every third video, regeneration on violation), things got worse: "card is enough" vanished entirely, and hour-long segments survived the repeat request. The final four "enough" verdicts were assigned by a human: to Gates, Hinton, Hotz and Sacca. The author candidly notes that nobody rewatched the videos in full.

What it costs

With reasoning, the model spends 10–20 thousand output tokens and 2–5 minutes per card, though only a few cards were checked this way. Without reasoning, a card costs about a thousand tokens and 20 seconds. A run over 30 videos in three threads takes about 5 minutes and about 640 thousand input tokens. The author chose the second mode plus manual proofreading.

Takeaways

If you are building a similar pipeline, split the model's output into verifiable and unverifiable parts. For the former, write deterministic checks: quote search, validation of indices and ranges. For the latter, build in a human and separately search for words like "only", "first" and "nowhere else". Decisions that risk blame, such as "this isn't worth your time", are best not delegated to the model: it will gently dodge them no matter how you tweak the prompt.

One more side takeaway: bias in the source data is inherited. Karpathy's ranking rewards long-repeated recommendations, so the newest video is three years old and almost half are ten years old or more. For classics that's not a problem, but for current topics a different selection is needed.