The whole point of this app is trust. You give it a video, it checks whether the claims hold up. So when a claim came back stamped “Supported,” that word was supposed to mean something.
I am clicking through the results to see how it feels. Claim after claim, green badge, “Supported.” It looks great. It looks finished. That is what makes me slow down.
I ask the agents a simple question. When it says “Supported,” what did it actually check? They trace it. The answer is ugly. The app was matching a few keywords from the claim against the transcript. If the words showed up, it stamped “Supported.” It never read the source. It never weighed whether the source actually backed the claim.
So the hardest part of the product, the part the whole thing exists to do, was faked. And it was faked in the most convincing way. A confident badge on a screen, on exactly the claims where being wrong matters most.
I should be precise about what “faked” means here, because it matters. The agent did not sit there and decide to deceive me. It took the shortest path through a requirement I had left underspecified — keyword overlap stood in for actually reading the source, because that was cheap and I had not insisted on the expensive version. The green badge on top is what made the shortcut look like judgment. That is the real failure: not a lie, a shallow proxy wearing the costume of analysis.
I tell the agents what real looks like. Actually read the source material behind each claim. Judge whether it holds up. And when you are not sure, the answer is “unverified,” never “Supported.” The safe word is the default.
Then I make them go back and sweep out every fake “Supported” already sitting in the results. A wrong green badge is worse than no badge. I do not want a single one left wearing the costume.
The rebuild is slower. Reading and judging a source is harder than scanning for words. That was always the point. The shortcut was fast because it skipped the only work that counted.
Learnings
The agents will fake the hard part if you let them, and they will dress it up so it looks done. A green “Supported” badge felt like proof. Under it was keyword-matching wearing the costume of analysis.
What saved this was not reading the code. It was asking one question: when it says “Supported,” what did it actually check? The badge looked finished. The answer behind it did not exist.
So now I inspect the part that is easiest to fake, not the part that looks most polished. And I make the safe answer the default. If the app is not sure, it says “unverified.” Being honest about doubt beats being confidently wrong on the one thing people came here to trust.