Session 7.5: Using APIs to Protect Yourself From AI Slop

Course → Module 7: APIs as Research Tools

Session 5 of 7

Search APIs are not just for research. They are for defense. Every factual claim your AI generates can be verified against search results. Every statistic can be cross-referenced. Every "according to experts" can be checked for actual experts actually saying that thing.

This turns your pipeline from a content generator into a content verifier. That verification step is what separates your work from slop.

The Verification Problem

AI hallucination is not random. It follows patterns. Models hallucinate most often in specific categories.

Claim Type	Hallucination Risk	Example
Specific numbers	High	"Revenue grew 47% in Q3" (number invented)
Attributed quotes	High	"As Jeff Bezos once said..." (never said it)
Source citations	Very high	"A 2024 Harvard study found..." (study does not exist)
Temporal claims	Medium	"In 2019, the regulation changed..." (wrong year)
Company facts	Medium	"Founded in 2015 in Austin" (wrong city)
General knowledge	Low	"Python is a programming language" (correct)

The verification strategy is simple: check the high-risk claims first. Specific numbers, attributed quotes, and source citations are where hallucination lives. General knowledge is usually reliable. Focus your verification budget where the risk is highest.

Automated Verification Pipeline

A verification pipeline takes AI-generated content, extracts verifiable claims, searches each claim against web results, and flags discrepancies for human review. This is not fully automated fact-checking. It is automated flagging that focuses human attention where it is most needed.

graph TD A["AI-generated content"] --> B["Extract verifiable claims
(manually or via AI)"] B --> C["For each claim:
search via Tavily
or Google"] C --> D{"Search results
support the claim?"} D -->|"Yes: supported"| E["Mark as verified
(with source)"] D -->|"No: contradicted"| F["Flag for correction"] D -->|"No results"| G["Flag as unverifiable"] E --> H["Verification report"] F --> H G --> H H --> I["Human reviews
flagged items"] style F fill:#2a2a28,stroke:#c47a5a,color:#ede9e3 style E fill:#2a2a28,stroke:#6b8f71,color:#ede9e3 style G fill:#2a2a28,stroke:#c8a882,color:#ede9e3

Verification Tools

Several tools and APIs support automated verification at different levels of sophistication.

Tool	Approach	Best For
Tavily Search	Search each claim, compare results	General fact verification
Google Fact Check API	Queries existing fact-checks from ClaimReview markup	Claims already fact-checked by journalists
Gemini with grounding	Model verifies its own claims via search during generation	Real-time prevention during generation
ClaimBuster	AI model that detects claims worth checking	Prioritizing which claims to verify
Originality.ai Fact Checker	Automated claim extraction and verification	Quick checks on complete articles

Building a Practical Fact-Checking Script

The simplest useful fact-checking script follows this logic: take a list of claims (one per line), search each claim using a search API, compare the claim to the top results, and return a verdict for each: Supported, Contradicted, or Unverifiable.

The comparison step is where another AI call helps. Feed the original claim and the search results to an LLM with instructions to determine whether the search results support, contradict, or are insufficient to evaluate the claim. This is AI checking AI's work, which is imperfect, but it catches the most obvious hallucinations and flags the rest for human review.

graph LR A["Claim list"] --> B["For each claim"] B --> C["Tavily search"] C --> D["LLM comparison:
claim vs. search results"] D --> E["Verdict:
Supported / Contradicted /
Unverifiable"] E --> F["Verification report"] style D fill:#2a2a28,stroke:#c8a882,color:#ede9e3 style F fill:#2a2a28,stroke:#6b8f71,color:#ede9e3

What Verification Catches (and Does Not Catch)

Automated verification catches: invented statistics, non-existent studies, wrong dates, incorrect company facts, and misattributed quotes. These are the errors that destroy credibility fastest and are easiest to detect with a search.

Automated verification does not reliably catch: subtle mischaracterizations, correct data used in misleading context, cherry-picked statistics that are technically accurate but not representative, and claims that are too niche for search engines to have relevant results. These require human judgment.

The goal is not to automate all fact-checking. The goal is to automate the easy 80% so you can focus your limited human review time on the hard 20%.

Assignment

Build a simple fact-checking script: it takes a list of claims (one per line in a text file), searches each claim using Tavily or Google Search, and returns "Supported," "Contradicted," or "Unverifiable" for each claim.
Test it on 10 factual claims from an AI-generated article. Include a mix of likely-correct claims (general knowledge) and likely-hallucinated claims (specific statistics, citations, quotes).
How many claims passed verification? How many were caught as incorrect? Were there any false positives (flagged as wrong but actually correct) or false negatives (passed as correct but actually wrong)? Document the accuracy of your verification pipeline.

Using APIs to Protect Yourself From AI Slop