Running agents

Claim drift: your agents are still saying the thing you already corrected

I deleted a false claim from my website on July 8. On July 27 I found my own agent still drafting from it.

7 August 2026 · written by the team behind freeapp.ai

On July 8 I deleted a sentence from my product's website because it was not true. The page said my software worked with several AI providers. It works with one.

On July 27 I opened the config file my marketing agent reads every morning. The sentence was still in there, sitting in a section labelled safe_to_state_as_fact.

Nineteen days after I corrected the site, a machine on my own box was still working from the old version. I have started calling this claim drift, because I could not find a name for it and I have now hit it three times.

The audit, about ten minutes

Run this the next time you correct anything public. It is the whole point of the article, so take it and skip the rest if you are busy.

# 1. Where else does the old claim live? Search the claim, not the file you fixed.
grep -rIl -e "ollama" -e "model-agnostic" ~/ --exclude-dir=.git --exclude-dir=node_modules

# 2. Which files do my agents read, and is any of them untracked?
#    Use find, not a ** glob: bash without globstar silently skips
#    anything more than one directory deep, which is where mine was.
find ~/.claude/agents ~/prompts -name '*.md' -print0 2>/dev/null |
  while IFS= read -r -d '' f; do
    git -C "$(dirname "$f")" ls-files --error-unmatch "$f" >/dev/null 2>&1 \
      || echo "UNTRACKED: $f"
  done

# 3. What can each agent actually do if its config is wrong?
grep -rn -e "disallowed-tools" -e "allowed-tools" -e "permission-mode" ~/scripts/

Three questions, in the order that matters: where else does this claim live, can I date it if it goes wrong, and what is the worst thing that happens if it does.

The rest of this is why each line is there.

1. Search the claim, not the file

I fixed a page. The claim was never a page.

It was a fact about my product, and by July it existed in at least four places: the landing copy, an agent config file, a role document, and my own head. I edited the one I happened to be looking at.

Every agent you run carries a copy of what it believes about your business. A prompt file, a system message, a retrieved document, a fine-tune. The moment you correct something publicly, all of those copies are wrong. Unlike a stale webpage they do not sit still. They keep asserting the old version, on a schedule, in your name.

The trigger: any time a public claim changes. Not at the end of the sprint, not in the next audit. The same hour, while you still remember the exact wording.

The rule: grep for the string and the paraphrase. A copy audit is not finished when the site is right, it is finished when nothing on the machine still says the old thing.

2. If it is not in git, you cannot date your own mistake

When I went to fix the config, I checked its history to find out when the false claim had been added.

There was no history.

The version in git was the original, sixteen lines, written on June 6. The file the agent was actually reading had grown to fifty-five lines through edits nobody ever committed. It had been quietly diverging from the repository for weeks.

So I cannot tell you what day that claim entered, and I cannot tell you how much of those nineteen days it was wrong for. Only that it was wrong at the end of them.

That is worse than the claim itself, and it is common, because agent config does not feel like code. It feels like settings. You edit a prompt at midnight to fix one bad output, it works, you move on. Nobody opens a pull request for a sentence.

The trigger: you are about to edit a prompt or an agent config to fix something. Before you close the file, check whether git has ever seen it.

The rule: a file your agent reads every morning is production. Track it for the timestamps, not for the review. The question you will need answered later is "since when", and only git can answer it.

3. Ask what it could do at three in the morning

Nothing false was ever published. I would like to say that was vigilance. It was not.

The script that runs that agent invokes it with file editing turned off, and there is no publishing integration wired into it at all. It can put text in a Telegram message. That is its entire surface.

So the worst realistic outcome of a poisoned config was me reading a draft, believing my own config file, and posting a lie by hand. Bad, and recoverable. The machine had no path to publish on its own, because that path was never built.

I want to be careful about what this proves, because the flattering reading is wrong. It does not prove I was disciplined. It proves that capability you never grant cannot be misused by a bug you have not found yet. The containment was decided in June, months before there was anything to contain.

Most agent-safety advice runs the other way. Give the agent the tools, then add review, monitoring and guardrails on top. Every one of those layers depends on somebody noticing. I did not notice this for nineteen days, and I am the only person looking.

The trigger: you are about to give an agent a new tool, a new key, or a new integration.

The rule: write down what it could do at three in the morning if its instructions were wrong, then delete whichever answer you do not like. Doing that in advance is a decision. Doing it afterwards is a postmortem.

What it costs to skip this

The three steps take about ten minutes and they do not need a budget or a platform.

Claim drift is quiet. It does not throw an error, it does not page you, and the agent stays confident throughout. You find it the way I found it, which is by opening a file for an unrelated reason on an ordinary Monday.

The general shape is this: your agents are a distributed copy of your beliefs, and correcting the original does not correct the copies. Everything above is one way of noticing before somebody else does.