Blog
From Signals to Causes: What AI Investigation Needs
Manifesto, Oct 14, 2026, 7 min read
Dalton once flagged a config change I'd made. It said the change was going to make part of our system stop sending data. I'm a DevOps person by training, and I looked at that change and couldn't see the problem for the life of me. I honestly thought it was lying.
It wasn't. I don't think my dashboards would have shown it either, because the problem was data that was about to go missing. What caught it was knowing what my change touched and what depended on it, which is a different thing from reading signals faster.
Most of what's being sold as AI investigation right now is about the agent. That story is why I think the agent is the wrong part to look at. When AI investigation goes wrong, it's usually missing context, and a stronger model won't help much.
Signals are not causes
Signals tell you something moved: an alert, a status change, a sharp shift in a metric, a burst of errors in the logs, a slow span in a trace. They don't tell you why it moved.
Try it. This is the example incident from our homepage, cut down to what an agent gets if signals are all it reads.
What an agent sees if it only reads signals
- 03:41 db-primary connections start climbing
- 03:47 checkout-svc p99 crosses 1s
- 03:53 nothing has fired yet
What caused it?
- A traffic spike fits. More requests, more connections.
- A slow database fits. Slow queries hold their connections longer.
- A bad deploy fits. But which deploy, and what in it? The signals don't say.
- A config change fits. A smaller pool would fill up the same way.
All four fit what you saw. From the signals alone, any answer is a guess.
What the rest of the system knew. None of it is a signal.
- MetricsTraffic on checkout is flat against the last 7 days.Not a traffic spike
- LogsOnly v2.14.8 pods hold connections past the old cap.Not the database
- Configmax_connections on db-primary did not change. It is still 256.Not a config change
- Kubernetes03:39: checkout-svc v2.14.8 deployed. The climb began two minutes later.The deploy
- GitHubCommit a3f2 removed the cap in getPool(n).The cause
If you picked the deploy, good instinct. But nothing in the signals said which deploy, or what in it, so you were guessing too, just luckily.
The why lives somewhere else: a commit, a deploy that went out just before the climb, a config value, who owns the service and what else breaks when it does. That's code, CI/CD and cloud, and a lot of it still lives in people's heads.
When I wrote What is AI SRE? I said observability made signals richer without removing the person who has to connect them. That's still true. An alert fires at 3am and someone ends up with Grafana in one tab, the deploy log in another, a diff in a third and a Slack thread asking who owns this thing. They find the cause because they know how the pieces fit together, and most of that knowledge isn't written down anywhere.
An agent that only reads signals is doing the same job without that knowledge. It can correlate. It can tell you latency went up when the error rate did. I say this a lot, but everybody's doing correlation and almost nobody gets to the cause. A cause needs the link between the symptom and the change behind it, and that link isn't in the signal.
So without a layer that connects code, CI/CD, cloud and observability, any model is guessing. A good model guesses well, which makes it harder to notice that it's guessing.
We saw it in our own numbers
Earlier this month Michael, our CTO, benchmarked the small routing decisions Dalton makes before it does anything interesting, like which agent should take a task. He pointed Jev, a model TypeSafe built for narrow scored decisions, at five of our LLM classifiers. It came back 15x faster, up to 44x cheaper and no more accurate. Every router he tested, our own included, stalled near 70%.
When the team traced all 507 decisions, almost none of the mistakes were the model's. They came from sixteen short text files that describe what each agent can do, and only one of those files was accurate. Correcting them gave our own LLM checker 14.5 points of precision with the same model. He's careful to say that shows what accurate context can do, not a held-out test, and both the write-up and how he ran it are worth your time.
The agent is the cheap part
Most of the AI SRE market is racing on the agent right now, and every launch I see is a video of an agent typing into a chat window. I get it. The agent is what you can put in a demo, and if I were on stage I'd want it on screen too.
But the agent is the cheap part. Well, cheap isn't the right word, models cost real money. Swappable is closer. We just swapped one in and it came back faster and cheaper, and no more accurate. What you can't swap is what the model gets to read.
Investigation is a long chain of small calls like those: where to look next, which result to believe. Each one is only as good as what the agent can see. Give the best model in the world the wrong context and it will tell you, fast and with total confidence, about the wrong service.
Missing data is a signal too
Ben wrote about this in The Three Meanings of "Nothing". Mid-investigation, a tool call comes back empty, and that empty answer can mean the tool broke, the query worked and found zero matches, or the thing you asked about doesn't exist. Each one needs a different next step. An agent that treats them as one either retries forever, reports a failure that isn't one, or keeps searching for something that was never there.
My config change was the same problem from the other side: data about to go missing. An agent that only reads signals has nothing to read when a signal stops arriving, and often nothing fires when data quietly stops.
An empty result is information, and so is "we have no logs for this service." A system that holds those facts can say "I can't see that, and here's why" instead of making something up.
Three questions to ask any AI SRE vendor
I don't love the name AI SRE, and we keep it away from what we call ourselves, but it's what people search for, so it's in this heading. If you're evaluating these products, the demo will show you the agent. Ask about what's under it.
What data does it see?
I don't mean how many integrations it lists. I mean which of your systems it reads while it works an incident. Is it signals only, or code, deploys, config, ownership and dependencies too? Take the cause of your last bad incident and ask whether this product could have seen it.
How does it connect that data?
A long list of connectors doesn't tell you much. Ask them to show you how a slow climb like the one above gets linked to the commit behind it. Is that link there before the incident and kept current as you ship, or does a model rebuild it from scratch, under pressure, every single time? And ask what happens after a deploy changes what normal looks like, because every change you ship moves it a little.
What does it do when data is missing?
Ask for the failure case. Point it at a service with no logs, or ask about one that doesn't exist. A good product tells you it can't see it and why. A weak one fills the gap with a confident story, and you find out later.
Where Dalton stands
So our bet is on what the agent reads. Dalton joins your code, CI/CD, cloud and observability into one map and keeps it current on every deploy. With that map, the climb in the figure above shows up already linked to the deploy and the commit behind it, while it's still a climb and before it's an outage.
Because the map is already there before anything breaks, Dalton doesn't have to wait for an alert to start. I keep saying that proactive after the alert isn't really proactive, it's just automated reacting. Dalton catches failures while they're still forming, like my config change, and hands you the fix to approve. Every write needs a person's approval, each time.
Before you watch any vendor's agent, find out what it reads.