Reliability

Catch production failures before they page you.

Dalton learns how your system really behaves, notices when something starts to drift, and investigates it at once. You get the cause and the fix to approve while there is still time to act.

Book a demoSee an investigation
db-primary connection poolinvestigation opened 03:51
  • Cause found03:53
  • Your alert fires at 240 in use04:12

19 minutes before your alert, with the pool 51% full.

How it works

From the first drift to the fix you approve.

Another incident, start to finish: a limit no alert covers, found a week before it bites, through to the fix your team approves.

Predict

It sees trouble a week out.

The largest ads-feed export grows 1.1% a day. Dalton works out that it crosses the 4 MiB message limit on about October 7, while every check is still green.

Investigate

Ask it what else is at risk.

The day after a reindex hit the same limit, someone on the team asks. Dalton searches every repository and answers with the jobs, their sizes and their dates.

It queues what needs a person.

ads-feed joins the investigations waiting for a person, with its severity and its age, next to everything else Dalton is watching.

It tests every theory, and shows its work.

The job will report a timeout, but it is not one: export times are flat. The cause is one message per merchant, capped at 4 MiB.

Propose

It hands you the fix, ready to review.

Pull request #2297 fetches every feed in pages, as the reindex's fix does. Nothing ships until someone on your team approves it.

Every source it reads

Learn

It remembers what your team teaches it.

Lessons from your team and from past investigations, each with who taught it and when. Edit one, or switch it off, and the next investigation follows.

How you stay in control
Dalton's prediction for ads-feed: it will start failing its hourly export in about 7 days. Its largest feed is 3.71 MiB and grows 1.1% a day, so it crosses the 4 MiB limit on about October 7.
Someone on the team asks Dalton whether anything else will hit the 4 MiB limit. It answers yes: ads-feed on about October 7, and recommendations-trainer about three weeks later, with a table of every caller it found.
The investigations waiting for a person: ads-feed's export at the top, then recommendations-trainer, a certificate that will expire, and the rest, each with its severity and its age.
Investigation: ads-feed exportcause found
  • A timeout between ads-feed and search-apiruled outExport times are flat. The timeout the job will report is the size limit.
  • A release that made the feed biggerruled outNo ads-feed release in 30 days. The feed grows as merchants add products.
  • search-api answering more slowlyruled outIts p99 is unchanged this month, at every hour of the export.
  • ExportFeed sends each merchant’s whole feed in one messageconfirmedCapped at 4 MiB. The largest feed is 3.71 MiB, growing 1.1% a day.metricscodeconfig
PR #2297: fetch every feed in pageswaiting for review

jobs/ads-feed/source/feed.go

  // Fetch returns one merchant’s feed.
  func (s *Source) Fetch(ctx Ctx, req *Req) ([]*Item, error) {
-   resp, err := s.client.ExportFeed(ctx, req)
-   return resp.GetItems(), err
+   // 500 items a page, as the reindex does
+   return s.feedInPages(ctx, req.Merchant)
  }
  • Touchesads-feed, recommendations-trainer
  • Testeda feed past 4 MiB arrives in pages
  • Rollbackone revert, ready
  • Appliednothing, until you approve it
Team memory4 lessons

  • ads-feed exports peak on the first of the month. Compare with the same day last month.

    Learned from an investigation, 30 Jul

  • A search-reindex run takes up to 40 minutes. Read its progress log before calling it stuck.

    Learned from an investigation, 21 Aug

  • The 02:00 backup job raises db-primary connections every night. Expected.

    Taught by Priya Raman, 12 Sep

  • legacy-cart is being retired. Its errors are not incidents.

    Taught by Leo Marchetti, 3 Sep

A real one, start to finish:

57 hours without Dalton. Minutes with it.

What teams ask about AI reliability.

What is AI reliability?

Using AI to keep production reliable before something breaks, not only after. An AI reliability platform learns how your system behaves, investigates anything that drifts, and hands you the fix before it becomes an incident. Dalton is one.

How is it different from observability?

Observability shows what is happening; AI reliability works out why. Your dashboards, logs and traces stay where they are. Dalton reads them with your code and deploys, and brings you the cause with the fix.

How is it different from AIOps?

AIOps sorts alerts; AI reliability explains causes. AIOps groups and filters the alerts your stack already fires. An AI reliability platform reads code, deploys and system behavior, so it can explain a failure, and catch one no alert covers.

Do we still need our monitoring?

Yes. Dalton runs on top of your monitoring and replaces none of it. It reads your metrics, logs, traces and alerts with read-only access and uses them as evidence.

Will it replace our SREs?

No. It takes the digging off their plate. Your engineers still decide what ships and how the system is built; Dalton does the investigation that used to wake them up, and shows them the evidence.

How should we evaluate a platform like this?

Give it a real incident from last quarter. Check whether it reaches the cause your team found, how fast, and with what evidence. Then check what it can read, what it keeps, and whether every write needs your approval.

Catch your next failure before it pages you.

A demo first, then a two-week proof of concept on your own production, read-only on every integration.

Book a demo