monitoring for chatbots
Your chatbot changed and nobody told you.
It watches every answer, learns what normal looks like, and when it drifts sends one Slack message with what changed, probably why, and the requests that prove it.
An incident, from normal answers to the investigation note
⠋ loading an incident
Math to notice. One LLM call to explain.
01 · detect
Compare with normal
Each answer is embedded and compared with the bot's own normal answers. Plain math (nearest neighbours and CUSUM), no LLM, no cost per answer.
02 · investigate
Gather evidence, then write
Code compares the numbers, clusters the changed answers and lines up version changes. Then one LLM call writes the note, and every claim must cite those results by id.
03 · alert
One incident, not fifty pings
Signals become one incident, with at most 3 Slack messages: opened with the note, escalated, resolved.
Built to catch when your bot's behaviour changes, not to fact-check single answers. Limits
Measured, misses included.
8 scripted scenarios replayed through the production code: 300 normal answers, then 100 changed ones. One seed, so these are single runs, not rates.
causes named right
7 of 8
"not sure" counts right where the cause can't be known
false incidents, normal traffic
0 of 2
more normal answers; a harmless version bump
prompt change caught after
10 answers
when the bot sends prompt_version
weak spot
topic shift
caught after 4 answers, but the note said "unknown"
Three lines.
Get a key in the app, then post each answer after your bot replies:
import requests
requests.post("https://sentinelai-qqph.onrender.com/ingest", headers={"X-API-Key": KEY},
json={"prompt": question, "response": answer})Have a bot with real traffic?
Send me 50 answers and I'll send back what it finds.
Issues are public: share anonymised answers only, never customer data or keys.