- An LLM-as-a-judge that scores each generation against a prompt you write
- A code-based eval that checks something you can express exactly
- Built-in sentiment analysis that classifies how the user sounds
Score your AI's answers with evals
We've all encountered a confidently wrong model in the wild. If this happens in your product, it's likely that nothing ever errors, so every dashboard you monitor stays green.
You probably have numbers that describe how the model performed – its speed, cost, and responsiveness. The piece you're missing is whether what it said was true.
What scores a model's output
An evaluation scores the model's output. AI Observability (AIO) runs three kinds of evaluations against your live traffic:
Each result lands on the generation it scored, with the reason it passed or failed – Fig. 1 is an example of that list in the app.
| Timestamp | Target | Result | Reasoning |
|---|---|---|---|
| 14:02:11 | gen_8f21c4 | True | Answer restates the refund window from the retrieved policy doc, no additions. |
| 14:01:58 | gen_8f21b9 | False | Quotes a $99 pricing plan. Retrieved context lists no such plan or price. |
| 14:01:40 | gen_8f21a2 | N/A | No context was retrieved for this question, so groundedness cannot be judged. |
What quality drop looks like
When you've collected enough of those scores, you get a pass rate: one number for how often your AI feature gets it right. If you watch a pass rate over time, a bad change stops being invisible.
In Fig. 2, volume and errors hold steady while the pass rate falls off at the moment a prompt changed. Without an evaluation running, the first you'd hear of that is a customer email, weeks later.
prompt v12
Generations
flat at ~9,000/day
Eval pass rate
94% → 71%
How to write evaluation criteria
Numbers are only worth watching if the eval behind it asks something real. Sentiment is built-in, so you only write the judge and the code-based eval yourself.
Asking "Is this good?" isn't going to cut it. Vague questions produce numbers that move for reasons you can't name. Ask for the specific failure you're worried about instead:
- For a support bot: does the answer contradict the retrieved context?
- For a code assistant: does it call a function that doesn't exist in the file?
- For a summarizer: does it introduce a name that wasn't in the source?
Each of those is checkable, so when the pass rate drops you know which behavior changed.
Split by model and prompt version
One pass rate covering every model, prompt, and surface will hide almost any regression. The drop in Fig. 2 only looks dramatic because one prompt broke on one model – averaged in with everything else that's working, it's hard to see what's going on.
Splitting by model is free: $ai_model is already on every generation.
Splitting by prompt version means being able to see which prompt version caused a drop in pass rate. Prompt management lets you create and update LLM prompts directly in PostHog. This lets you iterate on prompts without deploying code, and see which ones are causing issues.
Run your first eval
The prompt below sets up a judge on a retrieval-backed feature, with the criteria from earlier: does the answer contradict, invent, or refuse what the context supports? Swap those for the failure your product actually has – the structure is what matters.