Catch a code assistant inventing functions

The coding assistant you built suggests a call to formatCurrency(), a function that doesn't exist. The syntax of the call is valid, the tone of the conversation is confident, and the call looks like it succeeded.

This use case walks through one code-based evaluation end to end: the eval itself, the generations it can't score, and what its pass rate does when a prompt change goes wrong.

What an eval can answer

Whether the assistant called something that doesn't exist is a yes or no question. You can answer it by comparing the file it was given to the code it wrote back.

That's what a code-based eval is perfect for: code you write that runs against each generation and returns pass or fail. No model is involved, so it costs nothing to run. You can score every generation rather than sampling a slice of them, and a failure that only turns up a few times is still visible.

Write the eval

The eval runs inside PostHog rather than on your servers, so you write it in Hog, the language PostHog compiles and runs itself. Nothing of yours has to be awake for a generation to get scored.

You don't have to write it by hand either. Generate with AI in the code editor opens PostHog AI with your evaluation loaded and drafts the Hog for you. The prompt at the end of this guide takes the same route.

The eval hands three things to you:

Return true to pass and false to fail. Whatever you print() becomes the reason stored next to the result, so print something you'd want to read a week later.

Start with one function you keep seeing invented. The eval looks for a call to it in the output, then looks for the same name in the file the model was given. In one and not the other means the model made it up.

let context := ifNull(input, '')
let suggestion := ifNull(output, '')

// Asked with no file attached, so there is nothing to check the call against
if (length(context) == 0) {
    print('No file context on this generation')
    return null
}

// Called it, but the file it was given never defines it
if (suggestion ilike '%formatCurrency(%' and context !ilike '%formatCurrency%') {
    print('Called formatCurrency, not defined in the provided context')
    return false
}

print('No calls to undefined helpers')
return true

Add the next function as another if. Once you're checking a dozen of them, pull the names into a list and loop over it.

Settings the eval needs

If someone asked a question with no file attached, there's nothing to compare the call against. Marking it pass or fail either way puts a number in your pass rate that isn't true, which is why the eval returns null there.

The code is only part of an eval. These settings decide what it runs against:

Then, test it on a sample. This runs the eval against a subset of recent generations and shows the pass, fail, or N/A each one would get, along with anything you printed. Nothing is written, so you can keep editing until the reasons read the way you want.

Read the runs

Every result lands on the generation it scored:

An example of these results is outlined in Fig. 1 below.

When it starts failing

Once enough generations have been scored you have a pass rate: how often your assistant only calls things that exist.

In Fig. 2, generations hold flat at around 9,000 a day and nothing errors, while the pass rate drops from 94% to 71% the day prompt v12 shipped. That prompt told the model to reuse existing helpers where it could, so it invented them to comply. Without the eval running, the first you'd hear is a developer complaining weeks later. You can also split the pass rate by $ai_model and $ai_prompt_version to see which one broke.

Build it with PostHog AI

This prompt builds the evaluation from this guide:

Swap the function list for the ones your own assistant keeps inventing:

Create a code-based (Hog) evaluation on my $ai_generation events for a code assistant. It should return null when the generation has no input context to check a call against, and fail when the output calls a function that appears nowhere in that context, printing the function name as the reason. Enable Allows N/A, sample at 100%, exclude generations where the environment property is dev or local, and show me the pass rate over the last 30 days broken down by $ai_model and by my prompt-version property if my generations have one. If my project has no real $ai_generation traffic yet, create the eval disabled anyway, skip the breakdowns, and tell me how to instrument AI observability using the wizard or manual install.

npx @posthog/wizard ai-observability