Evaluating

Scored evaluations with a judge, and reading the score honestly.

How well does the agent answer? Said in a file beside the flow and run by lgflow eval.

A test and an eval are different questions#

flow.json says what the agent is. A *.test.json beside it says what the agent must do. A *.eval.json asks something neither can: whether what it did was any good.

They are not two spellings of one idea. A test is cheap, deterministic and belongs in CI on every commit; an eval calls a model twice over and tells you whether yesterday's prompt change made the agent better or worse. Keep both.

customer-agent/
├── flow.json              the agent
├── tests/
│   └── routing.test.json  what it must do
└── evals/
    └── answers.eval.json  how well it does it

Nothing about evals goes in flow.json. A flow is independent of the suite that measures it, the same way it is independent of the suite that tests it.

The smallest eval that says something#

{
  "flow": "../flow.json",

  "evals": [
    {
      "name": "Where a customer is",
      "input": { "question": "Where is ALFKI located?" },
      "reference": { "facts": ["City: Berlin", "Country: Germany"] },
      "criteria": {
        "correctness": { "description": "The answer agrees with the reference facts." }
      }
    }
  ]
}

lgflow eval evals/answers.eval.json

The flow is compiled and run for real — the real handlers, the real model, a fresh conversation per case — and then a judge is asked, once per criterion, how well the answer met it.

Criteria#

A criterion is a name and a description. The description is the whole of what the judge is told, so write it as an instruction rather than a label:

"criteria": {
  "correctness": {
    "description": "The answer agrees with the reference facts, and answers the question asked.",
    "weight": 0.5
  },
  "groundedness": {
    "description": "Every claim comes from the record the desk found. Anything else is unsupported.",
    "weight": 0.3
  },
  "conciseness": {
    "description": "It answers in a sentence or two, with no preamble and no detail nobody asked for.",
    "weight": 0.2
  }
}

Each is judged on its own, in its own call. Asked about three at once a model averages them in its head and hands back one number wearing three names.

Weights are optional, and all or nothing. With none, a case's score is the plain mean of its criteria. With them, it is the sum of score × weight, and they must add up to 1 — a set that adds up to 1.4, or one where two of three carry a weight, is refused before anything runs rather than quietly normalised.

Criteria may be said once for the whole file and applied to every case, or per case, or both — where a case names one the file already names, the case wins.

Reference information#

A case may say what is true, so the judge has something to judge against:

"reference": {
  "facts": ["City: Berlin", "Country: Germany"]
}

or the answer it would give itself:

"reference": { "answer": "ALFKI is Alfreds Futterkiste, in Berlin, Germany." }

Both are supported, and both may appear together. facts is the one to reach for: it is what "grounded" means — the answer may say these things and nothing else about the customer — and it does not invite the judge to mark the agent down for saying the same thing in different words. answer is passed as one good answer, not the only one, and the judge is told so.

A reference is never required. Conciseness and tone do not need one.

Metrics#

What the run cost, measured rather than judged:

"metrics": {
  "latencyMs": { "max": 5000 },
  "tokens":    { "max": 2500 },
  "toolCalls": { "max": 4 },
  "llmCalls":  { "max": 3 }
}

These come from the run itself — the timer, the compiler's own tracer, and the token usage the model client reports. Nothing is instrumented twice and nothing is estimated. Each is reported with what happened beside what was asked:

"latencyMs": { "actual": 1840, "max": 3000, "passed": true }

A client that reports no usage leaves tokens unavailable, and a metric that could not be measured does not pass:

"tokens": { "available": false, "max": 2500, "passed": false, "why": "tokens was not reported by the model client" }

Counting an unknown as zero would turn every such run into a perfect score on cost.

Datasets#

The same criteria over many cases: put the cases in a file of their own.

{
  "flow": "../flow.json",
  "dataset": "./customer-questions.json",
  "criteria": {
    "correctness":  { "description": "The answer is correct." },
    "groundedness": { "description": "The answer is supported by the customer data." }
  }
}

[
  { "name": "Customer location", "input": { "question": "Where is ALFKI located?" },
    "reference": { "facts": ["City: Berlin", "Country: Germany"] } },
  { "name": "Customer contact",  "input": { "question": "Who is the contact at ANATR?" },
    "reference": { "facts": ["Contact: Ana Trujillo"] } }
]

The rule is one or the other: evals is an inline suite, dataset is a file of cases, and a file that says both is refused. A merge order nobody could guess from reading the file is worse than an error.

A dataset's cases are held to exactly the rules an inline case is — same keys, same names, same refusals.

Thresholds#

A suite may say what it has to reach:

"thresholds": {
  "overall": 0.85,
  "criteria": { "correctness": 0.9, "groundedness": 0.9 }
}

The run then produces both a measurement and a gate: lgflow eval exits non-zero when a threshold is missed, which is what CI reads. A threshold does not turn an eval into a test. The criterion is still a score; the threshold only says whether this suite considers that score enough to ship. A suite with no thresholds measures and reports, and leaves the judgement to whoever reads it.

A case that could not be run fails the gate whatever the rest scored.

When something goes wrong#

Three failures are told apart, and none of them is a score:

A score of 0 means the agent answered and the answer was worthless. It never means the evaluation could not be performed — an infrastructure failure quietly recorded as zero is how a suite reports a broken API key as a quality problem.

The judge#

The judge is a model handed to the runner, never named in the eval file:

import { runFlowEvals } from "lgflow/evals";

const result = await runFlowEvals("./evals/answers.eval.json", {
  llm,     // what the flow runs on
  judge,   // what scores it - may be the same one
  concurrency: 3,
});

Both are (prompt, label, { onUsage }) => Promise<string> — the same shape buildGraph takes — so anything that can run a flow can judge one. The eval file says what to judge; which model does it is runtime configuration, like every other model choice in LGFlow.

lgflow eval builds both from the environment, the way lgflow generate builds its own.

Its answer is structured and checked by the same code that checks a v0.15 answer schema:

{ "score": 0.94, "reason": "It names Berlin and Germany and claims nothing else." }

A judge that will not answer in that shape is a judge_error. No prose is parsed for a number.

What comes back#

{
  "at": "2026-10-06T09:14:00.000Z",
  "build": "97b9cf7813cde16b",
  "total": 4,
  "completed": 4,
  "summary": {
    "score": 0.93,
    "criteria": { "correctness": 1, "groundedness": 0.85, "conciseness": 0.85 },
    "averageLatencyMs": 1840,
    "usage": {
      "agent": { "tokens": 1350, "available": true },
      "judge": { "tokens": 4080, "available": true },
      "total": { "tokens": 5430, "available": true }
    }
  },
  "thresholds": { "passed": false, "reasons": ["\"groundedness\" 0.85 is under 0.9"], "overall": 0.8 },
  "evals": [
    {
      "name": "Where a customer is",
      "input": { "question": "Where is ALFKI located?" },
      "status": "scored",
      "score": 1,
      "criteria": { "correctness": { "score": 1, "reason": "…" } },
      "metrics": { "latencyMs": 1720, "tokens": 450, "llmCalls": 1, "toolCalls": 0 },
      "checks": { "latencyMs": { "actual": 1720, "max": 20000, "passed": true } },
      "output": { "answer": "…" },
      "usage": { "agent": { "tokens": 450, "available": true }, "judge": { "tokens": 1020, "available": true } },
      "events": [ … ]
    }
  ]
}

Agent usage and judge usage are kept apart, because judging a suite can cost more than running it and one total would hide which half to go and look at.

Every case is identified by its name, which is unique and checked — never by its position — so a later run can be matched against this one case by case.

build is which version of the flow was measured - see *Which build it scored* below - and recorded is where the run file went, when one was kept.

Mocks#

An eval may stand in for a tool or a read, with the same mock a test uses:

{ "name": "…", "input": { … }, "mock": { "tools": { "chargeCard": { "ok": true } } } }

Use it for what you do not want an eval to really do — take a payment, send an email. Do not mock the model. An eval of a mocked answer measures the mock: it is useful for testing the eval framework itself and says nothing about the agent's quality.

Before you spend anything#

An eval costs money and a test does not, and most of what a first eval run is spent on is knowable without spending any: a criterion written as a label rather than an instruction, a case that mocks nothing and will call a tool for real, handlers that are not where the file says they are.

--preview runs every check a run runs and stops before a model is built. It needs no key, and it costs nothing:

lgflow eval evals/answers.eval.json --preview
lgflow eval evals/answers.eval.json --no-journal

LGFlow eval - preview. Nothing was run and no model was called.

  Where a customer is               3 criteria
  Who to talk to                    3 criteria

What one press would do

  Agent runs                        4
  Judge calls                       12
  Judge input                       ~5,575 tokens (an estimate)

Then it prints the prompt each judge would be asked, with one hole in it where the answer will go. That hole is deliberate: a preview that invented an answer would be making the same mistake as an eval that mocked one. Reading the prompts is the point — a criterion that reads as a label is obvious in its own prompt and invisible in the score it produces.

Because it is the same code, a preview that accepts a suite is a promise about the run and not a second opinion.

In LGStudio the Tests panel has a Preview button beside Score, which shows the same thing.

The story each case tells#

A case is an input, a set of mocks, a list of facts and a set of criteria, and reading one in the file means holding four things at once. So the preview lays each one out as what it does, in order:

• What is asked — the input, in its own words.
• What stands in for the real thing — each mocked read and tool, what it returns and how many times; anything nothing stands in for, which is called for real; and the one thing never stood in for, the model.
• What is taken as true — the reference facts, and the reminder that they are all the judge is told.
• What the judge is asked — each criterion with its weight, and how many separate calls that comes to.
• What must hold whatever the score is — the metrics and the thresholds.

Every line comes from the case. Nothing is invented and no model is called, because this is read before anybody has paid for anything. In LGStudio the Preview button shows the cases on one side and the chosen one's story on the other; on the command line they print in the same order.

What a run leaves behind#

A test run is free and repeatable: to know what happened, run it again. An eval run is neither — it costs money and it is not deterministic — and the question you want afterwards, why was that case 0.3, is the one the panel forgets the moment it closes.

So every run writes itself down, in a .lgflow folder beside the suite:

flows/evals/.lgflow/journal.jsonl              every model call, agent and judge
flows/evals/.lgflow/runs/<suite>-<when>.json   the whole result of one run

The journal is the same one a deployed agent keeps, written by the same code. Each line says which case it belonged to and whether it was the agent or the judge, so why was this case 0.3 is answered by reading what its judge was actually sent — the prompt and the reply are both there.

The run file is the structured result exactly as --json prints it: each case with its answer, its scores and the judge's reasons, its metrics, its steps, and any mock that never fired. The newest twenty runs per suite are kept, so a suite's history is a folder you can diff.

Both are best-effort and neither can fail a run — losing a paid run over a read-only folder would be absurd. --no-journal turns them off, and a spec handed straight to runEvals in memory keeps nothing, because it has nowhere of its own to keep it.

Which build it scored (v0.21)#

A run also records which version of the flow it was measuring:

{ "build": "97b9cf78a1e4...", "summary": { "score": 0.82 } }

That is hashOf(flow) - the same hash a deploy writes into the project's .lgflow/manifest.json, by the same function. So three things that used to be separate are now one question:

A path could not do any of this. Two runs of the same file are the same path and almost never the same flow, and a flow edited between them is exactly the case worth noticing.

A build scored more than once shows its latest score and says how many times it was scored. A suite with a judge in it does not return the same number twice, and an average of two attempts is not a thing anybody asked for. Runs saved before this existed are left out of the grouping: they scored something, and nothing can say what.

LGStudio's test panel draws this under each suite - the last score on the suite's own line, and the builds under it with how far each moved from the one before. It appears once there are two, because one build is a score and the line above already says the score.

A mock that never fires#

The quietest way for an eval to go wrong is a mock that is never used. The file says "when it reads the schema, give it this"; the flow takes a route that never reads at all; everything after that is about a document that was never there. Nothing fails. The answer is simply about something else, and the score is low for a reason no judge can explain.

So every case reports the mocks the run never used:

  all customers from the UK — Unused reads mock
    this case stood in for load_schema, and the run never used it. The route
    that would have was not taken, so everything after it is about something
    else.

It is not a failure — a mock for a route this case did not take is perfectly reasonable — but it is never silent. It is the first thing to check when a score is nothing like the one you expected.

Concurrency#

Cases run two at a time by default, and --concurrency changes it. A dataset of 500 cases launched at once is a large bill arriving in one second, so nothing here defaults to unbounded parallelism.

lgflow eval evals/answers.eval.json --concurrency 4

The command line#

lgflow eval evals/answers.eval.json
lgflow eval evals/answers.eval.json --json
lgflow eval evals/answers.eval.json --concurrency 4
lgflow eval evals/answers.eval.json --preview

LGFlow eval

✓ Where a customer is               1.00
✓ Who to talk to                    1.00
✗ How much they have ordered        0.76
✓ A customer it has never heard of  0.94

Criteria

  Correctness                       1.00
  Groundedness                      0.85
  Conciseness                       0.85

What fell short

  How much they have ordered — Groundedness: 0.40
    "our oldest customer in Sweden" is not in the reference facts.

Run

  Cases                             4
  Average score                     0.93
  Average latency                   1.3s
  Agent tokens                      703
  Judge tokens                      7,499
  Total tokens                      8,202

Threshold

  Overall                           0.80
  Groundedness                      0.90

  "groundedness" 0.85 is under 0.9

FAIL

Every score that fell short is explained in the judge's own words. The terminal report is a convenience; --json is authoritative.

From LGStudio#

The Tests panel scores as well as runs. Score, beside Run, finds the suites about the flow on screen, says how many cases and how many model calls that is, and waits for a yes — reading a suite is free, running one is not. Run does the tests first and then offers the evals, so both halves of the question are one press apart.

Each case opens on what the judge said: what was asked, the reference facts, what the agent answered, every criterion with its score and its reason, and the run's latency, tokens and calls.

Test Scenarios writes both files. From v0.18 the test writer proposes an eval suite beside the tests — criteria that suit the flow, a case for each thing worth judging — checked by the checker on this page before you see it, shown in the same table marked EVAL:, and written when you approve. It runs the tests but never the evals: nobody should be charged for a proposal they have not read.

A worked example#

examples/answer-desk is a desk that answers questions about five customers from a directory its code holds. It ships with both:

• tests/answering.test.json — it routed to the answering node, it never asked a model about a customer it had not found, the company name finds them as surely as the id does.
• evals/answers.eval.json — was the answer correct, grounded and short?

The same agent, two questions, two files.