Testing

Scenarios over a flow, and what a test can and cannot tell you.

What a flow is expected to do, said in a file beside it and run by lgflow test.

A test asks whether the flow behaved: did this route run, was that tool called, does this channel hold that. Pass or fail, and nothing leaves the machine. Whether what it did was any good is a different question, with a file and a runner of its own — see Evaluating a flow.

flow.json says what the agent is. A *.test.json beside it says what the agent is expected to do:

examples/structured-classifier/
  flow.json
  handlers.js
  tests/
    routing.test.json

{
  "flow": "../flow.json",
  "tests": [
    {
      "name": "Billing request routes to the billing answer",
      "input": { "question": "Why was my credit card charged twice?" },
      "mock": {
        "nodes": {
          "classify": { "category": "billing", "confidence": 0.98, "reason": "A duplicate charge." },
          "answer_billing": "Send us the dates on the statement."
        }
      },
      "expect": {
        "state": { "classification.category": "billing" },
        "path": ["classify", "answer_billing"],
        "output": { "answer": { "notEmpty": true } }
      }
    }
  ]
}

$ lgflow test examples/structured-classifier/tests/routing.test.json
LGFlow tests

✓ Billing request routes to the billing answer
✓ Technical request routes to the technical answer
✓ Anything else routes to the general answer
✓ An answer of the wrong shape is not classified, and a person is told

4 passed
0 failed
4 total

Exit 0 when everything passed, 1 when anything failed or a file could not be read. --json prints the result instead. From code it is import { runFlowTests } from "lgflow/testing", which returns that same result and prints nothing — the console output is the CLI's, not the API's.

A test runs the real flow. The real compiler builds the real graph with the real handlers and the real routers; the v0.15 schema check is the real one. Nothing here re-implements what a flow does, so what a test proves is what the flow does.

Mocks replace the dependency, not LGFlow. Each goes exactly where the real thing is injected, which is what keeps everything downstream of it honest:

So a mocked structured answer must still satisfy its schema. Mock classify with { "category": "banana" } against an enum of three and the node fails exactly as a real model would have made it fail — retry retries, onError catches, nothing reaches state. There is no test-only road into a channel.

A model or a tool with no mock is an error, not a real call: a test that stands still is worth more than one that quietly costs money. An approval nobody speaks for is approved, so a test about something else is not blocked by a card it never mentioned.

What a test may expect — and nothing else, an unknown expectation being refused by name the way the DSL refuses an unknown keyword:

• state — dotted paths and what they must hold, "results.0.title" included. null means never written; { "notEmpty": true } means something. Anything else is compared as a value - 4444, "ALFKI", a whole object - and there are no other matchers. There is no equals, no contains, no greaterThan.

An invented one is now refused by name (0.35.5), the way an unknown expectation is: "amount" says { "equals": ... }, and "equals" is not a matcher. This paragraph said the opposite until that day - that telling an invented matcher from { "notEmpty": true } would mean having a vocabulary of matchers, which this deliberately does not. What changed the decision is where the mistake turned up: not in a scenario anybody was reading, but in a deployed agent's own test file, where it printed expected {"equals":4444} / actual 4444 over a flow that was right, in a deploy log, three times - and the first of those three times the handler underneath really was reading 444. One red line for two different faults.

The list is of assertion words - equals, contains, toEqual, greaterThan and the rest - and it exists so they can be refused, never honoured. value, type, length, name and id are deliberately not in it: a channel holds whatever the flow put there, and refusing { "value": 42 } would refuse a real expectation about real data. Two keys or more is an object being compared as one, and is left alone.

• output — the same, over the flow's declared outputs.
• path — the nodes that ran, in order. A fan-out's copies count as one entry: which node ran is deterministic, which copy finished first is not.
• toolCalled / toolNotCalled — a name or a list.
• approvalBefore — that node was put to a person and did not run before they answered. Approved-and-ran and declined-and-never-ran both keep that promise; what it rules out is the node running with nobody asked.
• fails — true, or the text the failure must contain.

The observation is the tracer the compiler already calls — the same trace option the chat page uses — plus a wrapper around the injected llm and callTool, which is how serve.js has always watched a run. Nothing is written to disk and nothing is read back: a test watches the run it started.

A test file is checked against the flow before it runs — a mocked node that is not there, a mocked tool the flow never declares, a path through a node that does not exist — and says all of it at once.

examples/structured-classifier/tests/ and examples/approval-desk/tests/ are the two shipped ones: routing on a structured answer, and a refund that waits for a person.

What a test may also say — v0.17#

• settings — a flow that cannot run until one is set is set from the file: "settings": { "flowsRoot": "../.." }, and a test may override its own. Only settings the flow declares.
• handlers — where the code is, when it is not beside the flow. A flow kept in flows/ is generated into a project of its own, so its test file has to say where: "handlers": "../../../dbconnect/src/registry.js". Read from the test file, exactly as flow is.
• { "fails": "why" } as a mocked tool result makes the call go wrong, which is the only way to reach a flow's retry and onError.
• A list of answers for a node that stops more than once: a tools node completes its calls — forms answered — before it asks whether to make them, so "add": [{ "customerId": "LGFLW" }, false] fills the form in and then declines.
• { "then": [ … ] } as a mocked tool result too, for a tool a tools node calls more than once in a run: what it answers the first time, the second, and so on — any one of which may be { "fails": "…" }, so one call goes wrong while the others do not. A bare list is still one result that happens to be a list.
• mock.reads — by node id, the document a read node got: a string for its text, an object for the whole document, { "fails": "why" } for a document that cannot be read at all. A read node reaches outside the machine without being a tool, so it is stood in for like one. Until this, a test of a flow with a read node really went to the network.
• { "then": [ … ] } as a mocked model answer, for a node on a loop: what it answers the first time, the second time, and so on. One answer is given again every time round, so the loop can only end by running out of a route limit — which reads like a broken flow when it is a test that could not say what it meant.
• A step inside a flow this one runs — find_customer/look_up, named the way a trace, the journal and the canvas name it. A test may stand in for a model inside a child and expect a path through one. The flow node itself still cannot be mocked: it is a flow running, not a model answering, and the refusal says so.
• What a run did, reported with every test as run.events: each node in the order it ran, the route a router took and where it went, a fan-out and how many copies, a tool called, a document read, an approval answered, a retry, and what stopped the run when something did. A model step carries the prompt that node built - a test stands in for every answer, so nothing calls a model and this is the only place that prompt is ever seen. LGStudio folds it under each test and opens it for one that failed.

A node that never finished is not in path: one whose approval was declined, and one that failed, are both absent. A tools node never fails the run either: it writes a failing call's error beside that call in its output and carries on, so a test about that expects "results.0.error" rather than fails, and one that expects fails is refused before it runs.

Only an llm node has an answer to stand in for. A flow with none, and no tools — a desk built out of handlers and other flows — needs no mock at all: it is tested by what goes in and what comes out.

Having them written for you — v0.17#

examples/test-writer is an LGFlow agent that writes LGFlow tests. Give it a flow and it reads that flow and the DSL reference's own topic on tests, read at run time rather than copied, proposes scenarios, runs them, and keeps at it until they pass — anything it cannot make pass is taken out rather than handed over failing. It shows what is left in plain words, and writes the file only when you say yes.

npm run example:test-writer -- examples/approval-desk/flow.json

From LGStudio it is a button: Tests → Test Scenarios writes the tests for the flow on screen. Deploy examples/test-writer/flow.json first — it runs as its own project, like any other agent.

Deploy it again when these rules change. A deployed project vendors the LGFlow it was built with, so the writer checks what it writes by that one's rules while lgflow test uses this one's — and a check added since will refuse what the writer was happy with. The version cannot say: 0.17.0 stays 0.17.0 when a rule changes. So a deploy records lgflowSource in the project's manifest — a hash of src/, from sourceStamp() — and isBehind(stamp) says whether a project is running on older rules. LGStudio asks before it starts the writer, and says so.

Whose flow is this — v0.17#

A flow says what kind of file it is:

{ "version": "0.17", "kind": "system", … }

system means something reads it rather than learns from it — the DSL reference's worked flows, the test writer LGStudio runs. LGStudio draws no delete beside one and its server refuses to delete it, because losing one takes a feature away rather than a demonstration. user is the other, and is what saying nothing means: every flow written before this says nothing, and reads exactly as it did.

It changes nothing about how a flow runs, so it is not versioned — a flow at 0.1 may say it. But a word that is neither is refused (INVALID_KIND), because a typo would quietly read as "not protected". isSystemFlow(flow) answers it, and it is said in the flow and nowhere else: no list to drift out of step with the files.