mhl-testing

mhl testing

Assumes the mhl-language skill. mhl tests are .mh code: a test holds describe blocks, each a fresh scope running ordinary statements plus assertions. Tests must be hermetic: no real model, no network, no paid call, so they can run on every change.

Validation layers

Command	Proves	Cost
mhl lint <path>	syntax, types, names, imports, agent config	instant
mhl test <path>	behavior of tools, prompts, workflow control flow	fast, hermetic if you keep it so
mhl run x.mh --dry-run --input k=v	input contract and static step plan	instant, runs nothing
mhl run x.mh	the real thing	real agents, real effects

mhl test recurses a directory and exits 1 on any failure. Naming a single file with no test block is also an error. incomplete("reason") records a skip and never fails.

Treat a task as done when lint prints No problems found. and mhl test exits 0 with the new behavior covered by a test that failed before the change.

Assertions

are_equal(actual, expected), are_not_equal(a, b), is_true(v), is_false(v), is_null(v), not_null(v), greater_than(a, b), less_than(a, b), greater_than_or_equal(a, b), less_than_or_equal(a, b), includes(array, value), incomplete(reason).

are_equal is deep and order-sensitive. is_true errors on a non-bool, so compare first: is_true(x == 1). There is no raises assertion; use try/catch and assert the message.

Mocking agents

An agent is a subprocess, so a mock is a cheap command. Three shapes cover almost everything:

// 1. Echo: returns the built prompt, so a test can assert on its content.
agent Reviewer {
    command: "echo"
    args: ["${prompt}"]
}

// 2. Canned answer: a fixed response, for parsing and branching logic.
agent Classifier {
    command: "sh"
    args: ["-c", "printf '{\"label\":\"bug\"}'"]
}

// 3. Failure: exercises fallback and error paths.
agent Broken {
    command: "sh"
    args: ["-c", "echo overloaded >&2; exit 1"]
}

agent Resilient {
    command: "sh"
    args: ["-c", "exit 1"]
    fallback: [Reviewer]
}

workflow Triage {
    input title: string
    var label = ""

    step Classify {
        self.label = json.parse(Classifier.run(prompt: title)).label
    }

    step Review {
        log(Broken.run(prompt: title))
    }
}

test mocks {
    describe echo_exposes_the_prompt {
        is_true(Reviewer.run(prompt: "look at ${1 + 1}").contains("look at 2"))
    }

    describe failure_raises_when_called_directly {
        var message = ""
        try {
            Broken.run(prompt: "x")
        } catch (e) {
            message = "${e}"
        }
        is_true(message.contains("overloaded"))
    }

    describe fallback_takes_over {
        are_equal(Resilient.run(prompt: "x"), "x")
    }

    describe a_failed_step_is_reported_not_raised {
        var result = Triage.run(inputs: {title: "t"})
        are_equal(result.state, "failed")
        are_equal(result.step, "Review")
        is_true(result.error.contains("overloaded"))
    }
}