jev-judgment-design

Jev judgment design

Jev answers closed questions with probabilities and does not write prose. Moving an LLM pipeline's judgments to Jev makes them cheap and parallel, but Jev answers only within the state it is handed. This skill owns the design of that state and of the side that decides accept/reject from the answers. The general design guidance for Jev (state / instructions / criteria, choosing Choice, Noul, or Score, no-match, set thresholds on your own data) is canonical in plugin skill typesafe:typesafe-ai, so read that first and read only the judgments layered on top here.

Sources: the implementation in jev-research-pipeline (https://github.com/shimo4228/jev-research-pipeline) and the article "Moving the research judgments I left to an LLM to Jev, a judgment-only model" (https://zenn.dev/shimo4228/articles/jev-research-judgment-offload).

1. Move only closed judgments

Move to Jev the judgments whose answer can be stated as a yes/no probability, one pick from a set of options, or a point on a graded scale. Keep work with an open-ended answer shape, such as where to search next or how to write, with the LLM. After the move, watch the instrument "number of LLM judgments during a run", not "number of questions to Jev" — Jev's question count may grow by as many judgments as were moved.

2. Put what the judgment is made against into state

"Is it relevant" lets through anything that shares vocabulary when the comparison target is not in state. Jev is answering exactly what it was asked, so the fix belongs in state, not in the question. Make the comparison target the question that theme is currently seeking an answer to, and pass scope, exclusions, and already-accepted evidence along with it.

state = {
    "question": {
        "title": question.title,        # the question currently seeking an answer
        "brief": question.brief,        # scope
        "evidence": question.evidence,  # what counts as evidence
        "not": question.negative_topics,  # topics that overlap in vocabulary but are out of scope
    },
    "source": {"title": ..., "url": ..., "excerpt": ...},
    "evidence_set": accepted_claims,    # evidence already accepted for this question (the baseline for novelty)
}