mverify

A CPU classifier for the mcode verification pass. It scores one pair: does this output satisfy this prompt?

It is not an LLM. It is not a general correctness judge. It does not run tests or builds. The harness does that work.

• Code: https://git.simonharms.com/thesimonharms/mverify
• Weights: https://huggingface.co/thesimonharms/mverify
• License: MIT
• Contract: v1 (additive JSON fields only)

I/O

mverify check writes one JSON object on stdout.

mverify check --prompt "Write a Python add function" --output "def add(a, b): return a + b"
# or:
echo '{"prompt":"...","output":"..."}' | mverify check --stdin

{
  "pass": true,
  "confidence": 0.93,
  "reasons": ["output has a code block in the requested language"]
}

• pass: did the output do the thing the prompt asked
• confidence: P(output satisfies prompt), 0 to 1
• reasons: short strings, one per salient factor. Empty array allowed

Exit code 0 means the CLI ran. pass can still be false. A CLI failure writes {"error":"..."} to stderr and exits non-zero.

Schemas: schemas/check.input.json, schemas/check.output.json.

mcode should treat a hit as pass && confidence >= confidenceThreshold (default 0.7).

Weights

This tree includes artifacts/mverify.npz, vocab.json, and mverify.json. The same files are on Hugging Face:

hf download thesimonharms/mverify --local-dir artifacts

CLI

uv sync --extra dev
uv run mverify check --prompt "What is 6*7?" --output "42"
uv run mverify train --data data --out artifacts
uv run mverify eval --data data --artifacts artifacts

Train from this tree

uv run python scripts/synth.py
uv run python scripts/train.py
uv run python scripts/eval.py

scripts/synth.py builds (prompt, output, pass|fail) rows. Good rows come from templates plus mined Cursor / mcode sessions. Fail rows use the malaikat failure set: off-topic, empty, truncated, ignored constraint, wrong format, different question, confident refusal. Live malaikat stays off so the GPU stays free for mcode. The generator is local and uses a fixed seed.

Held-out eval is data/eval.jsonl (~500 pairs, blocked by prompt family, stratified by category). data/gold.jsonl is a small hand set.

The model is TF-IDF plus logistic regression on packed prompt/output text, plus pair features (overlap, format gaps, refusal, language). Inference is numpy on CPU. There is no GPU path.

SMOKE_TEST=1 uv run python scripts/train.py runs one epoch.

Layout

src/mverify/    features, TF-IDF logreg, infer, CLI
scripts/        synth, train, eval
data/           gold.jsonl, eval.jsonl; train/val/test are generated
artifacts/      mverify.npz, vocab.json, mverify.json, eval.json

Eval

Targets: blatant-fail recall (off-topic, empty, ignored constraint) at or above 0.95. False-fail rate on the good slice below 0.05. p95 latency under 200 ms on CPU.

Measured after scripts/train.py (seed 42) on the held-out eval set:

slice	n	accuracy	fpr on good	blatant fail recall	p95 ms
eval	500	0.860	0.030	0.956	0.22
same-data Bayes	500	0.794	0.758	0.918	—

Per-category fail recall on eval:

category	n	fail recall
off_topic	62	1.000
empty	62	1.000
ignored_constraint	59	0.864
truncated	63	0.952
wrong_format	62	0.839
different_question	63	0.254
confident_refusal	63	1.000
pass (accuracy)	66	0.970

artifacts/eval.json holds the full report. Different-question is the hard remainder. The harness should keep executable checks for facts a classifier cannot see.

What this is not

• Not a judge of subtle correctness ("the sort is O(n log n)")
• Not a code runner
• Not a safety filter (that is a later verifier, e.g. msafety)