mclass

A 22M extract-then-classify model for coding preferences. It reads user text, harness messages, and code hunks. It writes typed labels with a percent.

The extractor copies candidate text from the user or from a diff template. The encoder does not invent a preference sentence.

• Code: https://git.simonharms.com/thesimonharms/mclass
• Weights: https://huggingface.co/thesimonharms/mclass
• License: MIT

I/O

Stdin and POST /v1/classify share one JSON object.

{
  "user": "always use bun instead of npm",
  "harness": [{"role": "assistant", "content": "..."}, {"role": "tool", "name": "edit", "content": "edited package.json"}],
  "code": [{"path": "package.json", "before": "\"packageManager\": \"npm\"", "after": "\"packageManager\": \"bun\""}]
}

{
  "learnings": [{
    "preference": true,
    "text": "always use bun instead of npm",
    "kind": "durable",
    "category": "tooling",
    "confidence": 0.91,
    "confidence_pct": 91,
    "source": "user"
  }]
}

• preference is true only when kind is durable
• kind: durable | session | task | correction
• category: cli | language | architecture | testing | style | tooling | naming | git | docs | other
• confidence is P(durable) after temperature scaling, 0 to 1
• source is user, clause, or diff
• Schemas: schemas/classify.input.json, schemas/classify.output.json

The extractor drops lockfiles and secret names (.env, *.pem, id_rsa, credentials). Identical hunks are dropped.

Weights

This git repo does not store the trained files. Download them from Hugging Face:

hf download thesimonharms/mclass --local-dir artifacts

You need encoder/, heads.pt, and mclass.json. model.onnx is optional and is the default runtime when it is present.

CLI

uv sync --extra train --extra dev
hf download thesimonharms/mclass --local-dir artifacts

echo '{"user":"I want bun"}' | uv run mclass classify
uv run mclass serve --port 8091

Train from this tree:

uv run python scripts/synth.py
uv run python scripts/train.py
uv run python scripts/calibrate.py
uv run python scripts/eval.py
uv run python scripts/export_onnx.py

GET /health and POST /v1/classify are the HTTP hook. This is not a GGUF. llama.cpp has no classify kind. A later malaikat profile should spawn mclass serve and proxy that route.

--durable-only keeps learnings with preference: true. --torch skips artifacts/model.onnx.

SMOKE_TEST=1 uv run python scripts/train.py runs one step on 32 rows.

Train defaults: mean-pool MiniLM, LoRA rank 8 on the last 2 layers (--lora-r 8 --unfreeze-last 2). Full encoder backward can still nan on gfx1151. Use --lora-r 0 --unfreeze-last 0 for heads only.

The held-out wording set is data/ood.jsonl. scripts/eval.py also scores gold and a naive-Bayes baseline on user text.

Layout

src/mclass/     extract, pack, two-head model, infer, CLI
scripts/        synth, train, calibrate, eval, ONNX export
data/           seed.jsonl, gold.jsonl, ood.jsonl; train/val/test are generated
artifacts/      download from Hugging Face (encoder, heads.pt, mclass.json, model.onnx)

Base encoder: sentence-transformers/all-MiniLM-L6-v2 (22M). The forward pass mean-pools token states. On a Radeon 8060S, full encoder backward produces NaN grads. LoRA on the last two MiniLM layers stays finite and updates those query/value weights.

ModernBERT-base is the longer-context option: --base answerdotai/ModernBERT-base --lora-r 0 --unfreeze-last 0.

uv pip install --index-url https://rocm.nightlies.amd.com/v2/gfx1151/ --prerelease=allow --reinstall torch 'rocm[libraries]'

torch.cuda.is_available() is true on that ROCm wheel. The default PyPI torch wheel is CUDA and will not see this GPU.

malaikat

Do not add a profile in the malaikat tree from this repo. Consume mclass as:

1. mclass serve --host 127.0.0.1 --port 8091 --artifacts /path/to/artifacts
2. POST http://127.0.0.1:8091/v1/classify with the input object above
3. Keep learnings with preference: true and confidence_pct at or above your threshold (mcode durable capture used 55)

Eval

data/gold.jsonl is a hand set. data/ood.jsonl is held out of train/val/test. After train:

uv run python scripts/eval.py

artifacts/eval.json holds kind accuracy, durable F1, category accuracy, ECE, the naive-Bayes comparison on the user-text slice, gold, and OOD.

Measured after LoRA train (mean pool, last-2 adapters, 3 epochs):

slice	metric	encoder	same-data Bayes
test	kind acc	0.998	—
test	durable F1	0.997	—
user-text	durable F1	0.995	0.687
gold (n=65)	kind acc	0.969	—
gold	preference acc	0.985	—
ood (n=78)	kind acc	0.987	—
ood	preference acc	0.987	—

The OOD set includes general questions ("What is the capital of France?", "Who wrote Hamlet?"). An earlier train had no such negatives, so the model read those as durable preferences at 0.999 confidence. scripts/synth.py now generates general questions and requests as task negatives, and data/ood.jsonl holds them out to measure it.