mclass

A 22M extract-then-classify model for coding preferences. It reads user text, harness messages, and code hunks. It writes typed labels with a percent.

The extractor copies candidate text from the user or from a diff template. The encoder does not invent a preference sentence.

• Code: https://git.simonharms.com/thesimonharms/mclass
• Weights: https://huggingface.co/thesimonharms/mclass
• License: MIT

I/O

Stdin and POST /v1/classify share one JSON object.

{
  "user": "always use bun instead of npm",
  "harness": [{"role": "assistant", "content": "..."}, {"role": "tool", "name": "edit", "content": "edited package.json"}],
  "code": [{"path": "package.json", "before": "\"packageManager\": \"npm\"", "after": "\"packageManager\": \"bun\""}]
}

{
  "learnings": [{
    "preference": true,
    "text": "always use bun instead of npm",
    "kind": "durable",
    "category": "tooling",
    "confidence": 0.91,
    "confidence_pct": 91,
    "source": "user"
  }]
}

• preference is true only when kind is durable
• kind: durable | session | task | correction
• category: cli | language | architecture | testing | style | tooling | naming | git | docs | other
• confidence is P(durable) after temperature scaling, 0 to 1
• source is user, clause, or diff
• Schemas: schemas/classify.input.json, schemas/classify.output.json

The extractor drops lockfiles and secret names (.env, *.pem, id_rsa, credentials). Identical hunks are dropped.

Weights

This git repo does not store the trained files. Download them from Hugging Face:

hf download thesimonharms/mclass --local-dir artifacts

You need encoder/, heads.pt, and mclass.json. model.onnx is optional and is the default runtime when it is present.

CLI

uv sync --extra train --extra dev
hf download thesimonharms/mclass --local-dir artifacts

echo '{"user":"I want bun"}' | uv run mclass classify
uv run mclass serve --port 8091

Train from this tree:

uv run python scripts/synth.py
uv run python scripts/train.py
uv run python scripts/calibrate.py
uv run python scripts/eval.py
uv run python scripts/export_onnx.py

GET /health and POST /v1/classify are the HTTP hook. This is not a GGUF. llama.cpp has no classify kind. A later malaikat profile should spawn mclass serve and proxy that route.

--durable-only keeps learnings with preference: true. --torch skips artifacts/model.onnx.

SMOKE_TEST=1 uv run python scripts/train.py runs one step on 32 rows.

Train defaults: mean-pool MiniLM, LoRA rank 8 on the last 2 layers (--lora-r 8 --unfreeze-last 2). Full encoder backward can still nan on gfx1151. Use --lora-r 0 --unfreeze-last 0 for heads only.

The held-out wording set is data/ood.jsonl. scripts/eval.py also scores gold and a naive-Bayes baseline on user text.

Layout

src/mclass/     extract, pack, two-head model, infer, CLI
scripts/        synth, train, calibrate, eval, ONNX export
data/           seed.jsonl, gold.jsonl, ood.jsonl; train/val/test are generated
artifacts/      download from Hugging Face (encoder, heads.pt, mclass.json, model.onnx)

Base encoder: sentence-transformers/all-MiniLM-L6-v2 (22M). The forward pass mean-pools token states. On a Radeon 8060S, full encoder backward produces NaN grads. LoRA on the last two MiniLM layers stays finite and updates those query/value weights.

ModernBERT-base is the longer-context option: --base answerdotai/ModernBERT-base --lora-r 0 --unfreeze-last 0.

uv pip install --index-url https://rocm.nightlies.amd.com/v2/gfx1151/ --prerelease=allow --reinstall torch 'rocm[libraries]'

torch.cuda.is_available() is true on that ROCm wheel. The default PyPI torch wheel is CUDA and will not see this GPU.

malaikat

Do not add a profile in the malaikat tree from this repo. Consume mclass as:

1. mclass serve --host 127.0.0.1 --port 8091 --artifacts /path/to/artifacts
2. POST http://127.0.0.1:8091/v1/classify with the input object above
3. Keep learnings with preference: true and confidence_pct at or above your threshold (mcode durable capture used 55)

Eval

data/gold.jsonl is a hand set. data/ood.jsonl is held out of train/val/test. After train:

uv run python scripts/eval.py

artifacts/eval.json holds kind accuracy, durable F1, category accuracy, ECE, the naive-Bayes comparison on the user-text slice, gold, and OOD.

Measured after LoRA train (mean pool, last-2 adapters):

slice	metric	encoder	same-data Bayes
test	kind acc	1.000	—
test	durable F1	1.000	—
user-text	durable F1	1.000	0.920
gold (n=65)	kind acc	0.985	—
gold	preference acc	1.000	—
ood (n=40)	kind acc	1.000	—
ood	preference acc	1.000	—