Lab notebook · Apple Neural Engine

A decision model on the Neural Engine, and the test that a broken port still passes

2026-09-22 · Neural Engineportingverification

We ported Kev-0.6B, an open decision model, to Apple's Neural Engine. One document and a set of typed questions go in; one probability distribution per question comes out, in a single pass with no token generation.

This port merges 196 trained LoRA deltas into the compiled model. It is distinct from the later mutable-adapter Kev builds. Input construction and the pointer head run on the host; the transformer trunk uses the Neural Engine.

Numbers

Measured on an M5 Pro, macOS 27.0 build 26A5421a, reproduced from a clean clone:

MeasureResult
End-to-end warm latency, 64 tokens (60 samples)7.16 ms median, 7.62 ms p99
End-to-end warm latency, 256 tokens (60 samples)27.64 ms median, 28.61 ms p99
Predict-loop throughput, 64 tokens; inputs prepared once150.5 predictions/s
First-question decision agrees with Kev's reference, 155 real records155 / 155
Largest probability difference on those records (engine, fp16)3.80e-02

The part worth knowing

Kev answers several questions about one document in one pass, each in its own branch behind a block-causal mask. A check that reads only the first question missed the particular leakage fault we tested. With a plain causal mask replacing the block-causal one, later branches can see earlier branches: the first question still matched the reference (difference 3.10e-06), while the second and third went wrong (0.247 and 0.428). So the repo has a fifth gate that checks every question against the reference and also asks each question alone, which must equal its packed answer (measured 2.72e-07 on CPU).

How we know it runs on the engine

Limits

  • This measures agreement with Kev's own implementation, not decision quality. Agreement is not accuracy.
  • The fp16 probabilities on the engine differ from the reference by up to 3.80e-02 on real records, although the decision never changed.
  • One machine, one OS build. The multi-question isolation gate runs on one record, with its 2.72e-07 isolation result measured on CPU. The 155-record decision check reads the first question; it is not a 155-record all-question validation.

Check it yourself