We ported Kev-0.6B, an open decision model, to Apple's Neural Engine. One document and a set of typed questions go in; one probability distribution per question comes out, in a single pass with no token generation.
This port merges 196 trained LoRA deltas into the compiled model. It is distinct from the later mutable-adapter Kev builds. Input construction and the pointer head run on the host; the transformer trunk uses the Neural Engine.
Numbers
Measured on an M5 Pro, macOS 27.0 build 26A5421a, reproduced from a clean clone:
| Measure | Result |
|---|---|
| End-to-end warm latency, 64 tokens (60 samples) | 7.16 ms median, 7.62 ms p99 |
| End-to-end warm latency, 256 tokens (60 samples) | 27.64 ms median, 28.61 ms p99 |
| Predict-loop throughput, 64 tokens; inputs prepared once | 150.5 predictions/s |
| First-question decision agrees with Kev's reference, 155 real records | 155 / 155 |
| Largest probability difference on those records (engine, fp16) | 3.80e-02 |
The part worth knowing
Kev answers several questions about one document in one pass, each in its own branch behind a block-causal mask. A check that reads only the first question missed the particular leakage fault we tested. With a plain causal mask replacing the block-causal one, later branches can see earlier branches: the first question still matched the reference (difference 3.10e-06), while the second and third went wrong (0.247 and 0.428). So the repo has a fifth gate that checks every question against the reference and also asks each question alone, which must equal its packed answer (measured 2.72e-07 on CPU).
How we know it runs on the engine
- On this tested graph, the CPU-only backend returned zero hidden states while the CPU-and-Neural-Engine path returned nonzero states matching the reference decisions. This backend contrast is read alongside the direct dispatch trace and rail activity below.
- Apple's Neural Engine instrument counts 301 predict intervals for 301 predictions (plus 2 load intervals).
- The sampled engine power rail went from 0 mW idle to a loaded peak of about 7,100 mW. This is not total system power or energy per decision.
Limits
- This measures agreement with Kev's own implementation, not decision quality. Agreement is not accuracy.
- The fp16 probabilities on the engine differ from the reference by up to 3.80e-02 on real records, although the decision never changed.
- One machine, one OS build. The multi-question isolation gate runs on one record, with its 2.72e-07 isolation result measured on CPU. The 155-record decision check reads the first question; it is not a 155-record all-question validation.
Check it yourself
- kev-ane: the port, the six gates and the placement tool.
- Kev itself: github.com/jaredpalmer/kev.
- The later mutable-adapter diagram is a different build and is not used to illustrate these measurements.