Apple Silicon embedding benchmark
This started as a question. The Apple Neural Engine is supposed to be the fast, efficient inference path on Apple silicon. Should I contribute a CoreML/ANE embedding backend to llama.cpp? I planned two phases: a bench spike to check the premise, then the contribution. The spike is the part that shipped. The premise was wrong, and pinning down how wrong took three rounds of review.
The short version
- ANE is the slowest GPU-class backend for embedding a BERT-style model, on both an M4 Pro and an M5 Max. CoreML CPU tests slower still, but that’s a different class of hardware.
- MLX is the fast path: 3,099 sentences/sec on an M5 Max, batched 100 at once, short sequences. A more realistic batch of 16 or 32 would land lower.
- An earlier headline number, 6,950 sentences/sec, was wrong. Ten repeated runs reproduce 3,099. Retracted.
- Corrected M4 Pro vs M5 Max numbers below.
- Code and data are public: github.com/petejm/apple-silicon-embed-bench, Apache 2.0.
The bench
Five backends against one model and corpus, run on each machine in turn: CoreML on the ANE, CoreML on the GPU, CoreML on the CPU, MLX, and llama.cpp’s Metal path. The model is BAAI/bge-small-en-v1.5, 33M parameters, a BERT-12 encoder, FP16, converted to CoreML the standard way (coremltools.convert(torch.jit.trace(...)), traced at batch=1/seq=512). MLX loaded the same model as FP32, not FP16 like the rest; it’s still the fastest backend despite the extra memory work. The corpus is 300 sentences from Pride and Prejudice, public domain, bucketed short (~32 tokens), medium (~124), and long (~462).
Hardware: an M5 Max MacBook (18-core, 64GB) and an M4 Pro Mac mini (24GB), both macOS 26.5.
The numbers, corrected
Sentences/sec, short bucket, 10-run means:
| Backend | M5 Max | M4 Pro |
|---|---|---|
| CoreML ANE | 89 | 80 |
| CoreML CPU | 59 | 73 |
| CoreML GPU | 572 | 173 |
| MLX (batch=1) | 562 | 384 |
| llama.cpp Metal (batched) | 1,176 | 933 |
| MLX (batch=100) | 3,099 | 869 |
CoreML CPU is the slowest path on both machines, even slower than ANE: 59 on the M5 Max versus 73 on the M4 Pro, an unexpected win for the smaller chip. Among the GPU-class backends, ANE is the slowest on both. This covers one conversion path, a naive FP16 trace through coremltools. I didn’t test INT8 quantization or Apple’s ml-ane-transformers rewrite, the documented ANE optimum.
The retraction
The first version of the MLX bench script reported 43,000 sentences/sec, absurd on its face. The bug: MLX evaluates lazily, and calling the materialize function on the model’s output wrapper object is a no-op. It doesn’t force the tensor to compute. An early fix cut that number to 6,950 sentences/sec, which I published as the headline.
6,950 was also wrong: it was still an unevaluated lazy graph, just a smaller one. Only materializing the inner .text_embeds field directly, and confirming with a 10-run variance sweep on the same M5 Max, forced a real number: mean 3,099, range 3,070 to 3,125, CV 0.6%. Less than half of what I’d shipped. I retracted 6,950, in the README and everywhere else I could find it.
Not everywhere, it turned out. The retraction pass swept every literal 6,950 and 43K, and a third review round still found two derived figures carrying it: a README ratio that said MLX scaled “8x” from M4 Pro to M5 Max, which was 6,950 divided by the M4 Pro number, and a note that said the number “was 7K.” Both are fixed; the ratio is 3.57x. Retracting a number means grepping for every ratio derived from it too, not just the literal figure.
Cross-generation: M4 Pro to M5 Max
| Backend | Ratio (M5 / M4) |
|---|---|
| ANE | 1.11x |
| CoreML CPU | 0.81x (M4 wins) |
| CoreML GPU | 3.31x |
| MLX batch=1 | 1.46x |
| MLX batch=100 | 3.57x |
| llama.cpp Metal | 1.26x to 1.76x, depending on bucket |
ANE is flat across a generation. GPU and MLX both jumped over 3x. But the M5 Max is a thermally limited MacBook and the M4 Pro is a passively cooled mini, one chip each: silicon and form factor are confounded here. The gains still look GPU-bound for this workload, but a same-form-factor comparison would tell me more.
llama.cpp’s number is muted for a reason outside my code: the Metal init log on the build I tested reports has tensor = false. Apple’s M5 tensor accelerators aren’t wired up in that build. MLX uses them. Some of that gap is a llama.cpp bug I can fix. Some is Apple’s tensor-unit support, not yet exposed, which should close once it lands upstream.
I tried to fix that gap upstream and got it wrong. My theory was a llama.cpp refactor had dropped a synchronize() call the perf counters needed, so I patched it back in. It compiled clean, but comparing master with and without the patch on identical input showed both printing the same wrong number, zero. The bug was an ordering problem in decode(), not a missing call. I dropped the PR rather than send a patch I’d only confirmed compiled.
What else broke
coremltools 9 plus Python 3.13 on macOS 26 segfaults during cleanup, _PyObject_Free called off the GIL from a libdispatch worker thread, five crashes in twelve minutes at one point. Workaround: os._exit(0) right after writing results. Python 3.14 is a dead end here: coremltools has no wheel for it, and torch’s quantization code breaks on a 3.14 typing change. The traced .mlpackage also doesn’t reconvert reliably across chip generations. It worked on the M5 Max and threw a TypeError on the M4 Pro, so the repo ships the converted artifact as a release asset instead.
I did this with Claude: three rounds of tabula-rasa review (SWE, SRE, and domain lenses in parallel) found the silent-failure bugs and the incomplete retraction. I made the calls on what shipped; Claude did the reading and most of the fixes.
Next
The upstream opportunity is llama.cpp’s BERT-embed Metal kernels, not a new CoreML backend. That’s what this spike was for. Next phase needs a tensor-unit state check and a roofline analysis before I claim anything is fixable, plus real M1, M2, and M3 data points, which only exist if people submit them.