The Two-Tier Pattern in Apple Silicon: Cheap Deterministic Code First, CoreML for the Heavy Lifting

In the last two posts (about perceptual hashes in the context of Chat Control and about their adversarial collisions), we explored how 40 lines of Python can solve a problem that the industry often deploys at a global scale. Perceptual hashes are the canonical example of a cheap deterministic layer: no models, no GPUs, no heavyweight dependencies.

Hashes handle duplicate and similarity detection. But in a real-world image processing pipeline, there’s work that classical algorithms can’t do: detecting bounding boxes of arbitrary watermarks, filling gaps with inpainting that respects visual context, classifying whether a face is of an adult, or extracting semantic attributes. This work requires large neural networks.

The question this post answers: how to deploy those networks on Apple Silicon without relying on cloud APIs, without paying per-image tokens, and leveraging Apple Neural Engine. Short answer: PyTorch → ONNX → CoreML in three commands. The longer answer? The real value isn’t technical; it’s architectural. This pattern works across seemingly unrelated domains — sentiment analysis, image moderation — and it’s the same every time.

The Pattern: Two Tiers, One Philosophy

A few weeks ago, I released SentimentKit, a sentiment analysis library born from discovering that Apple’s NLTagger rated “delete the temp file” as -0.8 (very negative). The issue wasn’t unique to Apple: it’s a class of systematic biases in pre-trained ML models.

The solution wasn’t “find a better model”—it was architectural. SentimentKit uses a four-tier pipeline:

1. Deterministic keyword detector. Curated dictionaries of profanity, frustration, and positive expressions, in 8 languages. ~20 KB, runs first.
2. Apple’s NLTagger (with technical bias correction). Only invoked if no triggers were hit in the first tier.
3. Domain-specific model (rules + heuristics for code/technical content).
4. Foundation Models (local Apple LLM) only if the first three tiers return ambiguous results.

The pattern: run the cheap deterministic layer first, and use ML only if the cheaper layers can’t decide. About 70-80% of traffic is resolved in the first tier without involving a model. As a result, energy costs, latency, and hallucination risks decrease proportionally.

When I started designing a large-scale image cleanup pipeline a few weeks later, I realized I was using the same pattern without even thinking about it:

1. Deterministic validation (Pillow verify, magic bytes, size limits). Filters out the obvious cases.
2. Perceptual hashes (aHash + dHash + pHash). Clusters duplicates in ~10-15 ms per image.
3. Classical OCR with regex for known tokens. Flags watermarks by text.
4. Neural networks (YOLOv11 for detection, LaMa for inpainting). Applied only to canonical survivors — ~10-20% of the original volume.

It’s no coincidence. It’s the same philosophy applied to a different domain. The operational question — optimized a thousand times in production — remains the same: how to make neural layers run as cheaply as possible, avoiding cloud APIs and heavy GPU infrastructure.

The answer on Apple Silicon is ANE via CoreML. The bridge between PyTorch (where models originate) and CoreML (where we deploy) is called ONNX.

ONNX: The Intermediate Language You’ve Probably Overlooked

ONNX (Open Neural Network Exchange) is a model format for neural networks. It’s maintained by a consortium including Microsoft, Meta, NVIDIA, and others. The specification defines how to serialize a model’s computational graph — its operations, connections, and weights — into a portable file.

Why it matters: it decouples training from deployment.

Before ONNX, training a model in PyTorch meant a hard lock to PyTorch in production. Moving to TensorFlow Serving, CoreML, or an embedded runtime required re-implementing the model operation by operation. The process was manual, error-prone, and required separate maintenance for each combination.

ONNX introduces a pivot. The workflow shifts to:

PyTorch (training) → ONNX (exchange) → Runtime X (production)

Where Runtime X could be onnxruntime (multi-backend), TensorRT (NVIDIA), OpenVINO (Intel), CoreML (Apple), TFLite (mobile), Triton (server), or WebAssembly (browser).

Once exported, the runtime selects the optimal backend for the available hardware. On Apple Silicon, the backend to use is CoreML Execution Provider, which schedules across CPU, GPU Metal, and Apple Neural Engine as needed.

Honest Limitations

Not all PyTorch operations have direct ONNX equivalents. Modern architectures often use custom ops (flash attention, exotic convolutions) that don’t always export cleanly. Practical solution: if your model is YOLO, ResNet, a standard Transformer, or a U-Net, it will export without issues. If it uses rare operations, check the compatibility matrix first.

In our case — fine-tuned YOLOv11 and LaMa — both have first-class ONNX export. YOLOv11’s ultralytics library supports it with a single line. For LaMa, the Carve AI team has already published an ONNX version on Hugging Face; no export required.

CoreML: Apple’s ML Runtime and the Neural Engine

CoreML is iOS and macOS’s machine learning runtime. Available since iOS 11 (2017), it supports models running on:

• CPU. Always available, useful for non-parallelizable tasks or very small models.
• GPU Metal. Fast for large batches, ideal for training or mass inference.
• Apple Neural Engine (ANE). A co-processor optimized for low-power tensor operations. Features 16 cores in M-series chips and delivers ~18 TOPS in the M4-series. Optimized for convolutions, matrix multiplications, and inference in reduced precision (FP16, INT8).

CoreML decides where each operation runs. Compatible operations → ANE; others → GPU or CPU. The developer doesn’t configure the scheduler — CoreML handles optimization.

ANE’s key advantage over GPU Metal: energy efficiency. In single-shot inference, ANE uses approximately 3-5x less energy than GPU Metal for the same workload, based on public benchmarks from Apple and third parties (e.g., Geekbench ML, CreateML). On a MacBook Pro, this efficiency translates into the difference between running the pipeline once or leaving it running overnight without draining the battery.

The downside: ANE doesn’t accelerate training, and not all operations are supported. Models with good exports (standard architectures, FP16/INT8 quantized) utilize ANE for 90%+ of operations.

The Role of onnxruntime-coreml

Microsoft’s onnxruntime includes Execution Providers to abstract hardware. By installing the onnxruntime-coreml package (available since 2022), the runtime transparently uses CoreML as the backend:

import onnxruntime as ort

session = ort.InferenceSession(
    "yolo11x-watermark.onnx",
    providers=["CoreMLExecutionProvider", "CPUExecutionProvider"],
)

output = session.run(None, {"images": input_tensor})

These three lines are enough to run an ONNX model on Apple Neural Engine. CPU fallback is specified for unsupported operations. No Swift code, no Xcode, no .mlmodel packaging required. Just Python + ONNX + onnxruntime-coreml.

The Conversion in Three Real Commands

Here’s the complete workflow for a real production case: fine-tuning YOLOv11 for watermark detection.

Step 1: Get the PyTorch Model

For YOLOv11, I use corzent/yolo11x_watermark_detection — a fine-tuned YOLOv11-extra model trained on a curated watermark dataset, 114 MB, MIT license:

hf download corzent/yolo11x_watermark_detection --local-dir ./models

This downloads best.pt (the PyTorch model) along with metadata. No Hugging Face token is required for this public repository.

Step 2: Export to ONNX

The official YOLOv11 library ultralytics handles this in one call:

from ultralytics import YOLO

model = YOLO("./models/best.pt")
model.export(format="onnx", imgsz=640, simplify=True)
# -> ./models/best.onnx (228 MB)

The simplify=True flag collapses redundant operations to improve compatibility with ANE. Measured time on an M3: 2-3 seconds. Requires onnx, onnxslim, ultralytics, and torch dependencies.

Step 3: Inference with CoreML Execution Provider

import onnxruntime as ort
import numpy as np
from PIL import Image

session = ort.InferenceSession(
    "./models/best.onnx",
    providers=["CoreMLExecutionProvider", "CPUExecutionProvider"],
)

img = Image.open("photo.jpg").resize((640, 640))
tensor = np.asarray(img, dtype=np.float32).transpose(2, 0, 1)[None] / 255.0

outputs = session.run(None, {"images": tensor})
# outputs[0] shape: (1, 5, 8400) — cx, cy, w, h, score per detection

When loading the session, onnxruntime logs illustrate partitioning:

CoreMLExecutionProvider::GetCapability
  number of partitions supported by CoreML: 7
  number of nodes in the graph: 617
  number of nodes supported by CoreML: 609

This means 609 of 617 operations (98.7%) run via CoreML — and thus on ANE — while only 8 fall back to the CPU. The graph splits into seven subgraphs because the CPU operations intersperse at specific points.

Measured time on an M3 (post-warmup): 42-48 ms per image. The first run incurs a ~55 ms CoreML graph compilation overhead, amortized starting from the second image.

Optional Step: Convert to Native .mlpackage

If the model is intended for distribution within an iOS/macOS app (not a Python script), you’d need to convert it to .mlpackage. This introduces some friction in 2026:

(a) PyTorch → .mlpackage via ultralytics.export(format="coreml"). Internally uses coremltools. In practical testing, it breaks with Python 3.14 + torch 2.11 + coremltools 9 due to TypeError: only 0-dimensional arrays can be converted to Python scalars. This is not a model issue; it’s ecosystem pinning.

(b) ONNX → .mlpackage via coremltools.convert(onnx_file). This worked in coremltools ≤6. However, coremltools 9 removed ONNX support, failing with: “source framework not detected, choose from tensorflow / pytorch / milinternal”.

The pragmatic solution for .mlpackage: use an isolated venv with Python 3.11, torch 2.3, and coremltools 6.x for one-off conversion. For most pipelines, however, .mlpackage isn’t necessary. If running the model in a Python script on macOS (batch processing, CLI tools, internal servers), onnxruntime-coreml is sufficient.

Real-World Detection Results

I tested the exported YOLO model against real-life watermark samples covering five common patterns found online (four images per pattern, 20 total). Detections per image are post-NMS (non-maximum suppression):

The model — trained on a generic intrusiveness dataset — performs well for visually large marks (banners, mosaics, central text) and fails on fine corner text (domain logos, lateral text). This isn’t a model implementation or ONNX export issue; it reflects the model’s training limitations. YOLO models are inherently better at prominent bounding boxes.

Key design takeaway: YOLO and OCR-based detection are complementary, not alternatives. Full watermark detection pipelines need both:

• OCR with regex for known tokens (e.g., Apple Vision VNRecognizeTextRequest, PaddleOCR fallback) flags URLs, phone numbers, and small text in any size. Cheap, deterministic, CPU-only.
• YOLO flags visually large, non-textual marks (e.g., tiled mosaics, banners). More costly per image, but ANE-efficient at ~45 ms.

The core architectural lesson: don’t assume a single neural model handles all cases. Measure your data and combine specialized layers. Starting the pipeline with a general-purpose model — a CLIP, multimodal LLM, or YOLO trained generically — yields uneven coverage and diagnostic challenges. Two specialized tiers are more robust than one opaque monolith.

Measured times for 20 real images (M3, post-warmup): 42-48 ms per image, consistent with synthetic benchmarks. CoreML warmup costs ~55 ms on the first call.

Why This Beats PyTorch-MPS

In Python communities, the default often leans toward device="mps" in PyTorch. It works and is common in YOLOv11 notebooks/tutorials for Apple Silicon. But this approach has three significant drawbacks:

1. No ANE Usage. PyTorch-MPS uses GPU Metal. The Neural Engine sits idle. Energy consumption is 3-5x higher for equivalent tasks.
2. Heavy Dependencies. Full PyTorch requires ~5 GB installation. onnxruntime-coreml is ~100 MB. This matters in CI pipelines or reproducible environments (e.g., multi-platform Docker setups).
3. Python 3.13 Dependency Issues in Some Models. Tools like simple-lama-inpainting rely on outdated dependencies (pillow==9.5.0) that stall at Python 3.13. In multi-version monorepos using Python 3.14, isolated venvs introduce friction. ONNX models can run under any Python version, avoiding these pitfalls.

General rule: if the model runs in production (server-side, batch, cron jobs, etc.), export to ONNX and use onnxruntime-coreml. Reserve PyTorch for research, fine-tuning, and exploration. The conversion friction happens once, while the benefits (efficiency, portability, lighter dependencies) accumulate over each run.

Back to the Pattern: The Full Image Pipeline on Apple Silicon

With the components assembled, the complete image cleanup pipeline looks like this:

About 80% of the volume is resolved in tiers costing under 20 ms, solely using local resources. Neural networks — when necessary — run on ANE with low energy usage. Only the ~20% of traffic that survives to the later tiers goes to an external API, reducing the Qwen API cost from ~$3.50 per full pipeline to ~$0.50-$0.80.

It’s the same economic profile as the SentimentKit pipeline but applied to images. Different domain, same pattern, same benefits:

• Perceived latency is very low for majority cases.
• Energy consumption is driven by CPU and ANE, not GPU.
• Manageable dependency surface area.
• ML failures (e.g., hallucinations or API timeouts) impact a small percentage of traffic rather than 100%.

Closing: Architecture Over Technology

The takeaway isn’t “use CoreML,” “export to ONNX,” or “avoid PyTorch-MPS.” Tools are interchangeable.

The real lesson is the pattern: Deterministic first, ML second.

When designing a pipeline — whether for sentiment analysis, vision processing, moderation, recommendations, or anything else seemingly reliant on an LLM upfront — ask: what parts can be solved with rules, regexes, hashes, or dictionary lookups? These run first: free, deterministic, auditable. What remains escalates to the next layer.

On Apple Silicon, that next layer is CoreML via ONNX. It offers the most efficient operational setup available outside a datacenter. If you’re deploying a model on macOS or iOS, the PyTorch → ONNX → CoreML path is the right one.

And for your next library, remember what I’ve verified across two domains: the two-tier architecture isn’t premature optimization. It’s the default mental model. The mistake is designing it backwards — starting with a heavyweight model and adding rules later when costs surface.

This article was originally published in Spanish and translated with the help of AI.