Apple's sentiment analysis thinks 'delete the temp file' is a death threat

NLTagger is Apple’s API for sentiment analysis. It’s integrated into iOS and macOS, runs on-device, needs no server, and is three lines of code away. It’s the first thing you find when searching for “sentiment analysis Swift”.

Here’s what it returns for text any developer would write on a normal day:

The scale goes from -1.0 (very negative) to +1.0 (very positive). According to Apple, “delete the temp file” carries almost the same emotional weight as “this is fucking broken”. And “ok” – the most neutral response in the English language – scores -0.8.

These aren’t made-up numbers. They’re reproducible results on any Mac with macOS 14+.

How to reproduce it

import NaturalLanguage

let tagger = NLTagger(tagSchemes: [.sentimentScore])
tagger.string = "delete the temp file"
let (tag, _) = tagger.tag(
    at: tagger.string!.startIndex,
    unit: .paragraph,
    scheme: .sentimentScore
)
print(tag?.rawValue ?? "nil")
// "-0.8"

Eight lines of Swift. The result is deterministic: the same text produces the same score on every execution, on every device. This isn’t a random error. It’s systematic bias.

Why it happens

NLTagger uses a model trained on consumer text: product reviews, social media comments, restaurant opinions. In that domain, it works reasonably well. “This product is terrible” scores -0.9 and “I love this app” scores +0.9. Correct in both cases.

The problem is that technical vocabulary shares words with emotional vocabulary, but with completely different meanings:

• kill, terminate, abort – normal operations on processes
• fatal, critical, panic – log levels
• crash, dead, zombie – system states
• delete, destroy, drop – data operations
• reject, deny, block – access control

For a model trained on Amazon reviews, “delete” is destructive, “kill” is violent, and “fatal” is catastrophic. It has no context to understand that “kill the background process” is as emotional as “close the door when you leave”.

This is called lexical bias: the model assigns polarity to individual words without understanding the domain. It’s equivalent to an automatic translator interpreting “I’m killing it” as a homicide in progress.

Nobody has noticed

Here’s what’s interesting: this bias doesn’t appear in any academic paper. There are hundreds of articles about sentiment analysis in software engineering – code review analysis, toxicity detection in commits, issue classification – but none mention NLTagger. The NLP community completely ignores it.

The reason is simple: nobody uses NLTagger for production. Researchers use Hugging Face models. Companies use OpenAI or Google APIs. NLTagger is what the indie developer uses when looking for a quick solution for their iOS app, implements it in an afternoon, and never validates the results against real data.

That means there’s an unknown number of apps in the App Store that are classifying technical text with a model that thinks “ok” is almost as negative as an explicit insult. And none of them know it.

A real case: code session monitoring

The problem isn’t theoretical. I built Tokamak, a macOS app that monitors Claude Code sessions. One feature I wanted to add was frustration detection: if you’ve been fighting a bug for two hours, the app should be able to detect it from the tone of your messages.

The obvious approach was NLTagger. Three lines of code, zero dependencies, runs on-device. The prototype worked in 20 minutes.

And it classified “delete the temp file and run the tests” as -0.7. A completely neutral instruction you’d give to a code assistant. According to NLTagger, I was on the verge of an emotional breakdown.

How we solved it: deterministic layers first

The solution isn’t “use a better model” (though that helps too). The solution is not depending on a single opaque model for everything.

SentimentKit is the library I wrote to solve this. It uses a 4-layer pipeline where deterministic analysis runs first and ML only intervenes when there’s ambiguity:

Message
  │
  ├─ Layer 1: Keyword detector (deterministic, ~20KB)
  │  Curated dictionaries of profanity, frustration and positive expressions.
  │  8 languages (ES, EN, PT, DE, FR, ZH, JA, KO).
  │
  ├─ Layer 2: VADER rules (deterministic, ~500KB)
  │  Negation ("not good"), intensifiers ("very bad"),
  │  CAPS, punctuation.
  │
  ├─ Layer 3: CoreML DistilBERT (optional)
  │  Only for long and ambiguous messages.
  │
  └─ Layer 4: LLM scorer (optional, requires API)
     Only for cases the previous layers don't resolve.

The key is that layers 1 and 2 are deterministic. They produce the same result every time. There’s no opaque model. You can audit every dictionary, every rule. And most importantly: neutral technical commands score 0.0, not -0.8.

import SentimentKit

let analyzer = SentimentAnalyzer()

let result = analyzer.analyze("delete the temp file and run make test")
// result.score = 0.0 -- neutral, as it should be

let angry = analyzer.analyze("what the hell is this, nothing works")
// angry.score = -2.0 -- two profanity hits detected

VADER: the tool that should be better known

The rules in layer 2 are inspired by VADER (Valence Aware Dictionary and sEntiment Reasoner), a system created by C.J. Hutto and Eric Gilbert at Georgia Tech in 2014. VADER is a rule-based sentiment analyzer, not ML-based, that uses a dictionary of ~7500 human-annotated words with valence scores.

VADER’s strength is handling linguistic modifiers that bag-of-words models ignore:

• Negation: “not good” inverts polarity
• Intensifiers: “very good” amplifies the score
• Capitals: “GOOD” scores higher than “good”
• But: “the food was great BUT the service was terrible” – the but gives more weight to the second clause

VADER has limitations (doesn’t understand sarcasm, works better in English), but its transparency is its greatest virtue. When VADER classifies something wrong, you can open the dictionary, find the entry, and correct it. With NLTagger, you can only shrug.

The irony: Apple created the problem and the solution

Since macOS 26 / iOS 26, Apple offers the Foundation Models framework, a ~3B parameter LLM that runs on-device. It’s incomparably better than NLTagger for sentiment analysis because it understands context, not just individual words.

The same Apple that sold you a classifier that thinks “ok” is negative, now offers you a language model that understands “kill the process” is a technical instruction. The correct solution existed within the same ecosystem; it just arrived a decade late.

SentimentKit can use Foundation Models as layer 4 (LLM scorer). The irony of using Apple Intelligence to correct NLTagger’s errors isn’t lost on me.

How Anthropic detects frustration (spoiler: regex)

An interesting fact. In Claude Code’s source code (Anthropic’s CLI for Claude), user frustration detection is implemented with regular expressions. Not with NLTagger. Not with an ML model. With regex that searches for patterns like “this is broken”, “doesn’t work”, “what the”.

It’s a crude but effective approach: no false positives on technical commands, because the patterns are explicit. Anthropic, the company behind one of the world’s most advanced LLMs, uses regex to detect frustration. If that doesn’t tell you something about the state of sentiment analysis in production, nothing will.

Golden tests: 144 fixtures and counting

SentimentKit includes 144 golden messages with exact assertions: 35 in Spanish, 35 in English, 20 in Portuguese, 19 in German, 20 in French and 15 in Chinese. The data comes from published datasets (cardiffnlp/tweet_sentiment_multilingual, sepidmnorozy/Chinese_sentiment) and real code reviews from public repositories.

Each fixture has an expected score range, the expressions that should be detected, and those that shouldn’t. The test system includes PHANTOM detection (dictionary entries that never match real data) and UNCONSUMED detection (real expressions missing from dictionaries). It’s the same approach I use in Tokamak to validate DTOs against production data: if the LLM invented data, the test fails.

Try it

git clone https://github.com/frr149/SentimentKit
cd SentimentKit
swift test

Or as a Swift Package Manager dependency:

.package(url: "https://github.com/frr149/SentimentKit.git", from: "1.0.0")

The package works without CoreML and without internet connection. The deterministic layers (keywords + VADER) are sufficient for most use cases in technical text.

The lesson

If you use NLTagger for any text that isn’t product reviews, you’re getting garbage data with two decimals of false precision. The model isn’t broken; it’s out of domain. And the API doesn’t warn you.

The NLP community ignores it. Apple doesn’t document it as a limitation. And developers who use it in production never validate the results, because a Double between -1.0 and 1.0 looks rigorous even when it means nothing.

Before trusting a sentiment score, ask yourself: should “ok” be -0.8? If your tool says yes, the problem isn’t with the user. It’s with the tool.