My app names files by reading what is written inside them. The dumb version takes the first six words the OCR finds, which works great on invoices and terribly on everything else, so I spent a day measuring whether Apple's on-device model could do better. I fed it a scanned rental contract from my own Downloads folder. The contract is for an apartment in Ñuñoa, a neighborhood of Santiago. The model named the file contrato-arriendo-providencia.
Providencia is a different neighborhood. It appears nowhere in the document. It appears in exactly one place in the entire pipeline: my prompt, where I had written "name files like invoice-mueller-gmbh or contrato-arriendo-providencia" as format examples.
The model took the example I gave it to show the shape of a name, and used it as source material for the name itself.
If I had shipped that, someone's lease would be filed under the wrong part of town, silently, with a filename that looks exactly as trustworthy as a correct one. That is the failure mode that matters with these models. Not that they fail, but that their failures are formatted like successes.
The obvious fix is to take the examples out of the prompt. I did, and reran the same 38 files. Without examples to anchor on, the model drifted in ways the first version never did:
document_type-company-person_subject. That is not a summary of the document. That is my prompt's format description, copied out literally.POLIZA_DE_HOYADO. "Hoyado" is not a Spanish word. It invented it, in capitals, with underscores I never asked for.So examples leak content, and no examples leak the scaffolding. Whatever wording I chose, I was tuning probabilities, not setting rules. A prompt is persuasion. I needed policy.
The version that shipped keeps the examples, because they genuinely do anchor the format, and adds a check the model cannot talk its way past: every word in the proposed filename must exist inside the document. Alphabetic words of three letters or more must appear as words in the extracted text, compared case- and accent-insensitively. Numbers must appear as digits. One miss and the whole name is discarded, and the boring first-six-words method runs instead.
static func inventedWord(in name: String, text: String) -> String? {
func words(_ s: String) -> [String] {
let folded = s.lowercased()
.folding(options: .diacriticInsensitive, locale: .init(identifier: "es"))
let clean = String(folded.unicodeScalars.map {
CharacterSet.alphanumerics.contains($0) ? Character($0) : " "
})
return clean.split(separator: " ").map(String.init)
}
let corpus = Set(words(text))
for w in words(name) where w.count >= 3 {
if !corpus.contains(w) { return w } // first offender, in name order
}
return nil
}It fails closed. When the check rejects a name, the user does not see an error, or a warning, or a worse name. They see the same proposal they would have seen last month. The cost of a rejection is zero improvement. The cost it removes is a lie with a straight face.
Same 38 files from my own machine, prompt with examples, check enabled:
| Proposed name | Offending word | What actually happened |
|---|---|---|
| policia-de-hogar-mueller-gmbh | mueller | Stole the other example from my prompt. A Chilean insurance policy, named after a German plumber I made up. |
| matrimonio-pudauel-1999 | pudauel | The document says Pudahuel. The model respelled a real place. |
| mensaje-de-josy | josy | The person is named Joselyn. |
| apps-objects | objects | The document is German. It says "Objekte". The model translated instead of reading. |
| boleta-cafeteria-alta | alta | The café is called Altura. Truncating a proper noun is still inventing. |
Five fabrications in one run of 38, every one shaped like a plausible filename, every one caught by a set lookup. Overall: 22 names passed and shipped to the user, 15 were rejected and fell back, 0 dishonest names got through.
Strictness has a price and I want to be plain about it. German declensions lose: the model proposed kundigung-vertrag for a cancellation letter whose text says "Vertrags", genitive case, so the name died even though a German speaker would call it correct. Tokenization loses: "6500 W" in a spec sheet does not back up 6500w in a name. Those fall back to the dumb method too.
I tried softening the rule to recover them. Prefix matching, so a name-word may be a prefix of a document-word, rescues "vertrag" against "Vertrags" nicely. It also rescues alta against "Altura", which is the café truncation from the table above, which is a fabrication. Every relaxation I tried let at least one real lie through. So the rule stays strict, and the price of that is a model that sometimes improves nothing. A file organizer can live with "no better than before". It cannot live with confidently wrong.
I make Tidy, a Mac menu bar app that reads what is written inside your files and names them from that. The model naming described here shipped this week in 2.5, manual flow only, with the check above deciding what you see. $9 once, macOS 14 and up, everything on-device. I answer my own email at hola@tidymacapp.com — and if you asked me something on a certain orange website and got silence, it is not rudeness: my comments there get flagged within minutes of posting and email to the mods hasn't fixed it yet. The address above always works.