Local AI Came to Your Laptop. Your Notes Did Not.

What Actually Changed

Two things happened at once, and the combination is what made local inference ordinary rather than impressive.

The first is that small models got disproportionately good. Distillation and better training data closed much of the gap between a model you can run on a laptop and one that needs a data center, at least for the tasks most people actually perform: rewriting, summarizing, extracting structure from messy text, transcribing speech, answering questions about documents you supply. These are not the frontier reasoning problems where the largest models still clearly win. They are the bulk of daily use.

The second is that the hardware stopped being the constraint. Neural accelerators are now standard rather than premium across current Apple silicon, recent Windows laptops, and the phones in both ecosystems, with Apple putting a 32-core Neural Engine in the iPhone 18 Pro. Unified memory on Apple silicon in particular turned out to suit local inference well, because the model does not have to be shuttled across a narrow bus to a separate pool of video memory. A machine bought in the last couple of years can generally run a useful model without drama.

The resulting consensus is hybrid rather than absolutist, and it is worth saying plainly because purists on both sides tend to misrepresent it. Most shipping products, Apple's own assistant included, run routine work on the device and send genuinely hard requests to a server, ideally telling you which is happening. That is a sensible engineering compromise. It is also not the same thing as your data staying home, and the distinction is the entire subject of this piece.

The Asymmetry Nobody Mentions

Here is the arrangement a lot of privacy-motivated people have ended up in. The model runs locally. The notes it reads are synced to a service. So the inference is private and the corpus is not, which means the material that would actually hurt to lose or leak, the thing the model is merely reading, sits exactly where it always did. You have secured the verb and left the noun exposed.

It is worth being precise about what this does and does not cost you, because overstating it is its own kind of dishonesty. Cloud note services are not reckless; most encrypt in transit and at rest and have competent security teams. The exposures that remain are structural rather than negligent. A breach reaches data that is held, and held data is a target in a way that absent data is not. Legal process reaches a company that has your files; it cannot reach files that only exist on your disk. Policy changes apply to whatever the company holds at the time, and terms get revised on the company's schedule, not yours. And shutdowns take the library with them unless you exported first, which is the failure mode that actually happens to people most often.

None of that is an argument that cloud sync is a mistake. It is an argument that if the reason you moved inference on-device was privacy, and your notes are still hosted, you have bought a smaller improvement than you think. The exposure you removed was the model provider seeing one query. The exposure you kept was somebody holding the whole archive.

What a Local Model Needs From Your Notes

Running inference on your own machine imposes requirements that a cloud service quietly handled for you. Three of them matter.

The files have to be on the disk

This sounds obvious and is routinely violated. A local model can only read what it can open. Notes that live behind a web application's API are not available to it in any straightforward way, and the workarounds, browser automation, periodic exports, unofficial sync scripts, are exactly the brittle machinery that breaks the week you need it. Placeholder files from cloud storage that download on access are a subtler version of the same problem: the bytes are not actually there until something fetches them, which fails the moment you are offline, and offline is one of the main reasons to run locally at all.

The format has to be readable without the app

A local model reads text. Markdown, plain text, and standard document formats are trivially available to it. Proprietary databases are not: if your notes live in an application-specific binary store, extracting them for a local model means either an official export step you have to remember to run or reverse-engineering someone's schema. This is the same portability argument that applies to surviving a shutdown, arriving from a different direction, which is usually a sign it is the right argument.

Something still has to do the selecting

Local models generally run with smaller practical context than the largest hosted ones, because memory on your laptop is finite and shared with everything else you are doing. So the retrieval step matters more locally, not less. You cannot compensate for a disorganized archive by throwing all of it at the model, which is the escape hatch a very large hosted window gives you. The library has to be navigable, which in practice means notes carry enough structure, titles, dates, tags, links, for something to find the right dozen of them.

Local-First Is Not All-or-Nothing

The useful frame is not "local versus cloud" but "where does the canonical copy live." Local-first means the authoritative version of your library is on your device and everything else, sync, backup, a hosted model you occasionally call, is a convenience layered on top of that. It does not mean refusing to use the network. It means that if every service you use disappeared tomorrow, you would still have your notes, and you would still be able to read and search them.

Under that definition a hybrid setup is entirely coherent. Keep the library local. Run the routine work, summarizing, tagging, search, transcription, on the device, where it is fast, free, and offline-capable. Call a frontier model deliberately for the genuinely hard question, with the specific material that question needs, understanding that you are making a choice in that moment rather than having it made for you by default. The distinction that matters is not whether data ever leaves, but whether you decided each time it did.

A Setup That Holds Up

1. Decide where the canonical copy lives, and make it your disk. Everything else in this list follows from that one choice, and skipping it makes the rest cosmetic.
2. Get your existing notes out of anywhere that will not hand them over cleanly. Run the export now, on a calm day, and open the result to confirm it contains what you think it does.
3. Standardize on formats that outlive applications. Markdown and plain text for notes, standard formats for documents, real files in real folders rather than rows in a proprietary store.
4. Back up locally, then back up off-site. Local-first without backups is not privacy, it is a single point of failure with better marketing. An encrypted off-site copy you control is not the same as hosting your library.
5. Let something organize the library continuously. The reason people abandon local setups is not ideology, it is that an unmaintained folder of thousands of files stops being searchable and they drift back to a service that did the work.
6. Then point models at it, local by default and hosted by decision. This is the only step most guides cover, and it is the one that matters least if the first five are missing.

Mindly is built on that first decision: the library is a local store on your Mac, organized on arrival rather than by hand, and readable without us. A local-first library →

The Part Worth Remembering

On-device AI is the most genuinely user-favorable trend in this field right now, and it deserves more enthusiasm than it usually gets. It makes capable tools cheap to run, available offline, and private by construction rather than by promise. It is worth reorganizing around.

Just reorganize around the whole thing. The model is the part that will be replaced repeatedly over the next few years, and replacing it should be easy. Your notes are the part that should still be there afterwards, in a form the next model can read. If the inference came home but the library did not, you have moved the easy half.