The Notion AI Breach Shows Where Your AI Chats Actually Go

A Fireworks AI incident exposed transcripts Notion kept for analytics. Here's how to find, and shrink, the hidden copies of your own AI data.

When you type a question into an AI box, the mental picture is simple. You on one side, a model on the other, one conversation in between.

That picture is wrong in a way that matters this week. A single AI chat can exist as several copies: the one the product shows you, the one the model provider holds for a while, and the ones the product keeps for its own purposes, such as measuring how people use the feature. The last kind is the one nobody thinks about. It is also the one that just leaked.

On November 10, a notice from Notion climbed to the top of Hacker News. Some Notion AI chat transcripts had been caught up in a security incident at Fireworks AI, an inference company (a business that runs AI models on its own servers for other companies) that most Notion users had never heard of. And according to Notion, the transcripts involved were the ones kept for an analytics feature.

The result is a useful lesson for anyone who uses AI tools at work, and a sharper one for anyone who builds them. The conversation you see is rarely the copy that gets you in trouble. The copy kept to study the conversation is.

What actually happened

Here is the sequence, using only what the two companies have published.

Notion's incident notice says Fireworks told it on November 6 that "malicious actors may have accessed potential customer data," and confirmed the dataset on November 8. Notion says the exposed data was "unattributable Notion AI chat transcripts used for the Insights feature." Insights is a Notion feature that helps customers understand their own AI usage, and Notion says it used Fireworks to run classification jobs for it (sorting chats into categories), as well as to serve some open-weight models (models whose weights are published, so any provider can run them) inside the product.

Notion is careful about scope. Its security advisory, updated November 9, says that for the vast majority of affected workspaces the transcripts were unattributable: they carried no label saying who wrote them or which workspace they came from, and they contained no tool-call results, meaning none of the page content or outside data that Notion AI pulls in while answering. Notion says fewer than 1% of workspaces may have been affected and that it has contacted the owners of those workspaces.

Fireworks tells a wider story from its side. Its security bulletin, last updated November 9, says a third-party cloud provider that hosted Fireworks research machines had a security incident on October 23. The next day, the attacker used that access to reach Fireworks development machines and run "an unauthorized workload." Fireworks now reports impact to account and billing information and, for a limited set of accounts, to secrets, datasets and incomplete model weights. It says it has seen no impact to its main inference products, which it says are zero data retention by default, meaning they do not store customer inputs after answering.

One detail in that bulletin explains the attacker's motive. Fireworks links the group to "LLMJacking": stealing access to AI models and reselling it through illicit gateways. The attackers came for computing power they could sell. Customer data happened to be in the building.

Notion's response was fast and broad. It disabled Insights, stopped serving open-weight models through Fireworks, and says it disconnected its systems, deleted its data on Fireworks and halted training runs as of November 3.

The copy you never pictured

Think of an AI chat the way you'd think of a letter you mail.

The letter itself goes to the recipient. The post office keeps a record that it moved. And in this case, a researcher down the hall also photocopies a stack of letters every week to study what people write about. You agreed to send the letter. You probably never pictured the photocopy.

That photocopy is the secondary copy, and it has three properties that make it the weak spot.

It exists for a different purpose than the chat. The chat exists to answer your question. The analytics copy exists to answer the company's questions about you and everyone else, which means it gets pooled, kept longer and handed to whatever service is cheapest or fastest at classifying text.

It often lives somewhere else. Notion's AI security page says its models come either from the companies that built them or "through inference providers," and that all of them are bound by the same contracts. That is honest and reasonable. It also means the list of companies touching your text is longer than the logo on the app.

And "anonymous" doesn't mean harmless. Notion says most of the transcripts carried no author or workspace labels. Good. But a transcript is text, and people paste things into text. That's why Notion's own advice is blunt: "If you or anyone on your team pasted sensitive information like passwords, API keys, or other credentials into Notion AI during this timeframe, we recommend rotating them right away." Removing the name from a message that contains a database password still leaves you with a working database password.

Why this is bigger than Notion

Notion is just the clearest example this week, because it published details. The pattern is everywhere.

Any product that offers "usage insights," "conversation analytics" or "quality review" on its AI features is building that photocopy. Any company that experiments with training its own smaller models on customer chats is building another one. Notion's notice mentions that data volunteered through its LEAP program had been used "to experiment with training task-specific models on Fireworks." Every one of those copies is a reasonable product decision. Each one adds a place for your text to sit.

Retention defaults are moving too, and not always toward less. On November 9, GitHub made Claude Fable 5.2 generally available in Copilot, with a line every admin should read: the model "requires data retention by default." Anthropic keeps prompts and outputs to run safety classifiers (automated checks for misuse) and does not use them for training, according to GitHub's changelog. Zero data retention is available only to certain approved enterprises. That's a defensible safety choice. It also means that switching models inside the same tool can change where your prompts live and for how long.

So the question to ask about any AI tool has changed. "Do I trust this company?" is not enough. The better question is: how many copies of what I type exist, who holds each one, and what is each one for?

Put this into practice

None of this requires a security team. Most of it takes an afternoon.

1. Rotate anything you have pasted into an AI chat. If you are a Notion user who was contacted, start with Notion's own review screen: in the desktop app or on the web, open the workspace sidebar, click "?", then "Security incident review." Enterprise owners get a downloadable file, and Notion's advisory says affected customers also received instructions by email. Notion's notice doesn't say how far back "this timeframe" reaches, so the safe version of this step is broad: any API key, password or token that anyone on your team has ever pasted into an AI assistant should be rotated. Expect this to take longer than you think, because the hard part is remembering where you pasted things.

2. Find the vendor list behind each AI tool you use. Most companies publish a "sub-processor" list, the formal name for the outside companies that handle your data on their behalf. Search for "[product name] subprocessors." Look for inference providers and AI labs, not just cloud hosts. You are looking for names you didn't expect. (Notion's list failed to load when we checked, so we could not confirm whether Fireworks appeared on it.)

3. Look for the secondary features and switch off what you don't use. In each tool's admin settings, look for anything called insights, analytics, quality review or "help improve our product." These are the features that create the photocopy. If nobody on your team reads those dashboards, turn them off.

4. Check your retention settings per model, not per vendor. In GitHub Copilot, Business and Enterprise admins control Fable 5.2 through a dedicated policy that is off by default. Before turning it on, read the retention line in the changelog and decide whether your code is fine with being held for safety review.

5. Move secrets out of chat for good. Share credentials through a password manager or a secrets store, never through a chat box, AI or otherwise. If your team needs an AI to work with a key, give it an environment variable name, not the value.

6. If you build AI features, scrub before you store. If your product keeps chat logs for analytics or evaluation, run a secret scanner such as gitleaks over transcripts before they leave your systems (it catches known key formats, not every sensitive value), keep the analytics copy for days rather than months, and send it to as few vendors as possible. The cheapest data to protect is the data you never kept.

Honest limitations

A few things in this story are still unknown, and they matter.

Notion's notice doesn't define the exposure window, and "fewer than 1% of workspaces" is a ceiling, not a count. Notion says it has contacted affected owners, and that if neither you nor your owner has heard from them, no action is needed right now.

The two companies' accounts don't line up perfectly. Fireworks' bulletin describes secrets, datasets, billing data and model weights but never mentions Notion or chat transcripts. Notion's notice describes transcripts. Both may be true. They just describe different slices of the same incident.

And the steps above reduce exposure. They don't remove it. Every AI tool you use still sends your text to at least one model provider, and contracts, not physics, keep that copy from being misused. That's the deal, and for most work it's a fair one. It's worth making on purpose.

The copy you control

The good news hiding in this incident is that the riskiest copy of your AI chats is also the most optional one. You need the model to see your question. You probably don't need a vendor two steps away to keep a pile of transcripts for a dashboard nobody opens.

So spend the afternoon. Rotate the keys, read the vendor lists, flip off the features you don't use. Next time a notice like this lands, you'll already know whether it's about you.

Sources: Notion incident notice, Notion security advisory (updated November 9, 2026), Notion AI security practices, Fireworks AI security bulletin (updated November 9, 2026), GitHub changelog: Claude Fable 5.2 in GitHub Copilot (November 9, 2026), Hacker News discussion.