Notion’s account of the Fireworks AI incident shows where a zero data retention promise stops. I think every security team that has signed off an AI feature on the strength of a ZDR clause should now map, feature by feature, where its prompts and outputs are actually stored, and by which vendor.
What the two companies have said
Fireworks AI, an AI infrastructure platform, updated its security bulletin to say that, beyond the stored secrets it first reported, its investigation found impact to account and billing information and, for a limited set of accounts, to secrets, datasets and incomplete models. It says it has not detected any impact to its Inference, Training SDK and Nexus products, which it describes as zero data retention by default, and that it is working with Palo Alto Networks Unit 42.
Notion, one of Fireworks’ customers, says in its help center notice that the incident affected fewer than 1% of Notion workspaces and exposed chat transcripts. Notion used Fireworks for classification jobs behind its Insights feature, which shows administrators how members use AI, and as an inference provider for open-weight models. Its notice explains that its ZDR commitment applies to the API requests it sends to model vendors, and that features which need stored data sit outside it: “Our zero data retention commitments are not intended to cover these use cases where we are required to retain the data to deliver the feature.” Notion also says that Fireworks had given it a contractual commitment to zero data retention for its use of the platform, and that during the investigation it learned data was retained from the Fireworks endpoints it used.
The case for trusting the contract
The strongest counter-argument is practical. A ZDR clause is the industry’s standard control for AI vendors, and it is what procurement and legal teams can actually negotiate. No customer can audit an inference provider’s storage layer, and few security teams have time to review every subprocessor behind every AI feature in every SaaS tool. On that view, the contract plus breach notification is the realistic ceiling, and the incident shows notification working: Notion contacted affected customers, built a review tool and cut Fireworks off.
I agree that customers cannot audit Fireworks. I disagree that the contract is the ceiling, because most of the exposure here was decided by choices the customer side can see and influence. ZDR, as Notion defines it, covers live model requests. Chat history and admin analytics are features that store prompts by design, and those stored prompts were then sent to a second vendor for classification. A buyer who asked which features retain its data, and which vendors process that stored data, would have found that path before any incident.
What I would ask every AI vendor
- Which features store prompts and outputs, for how long, and whether an admin can switch each one off. Chat history, insights dashboards, search indexes and evaluation sets all count.
- Which subprocessors receive stored data, per feature, including vendors used for analytics or classification rather than for answering the user.
- How ZDR is verified, with evidence the vendor can share: deletion attestations, audit reports that cover retention, or contract terms that give you notice when a subprocessor changes.
- What volunteered data sits in programmes such as early access or training pilots. Notion says customers in its Learning and Early Access Program volunteered data that it used to experiment with training task-specific models on Fireworks.
Notion has said it is now auditing critical vendors such as inference providers and data processors to verify that they follow data minimization practices, including zero data retention. Customers should run the same audit one level up, on the AI features they have turned on.
What it means for the security leader
Treat AI chat logs as a store of credentials until proven otherwise. Notion’s own advice is that anyone who pasted passwords, API keys or other credentials into Notion AI during the affected period should rotate them right away. That advice applies to every AI assistant your staff use, and it is cheaper to act on before a vendor notice arrives: scan exported chat history for secrets where the tool allows it, and tell staff plainly that a chat box is a stored document.
Then put AI features into your data map on the same terms as any other system that holds customer or employee data. The question for each one is where the text goes after the answer comes back. If the vendor cannot answer it in writing, keep the retention-dependent features off.
Earlier this month Wiz tallied 11 AI developer platform incidents, and Fireworks says the evidence links its own incident to a financially motivated LLMjacking group that sells unauthorized access to large language models. Our October argument to guard backups like production applies to AI chat history as well, because it is another stored copy of sensitive data.
Source: Notion Help Center: About the Fireworks AI security incident