Contact
FR EN DE
Our expertise
Apps & AI
Consulting & Support
Cloud & IT Architecture
About References Careers Blog
All articles
AI & Data

AI embedded in your tools: your conversations have subcontractors

Notion AI conversations were stolen at Fireworks AI, a vendor most users had never heard of. What the incident says about AI built into software, and the questions to ask before switching it on.

AI embedded in your tools: your conversations have subcontractors
Your conversations with Notion's AI leaked at a vendor you had probably never heard of.
I often hear: “the AI is built into our tool, so our data stays with the vendor”. Reassuring, but the incident Notion detailed this week shows it is rarely that simple.
What happened, in plain terms
Notion offers an assistant, Notion AI, right inside its pages. For part of its processing, the company relied on Fireworks AI, a platform that runs open models (whose weights are published) on behalf of other companies. Notion used it in particular for “Insights”, a feature that helps administrators understand how their teams use AI, by analysing conversations.
In late October, an attacker compromised a third-party cloud host that housed Fireworks development machines, then used it to reach those machines. In early November, Fireworks notified Notion. On 8 November, the verdict came in: the stolen dataset contained Notion AI conversation transcripts.
Notion says fewer than 1% of workspaces are affected, and that most conversations cannot be tied to a user or a company. But it recommends treating that data as exposed, and had to revoke API keys found in users' messages. Fireworks, for its part, believes the attackers were after reselling access to the models.
Three companies between the keyboard and the hacker
What strikes me is the length of the chain. The user talks to Notion. Notion hands part of the work to Fireworks. Fireworks runs machines at yet another provider. It was that last link, which almost nobody had heard of, that gave way.
And yet Fireworks had contractually committed to retaining none of Notion's data. The “zero retention” principle is real: by default, requests live only in memory, for as long as the response takes. But some features (batch processing, stored responses, training) necessarily write data somewhere. The contract reassures. The architecture decides.
Embedded AI redraws the map of your data
When I work on these topics, I often see the same thing: the tool was approved years ago by procurement and security, then an AI feature was switched on with one click, with no new review. Yet that feature sometimes adds new subcontractors, new storage locations and new retention periods.
On the business side, the risks are concrete:
Confidentiality: whatever teams paste into an AI chat (meeting notes, projects, customer data, passwords) becomes data in transit.
Compliance: every new subcontractor must appear in your GDPR register and in your risk assessments.
Sovereignty: the real chain often crosses several jurisdictions and several clouds, well beyond the vendor you chose.
The questions I ask before switching on an AI feature
Three questions already clear up a great deal:
Who runs the model? The vendor itself, a major model provider, an inference platform? The list of subcontractors must be public and up to date.
Where are conversations stored, and for what purpose? Analytics, product improvement, batch processing: every purpose means storage.
For how long, and who can delete them? A no-retention commitment must cover every feature, not just the main call.
And one simple rule for teams, whatever the tool: no secrets in an AI chat. A password or key pasted into a conversation must be changed, not just deleted.
Conclusion
AI built into everyday software is convenient, and it is often employees' first contact with generative AI. But it is also the AI we govern the least. Like any new subcontractor, it deserves a proper review before being switched on: one hour per tool, with the right questions, avoids a lot of nasty surprises.
Have you switched on AI in your SaaS tools? Do you know who processes your conversations? If you are doing that inventory, or wondering where to start, I would be glad to talk it over.
Sources
Notion, security advisory on the Fireworks AI incident: https://www.notion.com/security/advisories/security-incident-involving-a-third-party-ai-vendor
TechCrunch, 12 November 2026: https://techcrunch.com/2026/11/12/cyberattack-on-fireworks-ai-allowed-hackers-to-access-notion-customers-chat-transcripts/
BleepingComputer: https://www.bleepingcomputer.com/news/security/notion-says-fireworks-breach-exposed-customer-ai-chat-transcripts/
VentureBeat, on “zero retention”: https://venturebeat.com/security/attackers-breached-the-fireworks-ai-storage-where-notion-ai-chats-sat-outside-zero-data-retention
Author CAPCOD
Ready to start?

Let's talk about your project.

Tell us what you need, we reply within 48 hours.

Get in touch