An AI Vendor’s Breach Reached Customer Datasets, Model Weights and Chat Transcripts

Fireworks AI has confirmed that an October intrusion reached further than first reported, touching customer billing data, incomplete model weights and datasets, with Notion AI chat transcripts caught in the same haul. The scope is still provisional, but the categories are a useful warning for anyone running AI tools without a vendor register.

Most breach notices list the same three things: names, addresses, billing records. The update that Fireworks AI published on November 9 lists those, and then keeps going. Resource metadata. Billing and shipping addresses, invoices, account balances. And, for some customers, incomplete model weights along with the datasets behind them.

Then Notion, which used Fireworks to run part of its AI functionality, told its own customers that the data handed to it by Fireworks, included Notion AI chat transcripts.

That is three categories in a single incident: the data a customer uploaded, the model being trained on it, and what employees typed into an AI assistant. None of them appear on a standard vendor questionnaire.

Worth saying up front: this is not the biggest breach of the month, and the scope description comes from the breached companies themselves, both of whom say their investigations remain open. Treat the numbers as provisional. The useful part here is not the scale. It is the list of categories, which is new.

What Happened

Fireworks AI is an inference and model-training platform. Other software companies build on top of it, which is why its customer list matters more than its name recognition.

The first bulletin went out on October 9, after the company detected unauthorized use of internal credentials. At that stage the finding was narrow: stored secrets used in evaluation jobs had been reached. Fireworks said it had seen no evidence that customer models, datasets or training data were touched, and it revoked and rotated the affected credentials.

The second event is the one that changed the picture. A third-party cloud provider hosting Fireworks research and development machines had its own security incident on October 23. The next day, according to the November update, the intruder used that access to reach Fireworks development machines and run an unauthorized workload, which is where the additional customer data impact came from. Fireworks says it caught the activity the same day and has seen no further successful access since October 25.

The company brought in Palo Alto Networks Unit 42 for forensics and notified law enforcement. It says payment card numbers are not stored and were not involved, and that it has found no impact to its Inference, Training SDK and Nexus products, which it describes as zero data retention by default. Self-serve dataset uploads and downloads for batch inference and dataset-based training have been switched off while the investigation runs.

The motive, per Fireworks, was not espionage or extortion. The evidence points to a financially motivated group reselling access to large language models through unofficial routing gateways, the practice now going by LLMjacking. Google Threat Intelligence Group and Mandiant published research on the technique on November 2. Somebody wanted free compute to resell. The model weights and datasets were cargo that happened to be in the same room.

Notion’s side of it ran on a separate clock. Fireworks told Notion on November 2nd that data had been accessed, initially believed to be synthetic test data only. Notion disconnected from Fireworks on November 3rd and deleted its data there on November 4th. On November 6th it learned customer data might be involved, and on November 8th, Fireworks confirmed the final dataset. Notion says fewer than 1% of workspaces were affected, that the exposed material included AI chat transcripts used by its Insights feature, and that for most affected workspaces those transcripts were unattributable, meaning nothing in them ties a conversation back to a person, workspace or company.

Notion has since revoked Notion API keys found inside those messages and reported third-party keys it found to the vendors that issued them. Its advice to affected customers is blunt and correct: if anyone pasted a password, key or other credential into Notion AI during the window, rotate it now.

Why This Matters for Businesses

Start with the chain. A Notion customer had no contract with Fireworks, and Fireworks had no contract with the cloud provider that was compromised first. Notion is right that its own systems were not breached. That is accurate, and it is also cold comfort to a company whose staff spent a year asking an AI assistant about contracts, salaries and customer complaints.

Second, the data classes are ones that most risk registers do not contemplate. A fine-tuned model is a derivative of your proprietary data, and a training dataset is often the cleanest, most concentrated version of it that exists anywhere in your environment. Chat transcripts are worse in a quieter way, because people paste things into a chat box that they would never email.

Third, certification did not stop it, and nobody should pretend it was supposed to. Fireworks lists SOC 2 Type 2 and ISO 27001, 27701 and 42001 on its security pages. Those attestations say a program exists and was audited. They do not say a subcontractor’s cloud environment will hold.

Fourth, the obligation does not transfer. If regulated data was in that dataset, the notification duty, the contractual exposure and the awkward customer conversation all land on you, regardless of which company’s logo is on the breach bulletin. The relevant question is whether your agreement with the AI vendor says, who tells you, how fast, and in what detail. For a surprising number of AI tools adopted in the last two years, the honest answer is that nobody has read the agreement since signup.

What Organizations Should Consider

The fix here is procedural and cheap relative to the exposure. These are questions to put to your IT or security lead rather than instructions to hand down.

Which AI services currently hold our data, and who approved them? Include the ones embedded in tools you already pay for. The Notion customers in this incident were not Fireworks customers; they were Notion customers whose vendor used Fireworks underneath.

What are we allowed to paste into an AI assistant? One page, written in plain language, covering credentials, customer records, financials and anything under a confidentiality obligation. If the rule does not exist, people invent their own, and theirs is more permissive than yours.

If we fine-tune or upload datasets, what is actually sitting in that vendor’s storage right now? Many platforms retain uploaded datasets by default and delete them only when asked.

What does our contract say happens when the vendor is breached? Notification window, level of detail, subprocessor disclosure, your right to a copy of the affected data. Check whether anyone on your side is named as the contact.

Could we rotate every API key tied to our AI tooling this week? Not should: could. In this incident, keys were found sitting inside chat messages, and the ability to rotate quickly is the difference between an inconvenience and an open door.

Who is the vendor’s vendor? You will not get a complete answer, and it is still worth asking. The failure here began one layer below the company that sent the notice.

None of this requires new tooling. It requires an inventory, a classification rule and the same diligence you would apply to any other processor holding your data. Most organizations already do this for their payroll provider. AI platforms simply arrived faster than the paperwork.

How ISM Can Help

ISM works with organizations adopting AI at the point where enthusiasm meets governance, which is usually about six months after the first tool went live.

AI Readiness & Automation covers the inventory and the usage rules: what AI services are in play, what data they are permitted to see, and what a sensible approval path looks like for the next one.

GRC & Compliance Services handles the contractual and regulatory layer, including vendor diligence, subprocessor review, and mapping which AI-held data carries a notification obligation if a provider is compromised.

Cybersecurity & Risk Management puts the exposure in order of priority rather than order of discovery, so budget goes where the actual risk is.

Identity & Access Management addresses the credential problem this incident keeps circling back to: where API keys live, who can mint them, and how fast they can be revoked.

Talk With Our Team

If you cannot currently name every AI service holding your company’s data, you are in normal company, and it is a solvable afternoon’s work rather than a project. We are happy to walk through what an AI vendor inventory looks like for an organization your size, and what to ask a provider before the breach notice rather than after. Get in touch and we will set up a conversation with someone who does this regularly.

If you would rather take your own measure first, the Cyber Maturity Calculator is a quick way to see how your current controls hold up before you talk to anyone.

Sources