# Addon Rotation Does Not Revoke the AI Key.

> Northflank's status page says a Wiz alert arrived at 11:43 UTC on 30 October. At 13:58 it told customers to assume environment variables were decrypted. At 19:28 it had no further confirmed finding. At 23:38 it relayed user reports of AI API key use. Addon rotation does not revoke a key at the issuer.

Source: https://hackerlogs.com/blog/northflank-ai-key-assume-clock/
Published: 2026-10-31

## Key takeaways

- Wiz's alert is 11:43 UTC on 30 October. Northflank reviewed it at 11:44 and patched the network policy at 12:02. The assume-decrypted line is 13:58, 2h 15m after the alert.
- At 19:28 the page says no further confirmed findings. The AI-key sentences at 15:12 and 23:38 are user reports relayed by Northflank.
- Postgres and MongoDB rotation opened at 15:12. MySQL and Redis at 13:34 UTC on 31 October. Minio at 22:32. None of those calls an AI provider.
- Wiz's two Bedrock cases are about 800,000 and about 1.7 million calls. The post does not name Northflank as the vendor in that paragraph.
- The incident was still Identified at 22:32 UTC on 31 October. The banner that ends at midnight is not a close.

Assume every environment variable stored on Northflank was decrypted. [Northflank's incident](https://status.northflank.com/cmvv0rn5a092f14rutwccodpp) says that at 13:58 UTC on 30 October. At 19:28 the same page still had no further confirmed finding. At 23:38 it relayed user reports that AI API keys had been used.

The Hacker News front page this evening was a ChatGPT share, not this status page. US trends at 22:16 UTC were Halloween, boxing, and college football. [BleepingComputer](https://www.bleepingcomputer.com/news/security/northflank-security-breach-leads-to-abuse-of-customers-ai-api-keys/), Lawrence Abrams, is dated 31 October at 02:01 PM, with no zone on the page.

 Identity tokens in a platform secret store. High because the operator instruction is to treat every user-provided secret as gone, while the incident is still Identified. The AI-key sentences are user reports, not a Northflank count.

<img src="/northflank-ai-key-assume-clock.svg" alt="Three panels for the Northflank incident: 2 hours 15 minutes from the Wiz alert to the assume-decrypted line, user reports that are not a finding, and addon rotation that does not revoke an AI key." width="1200" height="675" />

## Assume decrypted is the instruction.

The investigating update is timestamped 13:40 UTC on 30 October. Its body lists earlier minutes. This read did not find a separate status event for each of them.

| Mark | What the page says |
| --- | --- |
| 11:43 UTC | Wiz threat alert |
| 11:44 UTC | Northflank engineers review the alert. 1 minute later |
| 12:02 UTC | Initial network policy patch. 19 minutes after the alert |
| 12:50 UTC | Additional patching |
| 13:00 UTC | Wiz CIRT support requested |
| 13:05 UTC | Additional patching. The note prints this line twice |
| 13:40 UTC | Investigating. "Rotate environment variables immediately." |
| 13:58 UTC | Identified. "Assume all environment variables have been decrypted. Disable/rotate all keys." |

From the alert to that identified line is 2h 15m. The page does not say when access started, how the environment variables were decrypted, or how many customers are in scope. [BleepingComputer](https://www.bleepingcomputer.com/news/security/northflank-security-breach-leads-to-abuse-of-customers-ai-api-keys/) says it asked and had not received a response.

At 15:12 UTC, secret rotation for Postgres and MongoDB was enabled. The same update says multiple users are confirming that their AI keys have been used, and it repeats the assume-decrypted instruction for secrets on workloads, secret groups, and global secrets. It also says to disable public expose on addons if you enabled it. From the identified line to this update is 1h 14m.

At 19:28 UTC the investigation was ongoing. The sentence that matters is "no further confirmed findings." Guidance was unchanged: assume the variables are compromised, rotate secrets in workloads, secret groups, and global secrets, and disable public access to addons where it is on. From the identified line to that sentence is 5h 30m.

## User reports are not a finding.

> The 15:12 line says users are confirming use. The 19:28 line says there is no further confirmed finding. The 23:38 line says users have reported unauthorised use. Those are three different strengths. None of them is a count.

At 23:38 UTC on 30 October the page says users have reported unauthorised use of AI API keys supplied to the platform. It tells customers who have not rotated to do it immediately: revoke at the issuing provider, generate replacements, and update the secrets and workloads in Northflank. It says to review AI provider usage and billing logs and to report suspicious usage to Northflank and to the provider. From the 19:28 "no further confirmed findings" line to this update is 4h 10m. From the 15:12 user-confirmation line it is 8h 26m.

The same update says the reports may be consistent with a broader pattern, and that similar IP addresses are in [Wiz's Black October post](https://www.wiz.io/blog/black-october-for-ai-developer-platforms). Northflank does not list the addresses. Wiz, Benjamin Read and Rami McCarthy, is dated 30 October and prints no minute. It lists nine disclosures, with Northflank as the 30 October row, and says there is no single vulnerability linking them. Do not fold Railway, OpenCode, Warp, or the other names on that list into this clock.

Wiz then describes two Bedrock cases it can attribute to one of those vendor breaches. It does not name Northflank in that paragraph. In both, the first unauthorised call came within 30 minutes of the reported credential access inside the vendor breach. One key then saw about 800,000 Bedrock calls over thirty hours. The other saw about 1.7 million over thirteen hours. Wiz says list price for the two cases together likely ran over 100,000 dollars. That figure is Wiz's, for cases it did not pin to this status page.

Wiz's IP table is explicit in the other direction. The addresses were used to test stolen keys. Wiz says it does not have evidence that links them directly to the nine disclosures in the post. A "similar IP" sentence on Northflank's status page is not a published match, and it is not those call counts.

## Addon rotation is not the AI key.

The impact clock for an AI key is the provider that issued it. The package clock here is Northflank turning rotation on for addons it operates. Those are not the same minute, and they are not the same credential.

| Update | What became available |
| --- | --- |
| 30 October, 15:12 UTC | Postgres and MongoDB rotation |
| 31 October, 01:04 UTC | Audit logs for every customer, including free users. 1h 26m after the 23:38 report |
| 31 October, 13:34 UTC | MySQL and Redis rotation. 12h 30m after audit logs |
| 31 October, 22:32 UTC | Minio addon rotation. 8h 58m after MySQL and Redis |

From the 13:58 assume-decrypted line to the Minio update is 1d 8h 34m. The page still said Identified at 22:32 UTC. It did not say resolved. The header reads "Major outage from 1:40 PM to 12:00 AM." Updates continue after that midnight. Do not treat the banner as a close.

The audit-log line says the logs were enabled. It does not say they cover the hours before 01:04 UTC. A log that starts then cannot show a call the provider already accepted.

> Postgres, MongoDB, MySQL, Redis, and Minio are credentials Northflank can rotate because it runs those addons. An API key you pasted into an environment variable stays valid at OpenAI, Anthropic, Google, AWS, or whoever issued it until that issuer revokes it. Replacing the value in Northflank after the old key has already been copied does not unwind usage.

## What to do tonight.

Revoke user-provided secrets at the issuer, then write the replacement into Northflank. Do AI and payment keys first. Then cloud tokens. Then database and addon passwords, including the rotations the status page has switched on. Disable public access on addons if you enabled it.

Read provider usage from before 11:43 UTC on 30 October. The page never dates the start of access. A bill that begins after the status post is the late part of the clock, not the whole clock. Report suspicious use to Northflank and to the provider, which is the 23:38 instruction. Do not file Wiz's 800,000 or 1.7 million calls as your invoice, and do not file them as Northflank's, unless a provider log shows the key.

The incident was still Identified at 22:32 UTC on 31 October. No customer count and no root cause were on the page. The operator fact is the 13:58 instruction, 2h 15m after the alert, and it has not been walked back.

## Sources

- [Security incident](https://status.northflank.com/cmvv0rn5a092f14rutwccodpp) (2026-10-30)
- [Northflank security breach leads to abuse of customers' AI API keys](https://www.bleepingcomputer.com/news/security/northflank-security-breach-leads-to-abuse-of-customers-ai-api-keys/) (2026-10-31)
- [Black October for AI Developer Platforms](https://www.wiz.io/blog/black-october-for-ai-developer-platforms) (2026-10-30)
