A from-scratch explanation of the Stripe billing incident of 2026-07-09 — every term, every tool, customer & internal impact, a system diagram, and all possible solutions. Sourced from the billing chat thread, FINCBO-789 / FINCBO-667, and the internal Billing handover wiki.
Cloudflare is moving all “usage-based” billing onto a new, faster Stripe mechanism called “meters.” Stripe turned on a rule: any price that charges by usage must now be attached to a meter.
Some old “usage” prices never got moved to meters. They sit harmlessly — until a customer (or an internal process) tries to change their subscription. Then Stripe rejects the entire change:
Starting with Stripe version 2025-03-31.basil, metered prices must be backed by meters.
Stripe updates a subscription all-or-nothing, so one bad leftover item poisons the whole subscription — the customer can’t upgrade, downgrade, add, or cancel anything. About 10,000 accounts have ≥1 bad item (~200 paying, rest mostly free). There’s no instant fix: cleanup means “reconciling drift,” and a live code bug in SubsAPI keeps re-creating bad items.
StripeStripe — the outside company that actually charges cards, makes invoices, and stores subscriptions. The cash register + ledger. We don’t own it; we send instructions via its API.
StripeSubscription — a customer’s ongoing billing relationship: “pays $X every month/year for these things.”
StripeSubscription Item (SI) — one line inside a subscription. Subscription = folder; each SI = one product. This incident is about bad SIs.
StripePrice / price_id — reusable object: “this product costs this, billed this way.” Every SI points to a price.
StripeInvoice — the bill Stripe generates each period. A broken subscription can double-bill or repeat invoices.
StripeProration — the partial-period math when you change mid-cycle (a credit for unused time, or a top-up charge). Where accidental “full-year credit” mistakes happen.
Stripe2025-03-31.basil — just Stripe’s version label for the behavior that enforces “usage prices must have a meter.” “basil” is Stripe’s codename, like a version number.
Usage-based / “metered” product — you pay by how much you use, not a flat fee (Workers requests, R2 storage, Spectrum data). Opposite of a flat “Pro plan.”
CloudflareUBB (Usage-Based Billing) — our pipeline that measures each customer’s usage and reports the number to Stripe.
StripeLegacy “usage records” API — Stripe’s old way to report usage (attached to one SI). Slow, bloats subscriptions. The broken kind.
StripeMeter — Stripe’s new usage mechanism: a named stream of usage events attached to the customer, not one SI. ~10× faster. Because it’s tied to the customer, duplicate subscriptions cause double-billing.
Metered vs non-metered price — metered = wired to a meter (new, correct). Non-metered / “legacy usage” = still on the old mechanism (broken). The error means Stripe found a non-metered usage price and refused it.
-meter lookup key — the naming tell: a metered price’s lookup key ends in -meter (a few early-migration exceptions). Support’s quick “is this good?” check.
ToolMigration script / “stripe-resync” — the job that converts old non-metered SIs into new metered ones (swaps in a -meter version).
CloudflareOPE (Object Products Element) — our own source of truth for what a customer has. A row in subs_object_products_element, one product component per zone/account. OPE = what we believe; Stripe = what they’re billed. They must match.
CloudflareObject Products (OP) — the parent record: the billable object (zone/account) the OPEs hang off.
Cloudflarelicense_source_id — field on each OPE saying how it’s billed. sub_X:si_Y = linked to a real Stripe item. This is the bridge between OPE and Stripe.
CloudflareEntitlements — the switches that actually turn features on/off for the customer, derived from billing state.
CloudflareSubsAPI / subscriptions-api / “The Engine” — the central service with all billing brains: computes changes, talks to Stripe, updates OPE, syncs entitlements. Where the live bug was found.
Cloudflarechanges_calculator / “Calc” — the single code path every change flows through: Gather → Validate → Plan → Sync to Stripe + Save to DB. Huge and fragile (no full regression tests).
Cloudflarebilling-webhooks / “The Pipe” — listens to Stripe events (50–100K/day) and feeds them back into SubsAPI (“invoice paid,” “downgrade took effect”).
Upgrade — bigger/pricier plan → applied immediately.
Downgrade — smaller/cheaper plan → delayed to period end so the customer keeps what they paid for.
CloudflareSSLC (Scheduled Service Level Change) — records “at period end, drop this product.” Captures the intent to downgrade later. Biggest user of subscription schedules.
StripeSubscription Schedule — “the sub looks like this now, then auto-changes to that on a future date.” Stores the whole future state — so one bad item inside makes the schedule fail with the metering error when it fires.
Dual subscriptions (by design) — each pay-as-you-go customer normally has two Stripe subs: one monthly (usage: Workers, R2, Stream), one yearly (zone plans: Pro/Biz/Ent). Two monthly = a bug (FINCBO-667).
Drift — Cloudflare’s records (OPE) and Stripe disagree. The metering error is one symptom of drift.
| Type | What it is | Risk | Usual fix |
|---|---|---|---|
| A | In Stripe, not in OPE (orphaned SI) | Overcharging | Remove it from Stripe |
| B | In OPE, not in Stripe | Undercharging / lost revenue | Add to Stripe (free rest of cycle) |
| C / D | Finer-grained mismatches | Both | Case-by-case |
ToolDrift tool — cfbill subscriptions resync-stripe — makes Stripe match OPE (OPE is truth). Use --skip-usage=false for usage items. Has sharp edges (see solutions).
Orphaned SI — a Stripe item with no matching OPE. Usually harmless for revenue (“a stupid record in the way”) but can be the one bad item freezing the whole subscription.
FINCBO — Jira project “Fintech Core Billing & Orchestration” (billing eng bugs).
ESCALATION-xxxx — customer-facing support tickets, linked up to the eng ticket.
ToolKibana — log search; used to count the errors (8,000 → 300).
ToolSigma — query tool over Stripe data; found 3,800+ schedules with the bad Spectrum price.
Write gates — internal on/off + % control (feature flag) for how much traffic uses the new metered write path. “Ramped to 100% Tuesday” = errors exploded.
v84 / go-nogo — the migration’s release version; the meeting that OK’d ramping up, accepting ~100 unmigrated items would be “bashed.”
“bash over” — forcefully overwrite an old non-metered price with the metered one, skipping careful migration. Safe here because only ~100 items actually billed last month.
Bad debt / DNU / ban — account states where the drift tool intentionally won’t act.
Two worlds that must stay in sync. A change starts in Cloudflare, gets pushed to Stripe all-or-nothing, and Stripe events flow back through “The Pipe.”
flowchart TB
USER([Customer clicks upgrade / downgrade / cancel]):::user
subgraph CF["CLOUDFLARE — what we believe"]
direction TB
SUBSAPI["SubsAPI · 'The Engine'"]:::cf
CALC["changes_calculator 'Calc'
Gather → Validate → Plan → Sync+Save"]:::cf
OPE[("OPE — source of truth
subs_object_products_element")]:::cf
SSLC["SSLC — delayed downgrades"]:::cf
ENT["Entitlements — features on/off"]:::cf
UBB["UBB — measures usage"]:::cf
end
subgraph STRIPE["STRIPE — what the customer is billed"]
direction TB
SUB["Subscriptions
normally 1 monthly + 1 yearly"]:::st
SI["Subscription Items (SIs)
each points to a price_id"]:::st
SCHED["Subscription Schedules
future state for downgrades"]:::st
METER["Meters (new)
attached to CUSTOMER"]:::st
INV["Invoices"]:::st
end
PIPE["billing-webhooks 'The Pipe'"]:::cf
DRIFT["Drift tool
cfbill subscriptions resync-stripe"]:::tool
ERR{{"❌ metered prices must be backed by meters"}}:::err
USER --> SUBSAPI --> CALC
CALC -->|write state| OPE
CALC -->|push change ALL-OR-NOTHING| SUB
CALC --> SSLC --> SCHED
CALC --> ENT
SUB --> SI
SI -->|usage price must link to| METER
UBB --> METER
SUB --> INV
STRIPE -->|events| PIPE --> SUBSAPI
OPE -. license_source_id = sub_X:si_Y .-> SI
DRIFT -. reads .-> OPE
DRIFT -. fixes .-> SI
SI -->|bad = non-metered usage| ERR
SCHED -->|holds bad Spectrum price| ERR
classDef cf fill:#16233f,stroke:#3a558f,color:#dce6ff;
classDef st fill:#241a3f,stroke:#6a53b0,color:#e7ddff;
classDef tool fill:#12301f,stroke:#2f7c58,color:#c7f5df;
classDef err fill:#3a1417,stroke:#7c3030,color:#ffc0c0;
classDef user fill:#3a2a10,stroke:#7c5a20,color:#ffe0b0;
sequenceDiagram
participant C as Customer
participant E as SubsAPI / Calc
participant S as Stripe
C->>E: Change subscription (e.g. cancel Spectrum)
E->>E: Build plan from OPE
E->>S: Update subscription (all items at once)
Note over S: Sees a non-metered usage SI
with no meter
S-->>E: ❌ Reject WHOLE update
"metered prices must be backed by meters"
E-->>C: Error — no change applied
Note over C,S: One bad item freezes upgrade,
downgrade, add AND cancel
Separate problems, same error message:
price_1NVI9FCM3YxTUEmtNLh1dCcR. 3,800+ such schedules still active; they fail when the downgrade date arrives.price_1Ofme…: only 3 OPE entries vs 113 Stripe subitems. The non-migration version: 41 OPE vs 1,000+ Stripe (out of ~23K).SubsAPI “smoking gun” 🔫FINCBO-789.Frozen subscriptions — ≥1 bad item blocks all changes (upgrade, downgrade, add, cancel). ~10,000 accounts affected; ~200 paying, rest mostly free.
Blocked cancels (ESCALATION-5104) and blocked upgrades (ESCALATION-5102).
Double / repeated billing (smaller set) — duplicate subs + meters: one account invoiced ~14× in 48h; another stuck at 18–19 “Pro plans.”
Accidental large credits — careless drift-tool use could hand a Pro-annual→ENT customer ~a full year’s credit.
Escalation pile-up — needed a central ticket (FINCBO-789 · P2 Critical).
Likely a formal incident next.
SSLC failing today — scheduled downgrades don’t execute.
Revenue / SOX exposure — drift is compliance-tracked (over/under-charging).
Knowledge risk — drift + SSLC expertise concentrated in one person.
No fast fix — only real remedy is slow drift reconciliation, with caveats.
cfbill subscriptions resync-stripe --skip-usage=false
price_1NVI9FCM3YxTUEmtNLh1dCcR (Sigma query), plus any other non-metered price IDs still in schedules — using the old→new price-ID migration sheet.Lookup key ends in -meter → metered → good (few early exceptions).
Usage price with no meter → bad (what Stripe rejects).
Master old→new price-ID list: migration spreadsheet.
Known bad Spectrum IDs: price_1NVI9FCM3YxTUEmtNLh1dCcR (non-metered, 3,800+ schedules), price_1OfmeRCM3YxTUEmtcuJ24Vdj (~50%-drifted migration variant).
FINCBO-789 · P2 this incident — link escalations here.
FINCBO-667 duplicate monthly subs (double-billing).
ESCALATION-5102 upgrade/downgrade error · ESCALATION-5104 can’t cancel.
Background: BILLACCT-282 race→dup subs · BILLACCT-285 idempotency false-success · BILLACCT-287 null price_id_phase_0.