Churn prediction at Tibber: from risk score to time-to-event

Tibber sells energy as a subscription, and in subscription energy the relationship is the product. The electricity itself is undifferentiated — a kilowatt-hour is a kilowatt-hour — so what a customer keeps paying for is the software around it, the price mechanism, and the sense that somebody is paying attention on their behalf. That makes churn the central risk in the business model rather than a marketing metric sitting downstream of growth.

The program covered four countries and roughly 550,000 customers. Four markets is enough that the differences between them stop being anecdotes: tenure profiles differ, seasonality differs, and the reasons people leave differ. One retention play applied identically across all four is the same compromise made four times, and at that scale the compromise has a price you can read in the cohort curves.

This is also the point where I stopped commissioning churn models and started writing them. Not out of any preference for the work itself. The distance between the person who owns the retention number and the person who fits the model is exactly where a retention strategy quietly turns back into a campaign calendar, and the cheapest way to close that distance was to learn to do both.

Gradient-boosted accelerated failure time models — XGBoost-AFT — fitted on Databricks against customer-level history. AFT rather than a classifier because most of the customer base has not churned yet: their true lifetime is unknown and only bounded below by however long you have been watching. A survival model treats a still-active customer as partial evidence. A classifier has to label them “did not churn”, which teaches it that everybody currently on the books is safe.

The features, at the level worth describing

Cohort behaviour rather than campaign response: tenure, consumption pattern and how stable it is, billing and payment history, support contact, and the market the customer sits in. Every feature reconstructed as it looked on the day being predicted from, never as it looks now. A table that leaks the future produces a beautiful validation score and no predictive value, and it is the most common way this work quietly fails.

One model family, four markets

Four markets do not want four unrelated models, and one model pretending the markets are the same is worse than either. What works is a shared model family with the market as a feature and calibration per market on top: the mechanism is common, the differences are explicit rather than accidental, and a change made for one country is visible to the other three before it ships.

Where it ran

Inside the company’s own environment, on its own data, refreshed on a cadence the retention program could plan against — and written back to where the orchestration layer could read it. A prediction that lands only in a dashboard changes nothing; the score has to arrive where the decision is taken, on the day it is taken.

A survival model can be scored on its own terms: concordance, calibration against observed tenure, and whether it beats the naive rule — tenure plus days since last activity — by enough to justify maintaining it. That is table stakes. It is also not the interesting question.

The interesting question is whether acting on the model beats not acting on it, and only one design answers that. A holdout: randomly withheld inside each treated segment, agreed before the first message goes out, and read against retention and contribution rather than opens and clicks. Defined afterwards, a holdout stops being a measurement and becomes a way of explaining a result you already have.

What the design has to survive

• Randomised inside the segment, not across the base. A control drawn from everybody answers a question nobody asked.
• Large enough to see the effect you care about. A holdout too small to detect the difference is a control group in name only, and it will report “no effect” for either reason.
• A window long enough for the event. Churn is slow. A two-week read on a six-month behaviour measures the campaign, not the retention.
• The same definition of churn the model was fitted on. Measuring one event and modelling another is the expensive mistake, and it is invisible until somebody reconciles the two numbers.

This is the discipline that makes it possible to say a thing did not work, including when the thing was mine. It is also the reason a lift figure borrowed from another company is worth so little: a number produced under somebody else’s churn definition, in their markets, against their baseline, tells you nothing whatsoever about yours. What moves between businesses is the design, never the number.

Settle the definition of churn with finance before the first feature is written

A churn event and a reported churn number are not automatically the same object, and the gap between them is discovered late and expensively — usually in the meeting where the model’s result is compared with the board pack. It is now the first thing written down on any engagement, before a table is built.

Ship the boring baseline first

Tenure plus days since last activity, wired into the program and running against a holdout, before the survival models exist. It costs about a week. It gives the real model something to beat, and it builds the measurement plumbing at the point where nobody is yet emotionally attached to the result.

Design the intervention before the prediction

A time to event is only worth having if something genuinely different happens at week two than at week ten. I would now write the intervention calendar first and let it specify what the model has to return, rather than fitting a model and then asking the program what to do with it. Half the modelling choices answer themselves once that document exists.