Zillow Offers lost $422M in one quarter of 2021 on a pricing model loss. A model that can buy things is a different kind of tool from a model that only informs a report, and it needs a different level of scrutiny.
What happened
Zillow Offers relied on a pricing model to set the prices it paid, and in the third quarter of 2021 the losses from that model reached $422M.
The model did what it was built to do: produce prices. The problem was that the prices were then acted on at scale, so any error was multiplied by every purchase.
Where an independent check was missing
A model is only as good as the conditions it was built in. When the market shifts, a model trained on earlier patterns can be confidently wrong, and the people closest to it may be the last to question it.
An independent challenge function, whose job is to ask when the model would fail and how that would show up early, is the check that was needed.
What to do differently
Set limits on how much a model can commit before a person reviews the results, and compare its predictions with outcomes continuously rather than quarterly.
Define in advance what would make you pause: a run of misses, a widening gap between forecast and result, or a sudden change in the market.
Keep a person accountable
Someone should own the decision to keep a model running and have the authority to switch it off. Without a named owner, a model keeps running because nobody is responsible for stopping it.
The goal is not to distrust models but to give their output the same scrutiny as any other large financial decision.
- A pricing model loss cost Zillow Offers $422M in Q3 2021.
- A model that spends money needs independent challenge.
- Compare forecasts with outcomes all the time.
- Name an owner who can pause the model.
