The Number Was Never Doing the Work: AI, BI, and the Limits of Better Evidence

Weaving Intelligence (House AI Style), with Jeff Shabel’s Experiential Commentary

Numbers don't move people. I said it in those words when somebody asked me to walk through how a figure on a screen changes a person's mind, and it runs against most of what gets sold in this category. Then I went looking for the place where it fails, and I found one. What I could not find was the edge of it. By the time I was finished I had lost the reading I started with. Not the facts. The facts held. What I lost was what I thought they meant.

The longer form of that answer is more careful than the slogan, and it is the claim this piece puts under pressure. A small number gets taken up by whoever it already agrees with and used to support a decision that was leaning that way anyway. A large one, at most, sends a question back for another look. In neither case is the figure what decided. Which means that when the reporting estate is sold as the thing that makes decisions better, it is being sold for the one job I do not think it does.

That is a strong thing to say about an industry that has spent a great deal of money on the opposite premise, so the rest of this piece does two things at once. It sets out the field's own case at full strength. And it goes looking for the evidence that would prove my answer wrong, on the principle that the evidence which agrees with you arrives on its own and the evidence which does not has to be hunted.

What the field is selling, in its own words

The consensus first, and stated properly, because it is better than the marketing wrapped around it and I would sign most of it.

Gartner named decision intelligence a top data-and-analytics trend and described a practice that “brings together several disciplines, including decision management and decision support”, one that “provides a framework to help data and analytics leaders design, model, align, execute, monitor and tune decision models and processes in the context of business outcomes and behavior” [1]. The word carrying the weight there is discipline. What gets instrumented is the decision. Not the report about the decision.

Set that beside what a buyer actually hears in the room, because the two have drifted apart and the drift is where the money goes. What is written down is a practice — model the decision, agree the measure, watch what happens afterwards, tune it. What gets heard is a product that will make the choosing easier. One of those is a change to how an organization works and takes a year. The other is a purchase order.

The AI half arrives in the same forecast under a different heading. Augmented analytics has the machine doing the pattern-finding and the explaining, so that “the most relevant insights will stream to each user based on their context, role or use” rather than waiting behind a screen somebody has to remember to open [3]. That is a real change and it is moving quickly. The same firm expects generative models to shape three-quarters of new analytics content by 2027 [4].

Read as a design goal it is admirable. Somebody looked at a manager with a report she had to remember to open, concluded that the finding should come to her instead, and started building. That is a reasonable call.

And the augmented half of it isn't vapour. A system that watches a measure and speaks up when the measure moves is doing something no person can do at three in the morning. What I'm arguing with was never the machinery. It's the sentence sitting underneath the machinery, unexamined and load-bearing, which says that decisions come out badly because the evidence arrived too slowly.

Here is where the field argues with itself, and the argument matters more than either side of it lets on. Cassie Kozyrkov, who has done as much as anyone to popularise the discipline, frames it as a fusion of data science with the social and managerial sciences, on the grounds that the hard part of a decision was never the arithmetic [2]. That puts the bottleneck inside the person. The tooling market puts the bottleneck in the data. Those are not the same claim, and which one a buyer believes decides nearly everything about what that buyer should purchase.

What the research settled before any of this shipped

The uncomfortable part is that the question is not open. People have been measuring what evidence does to a choice for about half a century, and the findings are unflattering to everybody selling a faster answer, myself included.

Feldman and March argued forty-five years ago that organizations gather more information than they use and then ask for more. Information works as a signal of competence, and as a justification for choices that were already made, while presenting itself as an input to them [7]. That paper is short, it is old, and I know of no better description of a status meeting.

What they described isn't stupidity and it isn't bad faith, which is why it survives everything anybody does to it. Information gets requested because requesting information is what a competent organization looks like, and the request is sincere — the people asking would tell you, honestly, that they're waiting on the number. Then the number lands, the meeting resolves in the direction it was already pointing, and the number is in the minutes. So the minutes record that the number decided. Nobody in that room has lied to anybody, including themselves, and that is exactly why the pattern is so hard to see from inside it.

Klein came at the same shape from the other end and found it waiting for him. Experts under time pressure do not score options against each other. They recognize a situation as typical, simulate a single course of action, and produce the analysis afterward [8]. The analysis is real work. It is simply not the work that chose.

That distinction is worth holding onto, because it's the first thing lost when this research gets summarized. Klein wasn't saying experts don't analyse. He was saying the analysis runs after the recognition, as a check on it, and that the checking is what expertise looks like from outside. Take that seriously and a great deal of what a reporting estate produces is doing its real job in the second position rather than the first — not choosing, but catching the recognition that was going to be wrong.

Kahan and colleagues went hunting for the people numeracy protects, and did not find them. Skill helped when a result agreed with what a subject already believed. Skill stopped helping when the result cut the other way [9]. More capacity to reason, aimed the same direction the reasoner was already facing.

Now the seams, because they are real.

Kahan's subjects were reading politically charged data, not a capital-approval pack, and the wider story that facts never change minds has taken a beating it earned. Wood and Porter ran five experiments across more than ten thousand subjects and found that people do update factual beliefs when corrected, including on the topics where they were supposed to dig in [10]. I take that seriously. It is the strongest thing anybody has against the argument I am about to make, and it holds.

What their result shows, precisely, is that a corrected fact lands. What it does not show is that the plan changes — and in a business the plan is the thing with money attached. Tell a Director that the conversion figure he had in his head is two points lower than he thought, and he may well accept it, say thank you, write it down, and approve the campaign anyway, because the campaign was never resting on those two points. He is not being irrational. He is holding a dozen things nobody else in the room can see, and the correction has moved one of them.

What it forces is a distinction, and the distinction turned out to be the useful part. Beliefs update far more readily than choices do. That gap has a literature of its own. Webb and Sheeran's meta-analysis found that a medium-to-large shift in intention produced only a small-to-medium shift in behaviour [11]. A stakeholder who nods at a readout is not lying to the room. He is reporting an intention, and an intention is a weaker predictor of what he will do than either of us would prefer.

So the reading I carried into this piece, the one in my own answer, went roughly like this. Evidence arrives late, into a room where the leaning already exists. A small number gets picked up by whoever it agrees with, and gets used as support. It works like a footnote added after the paragraph was written — it supports a conclusion rather than producing one, and from inside the meeting nobody can tell the difference.

The first thing that says I am wrong

I went looking for the observation that would break that reading, on the principle that confirming evidence arrives on its own and the disconfirming kind has to be hunted. It exists, it is published, and it is not close.

It is tempting to treat experimentation as a different trade — product work, not reporting work, somebody else's floor of the building — and a boundary like that is very convenient when the evidence on the other side of it is inconvenient. A claim about how people use numbers does not get to stop at the edge of a job title.

The place to look was obvious once that boundary was out of the way. A setting where the inclination and the number are known in advance to point opposite ways, and where somebody wrote down what happened. A team builds a feature it believes in. A controlled experiment then grades the feature. The builder wants one answer and the arithmetic supplies another, in public, with a record.

Microsoft's account of running that at scale reports that only about a third of well-designed experiments improved the metric they were built to improve. Roughly a third actively hurt it and were stopped. The rest came back flat and were also stopped [19].

The paper is careful about the four ways an experiment can land, and the carefulness is the point. An idea can be good and the experiment can show it. An idea can be thought good while the experiment shows that it hurts the very metric it was built to lift — at which point stopping the launch saves the company money, and that outcome, on their own count, is about a third of everything they ran. A further tranche comes back flat, and flat is also a stop. Then the authors draw the general conclusion out loud rather than leaving it for a reader to assemble: the results bring into question whether prioritizing work in advance of measurement is as good as most people believe it to be [19].

Read that slowly. The builder wanted it. The number said no. The number won, routinely, dozens of times, against people with every reason and every opportunity to argue back. On its face that is not a caveat to my line. It is a refutation of it, and I would rather set it down at full strength than leave it in a footnote for somebody else to find.

So the question became why the number wins there and not in the meetings my own answer describes. Nothing about those numbers was bigger or cleverer than the ones being ignored elsewhere in the same building, by the same people, on the same afternoon. What differed sat upstream of the measurement entirely. The measure had a name before the result existed. That is what an overall evaluation criterion is in this literature — the single quantity a variant will be judged on, settled while the experiment is being designed rather than picked out of the output afterwards — and the paper names one for every case it works through: the return rate of a tracked cohort over six weeks for the pre-roll advertising test, the clickthrough rate on one section of the page for the support-site test [19].

In the one case the paper works through in full, the agreeing went further than the measure. The MSN home page team wanted to put three offers below the shopping module, which for most readers would have sat below the fold, and the display-ads people expected tens of thousands of dollars a day out of it. The hard part was never the arithmetic. It was pricing a lost click against ad revenue, so they settled that first — what a page view was worth, what a click through to another property was worth, estimated two independent ways, and the authors record that the numbers “were close enough to get agreement on the monetization value to use”. Then the experiment ran on five per cent of users for twelve days. Clickthrough fell by about a third of a per cent, page views per user-day by the same, both significant, and the monetized value of the lost clicks came in above the expected ad revenue. The idea of placing more ads was, in their word, appropriately stopped [19].

Now notice what was fixed in advance there and what was not. What was fixed was the measure and the price of the measure — the exchange rate that would otherwise have been argued about after the result, by people who by then knew which way the argument pointed. What was never fixed was a threshold or an action. Nobody wrote down that a 0.35 per cent drop in clickthrough would kill the proposal.

It would be tempting to say the organization could not renegotiate the rule once it could see the answer. Read end to end, the paper reports something much closer to the opposite. Its cultural sections are about people arguing, for years. Some viewed experimentation as a risk to their power or their prestige; the authors quote “we know what to do. It's in our DNA” as a thing they actually heard. On reward systems they are blunter still: most goals in Microsoft's software organizations were about shipping products, not about their effect on customers or key metrics, and tying goals to metrics instead is offered as one possible change, in the conditional. Stopping a launch on a flat result is, in their word, a recommendation. And they close that section by saying it is hard for them to judge whether they had changed anyone's goals at all, and that they had probably not dented many people's yearly performance reviews [19].

So the mechanism I want is real but smaller than I made it. It is not an organization that cannot renegotiate. It is a measure, and a price for the measure, agreed by people who did not yet know which way the answer would go — which takes the one argument that would otherwise decide the thing off the table before anyone has a stake in winning it.

That is not an evidential act. It is a structural one, taken before the evidence existed. And the winning was not clean even there. The same account records that “when we first shared some of the above statistics at Microsoft, many people dismissed them” [19]. They had to build the platform, and then they had to win the argument about believing what the platform said.

Which leaves my claim narrower and considerably more useful than the one I opened with. Evidence becomes the mechanism only where a binding decision rule was fixed before the number arrived. Absent that rule, size rescues nobody and the inclination decides. What defeats ratification is procedure, not magnitude.

What I said when asked for the mechanism

The experiments supplied the procedure. The other half of the picture was already in my own answer. Asked for the mechanism — what happens in the seconds between a number landing on a screen and a person changing course — I declined the premise rather than answering the question:

You are asking core Sales questions. Simply, numbers don't move people — or, more accurately, small numbers don't move people. You are not going to see a project move from denied to approved over a budgetary change of a few percentage points. All changing a number can do is exert pressure in a certain direction. If the person's inclination is already in that direction then the number will be embraced and used to support that decision — but it is not the number doing the work, it is the inclination. The only time numbers move people is when they are reinforcing some other position someone already holds, or they're of sufficient magnitude to force them to reconsider their position.

The first half of that is the part that travels, because it is the half people want to hear. The last sentence is the one that matters. It says where a number does get through: when it lines up with a position somebody already holds, or when it is big enough to force a second look.

I had this written, at one point, as a big enough number overturning the inclination. My own correction, when pressed, took that escape hatch away:

It doesn't necessarily overturn it, at best it prompts reconsideration. The number combined with other pre-existing factors is what tips the balance and the number is just one of those factors.

Small numbers ratify. Large numbers, at most, reopen. Neither settles anything by itself, and the balance tips on the combination rather than on the figure. It would be convenient if a bigger number were the answer, because a bigger number is something a practitioner can go and build. It is not.

What's left is a claim with no lever in it at all, which is precisely why it's worth having. A claim with a lever in it is a claim somebody can sell.

The experiments and my own answer agree less than they seem to, and the seam is worth marking because the next case falls straight through it. My answer demoted magnitude — a large number reopens a question, it does not settle one. It did not remove it. Saying that procedure had replaced magnitude is a tidier sentence than either piece of evidence supports. What holds either way is that neither procedure nor magnitude is a thing a dashboard does.

The rule the prescribers never had

The narrower version — that a number changes a decision only where a rule was fixed before it arrived — had the word only in it. That makes it a universal negative, which is an expensive thing to say and a cheap thing to break, so I went looking for the break the same way I found the first one. What I needed was a documented population of decisions taken with no rule fixed in advance, where the number moved people anyway. The experimentation literature cannot answer that, because there the rule is fixed by construction — that is what makes it an experiment. Clinical practice can, because a doctor writing a prescription on a Tuesday has agreed nothing in advance about what a trial is going to oblige them to do.

In July 2002 the estrogen-plus-progestin arm of the Women's Health Initiative was stopped early. Sixteen thousand six hundred and eight women, a trial scheduled to run until 2005, halted after an average of 5.2 years of follow-up because invasive breast cancer had crossed a monitoring boundary. About six million American women were taking the combination at the time. The announcement carried a twenty-six per cent increase in breast cancer, twenty-nine per cent in heart attacks, forty-one per cent in strokes, set against a third fewer hip fractures, more than a third fewer colorectal cancers, and no difference at all in total mortality [20].

Then look at what prescribers did with it. A six-year observational cohort at one army medical centre — 71,592 hormone-therapy prescriptions, July 1999 to July 2005 — recorded monthly prescriptions falling from 1,272 at the start of the window to 493 at the end, with the authors reporting a significant decrease following the release of the WHI result [21]. Nobody in that building had written down in advance what a null trial would oblige them to do. Prescribing was the standing inclination of an entire specialty and had been for two decades. The number moved them anyway.

That is the word “only” gone from my claim, and I am not going to argue with it. But there is a rescue sitting right there, and I want to name it precisely so that nobody reaches for it later, myself included. The rescue is that the WHI did have a rule fixed before the number arrived. Its monitoring board pre-specified the boundary, and the biostatistician who led the analysis said so plainly at the time: because breast cancer is so serious an event they set the bar lower to monitor for it, they pre-specified that the change in rates did not have to be large to warrant stopping the trial, and they stopped at the first clear indication of increased risk [20]. Every word of that is true, and it does not save my sentence. The rule bound the trial. It did not bind the hundreds of thousands of prescribing decisions the trial reversed. If I am allowed to count somebody else's stopping rule as my prescriber's pre-commitment, then every effective number in history has a rule somewhere behind it, my claim rules nothing out, and it has stopped being a claim while keeping the shape of one. That is a worse outcome than being wrong.

What the case does hand me is a second route into a decision, and it is the one my own answer gave two sections ago, the one the tidier sentence put down. A number large enough reopens the question. This one reopened it for an entire profession inside three months [22]. So the honest version has two routes rather than one, and only the first is anything a business can arrange.

Two qualifications, because a counterexample this good deserves reading rather than waving at. The reversal was neither clean nor fast. The same paper records that the obstetrician-gynaecologists had begun cutting their prescribing before July 2002 and were ahead of the other specialties in doing it, so part of that fall predates the number entirely, and the fall it measures runs across six years rather than across a meeting [21]. And a separate national study of roughly 340,000 women found prescriptions dropping from 12.5 per cent to 9.4 per cent within three months of the trial — while the wide regional variation in who prescribed hormones survived the whole event, its authors concluding that local practice patterns exert a strong effect on clinical behaviour even after new evidence is available [22]. That is the inclination, still visibly doing a great deal of the deciding, inside the strongest counterexample I was able to find.

So here is the claim in its third and I hope final form, with the part that matters attached to it. Inside the range of evidence a business actually produces — a measure moving within its ordinary band, arriving on a screen, to people who are each free to decide privately what it meant — a number does not change a decision unless a rule was fixed before it arrived. Two properties take a number out of that range, and I am naming them now rather than afterwards, so that nothing can be rescued by declaring whatever happened to work to have been large enough. The number has to reverse the standing recommendation rather than adjust a quantity inside it. And it has to reach every decider at once, publicly, carrying an authority that no single decider can quietly relitigate. A dashboard has neither property, and there is no version of the product that gives it one.

Which means the claim still rules things out, and cheaply. It rules out finding decisions changed by ordinary reporting where no rule was fixed — and the fifty-decision protocol I set out near the end of this piece is exactly the measurement that would turn them up, which is why I would take it as a refutation rather than as noise. It rules out saying that an organization which fixes a rule then cannot argue with it, because Microsoft's own record is three years of people arguing. And it keeps me from getting my opening sentence back, because in the summer of 2002 a number cut hormone prescribing by a quarter across a 340,000-woman pharmacy-benefit population within three months [22], and the only honest thing to say about it is that nothing available in this category is going to do that for anybody.

Where the work earns a yes

If the promise is not better decisions, what is left? Quite a lot, and it is worth being precise about where it sits, because the AI work that gets bought and stays bought lives below the decision-consideration threshold — the point at which a problem becomes worth anybody's deliberation at all.

Here is my worked example. Picture a Director who knows there is a leak. Volume goes missing somewhere in the supply chain. Not dramatically. Persistently. Nobody finds it in the course of a normal working day, because nobody's day has room in it to go looking, and the loss is too small to justify a dedicated person or a project of its own. The cure has always cost more than the disease. So the leak sits there, being irritating, for years.

Nothing about that is a failure of will. The arithmetic genuinely doesn't work — half a person for a quarter, against a loss nobody has sized, precisely because sizing it would itself cost half a person for a quarter.

Put an agent on it and the arithmetic changes. I put the trade this way:

for the equivalent of a few days or a week of a dedicated resource's time, an AI can target the problem and trace the data. If the Director has the spare budget, this is an easy sale, because it solves a real problem at a cost below the decision consideration threshold.

Notice what did the work in that transaction. Not a better number. A long-standing irritation belonging to a specific person, priced under the amount that would have forced a conversation about it.

Sales books call this making it easy to say yes. That is folklore rather than research — I went looking for the study and did not find one — but it names the move I would make anyway. What has to fall below the ongoing cost, as I put it then, is the cost “to the decision maker — not to the business”. Those are two different numbers, and only one of them is ever in the room.

A project that ends something personally annoying to the person deciding, as a byproduct, clears far more easily than a project that does not. It clears more easily still when the money comes out of another department's budget — which isn't a smaller cost to the business, it's exactly the same cost, but it is a smaller cost to the person being asked to say yes, and that person's budget is the only one the decision is really running on. Read it as a diagnostic rather than as advice about selling: the proposal that clears is not reliably the strongest case on the table, it is the one whose irritation belonged to the person deciding, which is why a portfolio of approved projects, read end to end, so often looks like an accident of who was irritated by what.

The qualifying question. Not what is this worth to the enterprise. An arguable number loses to an inclination every time, and the enterprise figure is always arguable. Ask instead: whose recurring irritation does this end, and does the price sit below the threshold at which that person would have to justify it? If nobody in the room can name that person, the strongest case in front of the room is not the one being presented. It is a business case, which is a different object.

Below-the-threshold work comes in about three shapes, and it is worth being able to spot them, because they are the ones that pay. There is the hunt — a leak, a duplicate, a slow drift in a field nobody owns — where the value is patience rather than judgement. There is the assembly, where somebody spends four hours a month building a pack that gets read for eleven minutes, and the machine does the four hours. And there is the watch, where a measure gets a pair of eyes on it overnight and speaks up on a move that would otherwise be found on Monday. None of those three changes a decision. All three give hours back to somebody who then makes their decisions the way they were always going to, which is a real return and an honest one, and it is nothing like what is printed on the box. Nothing in the decision literature is troubled by any of that, because nothing in the decision literature is about it.

The two best finds I can point to from governed data were hunts of exactly that kind. Once the expense systems of several departments were tied together through the master data management system, the joined records showed contract and software licensing redundancy across the enterprise, down to no fewer than three resellers selling the company the same software. In another case, governed supplier data showed regional redundancy that went past anything in the company's plans, including a regional supplier sitting in the wrong region. I am leaving both unquantified on purpose. In the first, the waste lived between departments, where no single department's screen could show it, and it surfaced only when somebody joined what the departments held separately. Nobody funds a project to look for a supplier in the wrong region, because nobody suspects there is one to find.

Where it does not: the conversation nobody was having

A disclosure first, because this section assesses vendors' products. The practice behind this publication is a Profisee partner and holds a referral agreement with Profisee. Clients pay us for deployment work, and a licence sale we refer earns us a referral fee from the vendor. Beyond both, our services revenue depends on data management and analytics software being bought at all, so what I say about the promises made for it is said by people with a stake in the category. None of the products I examine below is Profisee's. Weigh my criticisms above any praise.

The most demonstrated capability in the category is the plain-English question box, and it is where I would expect the quietest failure. Quiet because nothing crashes. The thing works, and everybody involved will say so.

These pilots die because they cost too much and give the business too little. My two reasons, side by side:

The whole "conversation with your data" sales pitch is great save for two things — one, nobody talks with their data now; and two, natural language processing is more expensive, token-wise, than other AI implementations.

I cannot hand over a study proving these pilots fail. What I can point at is the adoption record any pilot has to climb out of. Business-intelligence penetration has sat at roughly a quarter to a third of employees, broadly flat across a decade of tooling spend [12]. My argument is consistent with that record rather than proven by it, and I would rather say so than dress it up.

The closest thing to a verdict at scale is a vendor's own. In 2024 Tableau retired Ask Data, its natural-language query feature, from Tableau Cloud at the 2024.1 upgrade and from Tableau Server at 2024.2. The reasons it gave were that customers found it “isn't 'natural' enough” — they still had to understand field names, keywords and basic analytical concepts to use it — and that “the declining adoption rates for Ask Data on-premises made us confident that this is the correct plan for the future” [23]. One product's withdrawal is not a study of pilots, and the vendor presents its generative-AI successor, Tableau Pulse, as the replacement rather than an abandonment. But it is a vendor telling its own customers, in public, that people did not keep talking to their data.

For a plain-English interface to pay for itself, something cultural has to happen first. A business has to genuinely want to talk to an agent about its numbers. On the executive floors that is plausible. A Director's day is already wall-to-wall conversation, so one more conversation costs very little.

A floor down, the premise comes apart, and I will defend that rather than soften it. Analysts do not run on conversation. I flagged the generalization myself when I first made it: “I know I am embracing a stereotype here, but self-selection and survivorship are real things”. The people who choose this work tend to prefer the terminal to the meeting.

More to the point, they are already deep in that conversation. It happens in SQL, in Python, in R. The less technical hold up their end in a spreadsheet, or in a report they built for themselves that nobody else quite understands.

The cost side compounds the same problem from the other direction. Putting a sentence in and getting a sentence back is, token for token, among the more expensive things a business can ask a model to do, and the population who'd use it most heavily is the population already fluent in the cheaper interface. So the spend lands where the need is thinnest. That isn't a scandal — it's a very poor shape for a licence priced per seat across a whole estate.

So the capability solves a real problem for a small, senior population, at a price set by the whole estate. My verdict at the time: “it is solving a problem that does not exist save at the senior, non-technical level and, despite the value it has on the executive floors, it is very expensive for what it provides”. Which is how a pilot “frequently fails by succeeding”. Everybody is pleased, and nobody can get the cost per answered question anywhere defensible.

Public-facing versions last longer, because they get measured against the price of outsourced support rather than against a licence. A longer runway, which I glossed at the time as a bigger budget.

None of that makes the capability worthless. For a small population — senior, non-technical, asking a narrow question against a well-governed measure — a plain-English box can genuinely remove a wait of days. That is worth buying. It is worth buying for eleven people rather than for four thousand, which is a very different line item and a much harder one to get signed.

And now the awkward part, which belongs in this section rather than at the end. The source I just used for the flat adoption record does not agree with me about what the record means. Its whole thesis is that plain-English querying and embedded analytics are precisely what finally breaks the ceiling, with a named prediction of more than half the workforce inside three years [12]. I cited the stagnation and walked past the explanation offered for why it was about to end. That is not a defect in the figure. It is a defect in my use of the figure, and it is the second time in this piece that somebody I was quoting had already thought about my argument; the first was the Microsoft paper, whose own account of years of argument answered the stronger sentence I was tempted to write.

Point the scrutiny at the line

Nothing gets governed evenly, so the practical question is where the scrutiny goes. The obvious answer is the reports people get paid on: tie the metrics to the people whose performance they measure, watch adoption climb, and call those the reports that matter. I turn that test down. My two instead:

If the dashboard's CDEs are company KPIs that are Reported, it is load-bearing. If dollars flow in or out of the company based on what is displayed on the screen, it is load-bearing. Surprisingly, it is not "if people get paid based on those numbers, it is load-bearing."

CDEs are critical data elements — the handful of fields the whole thing turns on. KPIs are key performance indicators, the measures an organization formally tracks to judge itself against its own goals. And Reported, capitalized, is carrying weight. It means the number leaves the building. A filing. A board pack. Not merely a screen somebody opens.

Two tests, then, and one very plausible test thrown out on purpose. Sit with the exclusion, because it looks like it belongs. Plenty of numbers have bonuses hanging off them — ticket clearance rates, on-time deliveries, satisfaction surveys, work and vacation hours. They are monitored, they are incentivized, people care about them intensely, and they still are not load-bearing. Compensation pressure and structural importance are different properties that happen to sit next to each other, and only the second one is worth a fence.

Push both tests hard and the landing is uncomfortable. The reports that drive or support sales are very nearly the only load-bearing reports in the building. A company lives on revenue — the top line of the income statement, the money coming in before any costs come out, as against the bottom line left after they do [14]. Anything that keeps that flowing is structural. Everything outside it is, at best, useful.

Other things can be levered into importance for a while. An ESG score. A supplemental stock offering. A line of credit, a bond issuance. While they are levered in they genuinely are load-bearing, and a fence around them is money well spent. But they are temporary and the line is not. I put it bluntly at the time: “the core of the business is always, always, always the line, and businesses that forget that almost always learn to regret it”.

Read that as a scheduling rule rather than as a hierarchy of worth. The fence around a levered-in number has to be as temporary as the leverage — built quickly, resourced honestly, taken down when the offering closes or the covenant lapses. The sales line never lapses.

I know the seam in that argument and I am leaving it visible, because I have not found the sentence that closes it honestly. Payroll and procurement move dollars in and out of a company on the strength of what is displayed on a screen, which satisfies the second test on its face, and neither one is the sales line. The test is broader than the conclusion I drew from it. A sharp line with a small seam still beats a hedged line with none.

What a fence is, in practice, is unglamorous. A named owner for each critical field. A definition written where a reader can find it without asking anybody. A test that fails loudly when the number moves further than the business it describes could have moved. And a rule about who may change the definition, which is the one everybody skips, because the day somebody quietly redefines an active account is the day two teams start reporting different numbers under the same name, and nobody notices until they compare them.

The semantic layer is a translation, not an honesty layer

A model pointed at a raw warehouse guesses at what a business means by its own words. A model pointed at a governed set of definitions inherits an agreed vocabulary instead — measures defined once, in one place, so that a word means the same thing whether it arrives through a report, an API call, or a sentence [6]. That is the standard account of why the semantic layer is what keeps a language model honest.

It is worth building. Whether it keeps a language model honest is another matter, and my answer to that was blunt:

Semantic layers are anything but honest. They exist to present carefully curated stories as facts. At its heart a semantic layer is nothing but a set of aliases and abstractions constructed as an overlay on top of an existing data model, designed to reframe and recharacterize the data into the format preferred by the consumer.

That is not a fringe reading and it is not a complaint about the people who build these things. It is what the category was designed to do, on the record from the beginning. The 1991 patent behind it describes showing users "terms that he is familiar with in his daily business" rather than "data organized in a computer-oriented way" [15]. Translation is the feature. It was never smuggled in.

The vendors are candid about the second half of it too. Training material for the tool that started the category says the universe exists “to hide the underlying physical data storage from the business user”, and that the SQL it generates runs “invisible to the business user” [16]. Hidden and invisible are the vendor's own words, offered as features, which is exactly what they are.

And the thing has since been put to a test. A 2023 workshop paper ran the same handful of queries through five production tools and found two of them choosing the join path, and the three that handled the hardest query applying deduplication rules, by heuristics they do not disclose, concluding that they “hide from the analyst the ability to interpret and control how the final metrics in a query are decided upon” [18]. Point at a different source table and a different answer comes back. Nothing in the interface reports that a choice was made.

The strongest evidence on the other side is a vendor's benchmark, and it deserves a close reading. dbt Labs' 2026 update ran the same questions through its semantic layer and through direct text-to-SQL on modeled data: accuracy rose from 90.0 to 98.2 per cent for one model and from 84.1 to 100 per cent for another, and questions inside the layer's scope were answered correctly every time [25]. Its authors put the difference in failure modes plainly — the layer “tells you it can't answer. It never returns invalid data. Text-to-SQL will cheerfully give you a wrong number” [25]. I accept every figure in that. What the benchmark scores is agreement with the definitions the layer already holds, which is the property I would call consistency. It cannot score whether a definition is the one the next department uses, because a test built on one layer has only one set of definitions in it. A model that agrees perfectly with a curated story is a faithful reader of the story.

So open the definition before trusting the number — not the dashboard, the definition, in whatever file the layer is built from — and read what the measure actually filters on. It needs nobody's permission, and it is the cheapest check available anywhere in this discipline. Often it confirms what was expected. When it does not, it can explain a disagreement two departments have been having politely and unproductively, in one line that neither of them had been shown.

Where the fault lies is narrower than that answer makes it sound. Whether one of these helps or hurts, I said, comes down to how well it was built and how much transformation it was asked to absorb, “assuming they are implemented properly according to the documented business rules, which is another source of fun”. That clause carries a great deal. It assumes the business rules were written down, that they were written down correctly, and that the person building the overlay read the document rather than the ticket. Renaming a column survives all three. Several rounds of normalization and denormalization run through a cube do not.

I put the failure mode in one line: “They only really cause problems when they provide conflicting information to downstream consumers who compare notes”. Which is to say the defect stays invisible right up until two people who trust the same system sit in the same meeting. And it is not rare, because “semantic layers will often provide different answers to different groups based upon their reporting and categorization requirements” — each group's requirements baked into what that group is shown, all of it lawful, none of it flagged.

And there's no version of this that a stricter build removes, which is the part I'd want a buyer to hear before signing anything. The defect isn't in the layer. It's in the belief that a shared word implies a shared measure — and the layer's whole job, done well, is to make that belief comfortable.

Airbnb published the bill for this. Before the company centralized its definitions, its chief executive would ask which city had the most bookings last week, and “Data Science and Finance would sometimes provide diverging answers using slightly different tables, metric definitions, and business logic” [17]. A first-rate data organization, two lawful translations, two numbers. Not incompetence, and not a governance failure either.

So build one, and simply do not file it under truth. It works like a translator at a negotiation — which is to say it is indispensable, it is faithful, and it is still making a hundred small choices a sentence that neither party in the room is able to audit. Point a language model at that and the fluency of the answer rests on decisions nobody in the conversation can see. None of which is an accusation: a translator who refused to choose would produce nothing at all, and the choosing is the job.

Governing a tool that mostly ratifies

The discipline for this already exists and it is vendor-neutral. The NIST AI Risk Management Framework organizes the work into four functions [5]. Re-aim each one at the premise above and they come out different from the way they are usually read.

Govern: not only who owns the model, but whose judgement it will end up supporting. Settle what may run unsupervised before somebody discovers by accident that it already does. The gate I would use for that is the cost to undo. Employees earn authority as their judgement grows; agents do not grow judgement, so their authority has to rest on something else, and reversibility is the obvious candidate. Hand an agent the decisions that are cheap to reverse, however large they are, and the work changes from watching every call to handling the errors.

Map: what does confidently wrong cost here? The load-bearing tests are the map. An assistant sitting on the sales line and a tool that summarizes ticket queues are not the same risk and should not get the same fence.

Measure: calibration, not merely accuracy. Then the awkward one. How often does a machine-generated finding change a decision, as against supporting one that was already forming? Almost nobody keeps that ratio. It is the most honest number available in this whole category, and it costs nothing beyond the discipline of writing down the intended action before the finding lands.

Manage: put the scrutiny where the dollars flow, and hold a plan for the day the thing is wrong in public.

Underneath all four, one conviction does most of the work. An answer nobody can interrogate is a rumour with a chart attached. Lineage and definitions travel with the finding. Confidence gets shown rather than implied. Under a ratification model that matters more rather than less, because the dangerous output was never a wrong number. It is a wrong number arriving at the exact moment somebody needed one.

The four functions do not tell an organization where to put its attention, and that is not a criticism of them — a framework that named your critical reports would be guessing. But it does mean the framework arrives with a hole exactly the shape of the previous section, and the load-bearing tests are what I would use to fill it. Govern and Manage are cheap once somebody has decided which twenty screens matter. They are unaffordable if the answer is all of them, which is the answer an organization gives by default, right up until the budget makes it stop.

Which is why I'd put the measurement burden on the finding rather than on the model. A model evaluation reports how often the machine is right in the abstract, which is worth knowing and isn't the question. What a business needs is narrower and far cheaper to collect, and I set it out at the end of this piece.

What I would watch for

For this section the warrant changes: what follows rests on my own answers when asked, and on the reasoning that follows from them, rather than on the sources above, and it should be read as judgement, not law.

The failure mode I would watch hardest

The pilot that succeeds. Adoption up, query volume up, survey scores warm, sponsor delighted, and an audit of a quarter's decisions turns up not one that came out differently. That is a perfectly fine outcome that will be written up as a transformation, and the gap between those two stories is where next year's budget gets set. Ask for the counterfactual while it is still cheap to ask — which means before the thing ships, rather than after it is loved.

The trade-off I would expect to bite

The AI work with the cleanest business case is, by the argument above, priced below the decision-consideration threshold. If the threshold at which something gets governed sits higher than the one at which it gets bought, and nothing above says it does not, then the wins will accumulate where nobody is watching: small, discretionary, department-funded, individually harmless, and collectively an unmapped analytics estate with a budget code. Make the safeguard an inventory rather than an approval gate. A gate gets routed around by the same logic that created the purchase, and it is not consulted until an argument starts, by which time the estate has already been built.

As agents spread, I would expect three failures, and I said so when asked, as expectation rather than record. Reports drift toward what their users want to see. Token costs climb with every push to have everyone use AI more. And agents outlive the person they were built for, still running their analysis on schedule and producing reports nobody reads. None of the three is new, because reporting systems have done all of them before. The inventory is the only control that finds the third.

The claim I would be sceptical of

That a semantic layer keeps the model honest. It keeps the model consistent, which is a different property, and not even guaranteed across an organization. Two numbers for the same measure can both be correctly derived. Related, and cheaper to check: treat "AI-driven decision-making" as a category label rather than as a description of anything.

The story I cannot tell you

Asked for the scar — the time a fast, confident, wrong answer got acted on and cost somebody something — I declined the question before answering it:

The way you are asking that question prompts me to answer: every time. More to the point, fast, fluent and confident answers that are unsupported by data or experience are frequently wrong and, all too often, acted upon.

The broader claim is the one worth keeping. Guarding against that kind of answer is, as I see it, the core of what business intelligence, data warehousing and master data management claim to be worth.

The research supports the shape of it. People reporting complete certainty turn out to be right something like seventy-two to eighty-three per cent of the time, and at ninety per cent confidence about seventy-five [13]. Confidence is not evidence of accuracy. It is evidence of confidence, and a language model produces it by default.

The anecdote itself is not mine to give, for a structural reason:

From the perspective of my career, almost all of those stories would sadly be from before my time — I am usually brought in to correct one of those decisions, and the individuals in question have departed prior to my arrival.

The correction is what is left of them. It also means that what I know about how a bad decision was reached, I mostly know from what it left behind, which matters again before this piece is done.

Nor do I have an account of where an inclination comes from. I cannot say how a position forms. Only what a number does alongside one that is already there. That is a real gap in this piece and I would rather name it than fill it with a framework.

The last thing that says I am wrong

I had a narrower claim I could defend and a field guide I stood behind. Then I went back and read the discipline I had spent this piece arguing with, properly, rather than through a press release.

Kozyrkov's decision intelligence asks the decider, before any data is seen, to state how the decision would be made with no additional information at all. What the default choice is. Then it asks for the metric and the cut-off, in advance — does that number need to be 4.2 or 4.5, and settle it now [2]. Her stated motive is close kin to my diagnosis. She had often seen executives make decisions steered by unconscious bias rather than by the data, and she says that people who never make their concepts concrete think that putting a bunch of mathematics near them fixes it.

So the thing I found in the experimentation record and presented as a correction — procedure rather than magnitude, the rule fixed before the number arrives — is not a correction to the discipline. It is the discipline. It has been step two the whole time. I spent this article arguing with a field about where the bottleneck sits, and the field's own founding account puts the bottleneck where I put it and prescribes what I prescribed.

It is also where I would have started. Asked how I would begin a decision-intelligence effort, I said I would name the success and the failure metrics before tracking anything, and build outward according to how close each piece of work sits to them — the decision treated the way test-driven design treats code, output first, then back to how the output was produced. That is step two, with a failure measure attached, which is the half most programs leave out.

Which breaks something, and I want to be exact about what. It does not break the decision research. Feldman and March still stand. Klein still stands. It does not break my own answer about what numbers do, and it does not break the load-bearing tests or anything else in the field guide above.

What it breaks is my quarrel with the discipline's method. The method is right. I had been arguing against a purchase and calling it a field.

What survives is a quarrel about standing, and it is older than this piece. Much of what gets sold as a modern discipline of deciding restates, usually without credit, what the study of human judgement worked out long before it had the name. Kozyrkov's motive echoes Feldman and March's diagnosis of 1981 [7], and the rule I found in the Microsoft record — a measure named before the result exists — is her step two, in print in the experimentation literature nine years before the Fast Company account of her framework [19] [2]. Lorien Pratt, writing in 2019 after ten years of evangelising the field, is candid about where the name came from. Her team first called it decision engineering and “learned along the way that this name just wouldn't sell,” and she concedes that “Decision Intelligence means different things to different people” [24]. None of that makes the method wrong. It does mean the label is not evidence of anything. I would treat any implementation carrying it as a rebranding of the business-intelligence team until it produced a significant benefit, and then ask what the same result would have cost under a plainer method with less overhead. That is a prior I hold, not a finding, and the second question is much the harder one to clear. It is also why, when I described where I would start, I added that the field's leaders embrace the whole paradigm and I am still not convinced it produces value a company could not get another way.

What it leaves me unable to settle is what a screen shipped without step two is evidence of. Either the discipline is wrong about decisions, which is what I have been saying, or the discipline is right and a dashboard released with no default stated, no cut-off agreed, and nobody having said in advance what the decision would do differently depending on the number is one instance of it built without the step that makes it work. Those two readings predict the same thing: a well-used screen, a decision that goes the way it was already leaning, and an audit afterwards that finds nothing the screen changed. From where I usually stand, which is after the decision and after the people who made it have gone, I cannot separate them.

And I have no way to run it again. To tell those readings apart I would need the same organization and the same decision, once with a rule fixed in advance and once without, and nobody has ever handed me that and nobody is going to.

I could look for the counterfactual in somebody else's data, and I have. What comes back is a controlled experiment, where the rule was fixed in advance by construction; or a population reversal like the prescribing data above, which has the rule missing but no comparison group and a six-year window; or a survey of adoption, which measures whether a screen was opened rather than whether a decision moved. There's plenty of all three, and not one of them is the same organization twice. The thing I want — the same decision, twice, once under a rule and once without — isn't a study anybody has run, and I'm no longer confident it's a study anybody could run.

What would change my mind is smaller than a study and somebody could do it this quarter. Take one recurring decision with a screen behind it. Before the number is released, have the owner write down what they would do without it. Then release the number and record what they did. Fifty of those, in one organization, would tell me more than everything I have cited here — not because it is rigorous, but because it is the one measurement the ratification reading and the badly-built-instance reading actually disagree about. Nobody has run it that I know of, and the obstacle is not cost. It is that the first field asks somebody to commit, in writing, before they are allowed to look, which is precisely the discomfort the whole argument is about.

I am not going to resolve this by choosing the reading that keeps my argument. I do not know which one is true. What I do know is that I had been telling a story about a field getting something wrong, and the field had written down the fix before I arrived, and I did not check.

What I cannot tell you

None of this is an argument against the technology. Used honestly, this work is remarkable. It will chase down a problem that was never worth a project. It will free good people from assembling the obvious. Take all of that. It is real, it is available now, and it is cheaper than it has any right to be.

Measure it by decisions changed rather than by screens shipped, and expect the honest count to be low. Write down what the room agreed to do about the number before the room was allowed to see it. That much I still hold, and everything in the field guide above rests on it rather than on the reading I have just lost.

Here is what I cannot tell you, and I would rather leave it open than close it with a sentence I would have to defend later. I do not know whether the screens nobody decides from are evidence about decision intelligence, or evidence about badly built instances of it. I do not know whether a number arriving with no rule attached does nothing at all, or does something slowly and unevenly, over a stretch long enough that nobody in the meeting connects the two. The literature I leaned on here spans forty-five years, and none of it measured that.

The honest shape of what is left is three things, and they don't resolve into one. A claim I still hold, in the third and narrowest form it took here — that inside the range of evidence a business produces, a rule fixed before the number arrives is what makes the number do any work at all, and that the two things which get a number out of that range are both outside a business's reach. A question about the screens that I cannot answer from where I stand. And a field that had written my answer down before I started arguing with it, in a step I had read past twice.

What I notice is that the fix is not evidence. It is a few minutes of somebody writing down what they will do before they are allowed to look, it produces nothing anybody would put on a dashboard, and no quantity of better numbers afterwards substitutes for it. I believed that about decisions when I started writing this, and I still think it is right. What moved is where I think the argument lives: not with the discipline, which prescribes the step, but with the purchase, which skips it.

Where to Go Deeper

On how experts actually decide, read Gary Klein, Sources of Power, and the naturalistic decision-making tradition around it [8]. On why organizations demand information they do not use, Feldman and March's 1981 paper is short, forty-five years old, and still the sharpest thing written on it [7]. On whether numeracy protects a reasoner, Kahan and colleagues [9], with Wood and Porter as the honest counterweight [10] and Webb and Sheeran [11] on why intention and behaviour part company. Kozyrkov [2] is the least hype-prone way into the discipline itself, and I would now read her before reading me; Pratt [24] is the frankest account of how it got its name. Kohavi and colleagues [19] is the record that broke my own line, and the best public account of an organization deciding on numbers as a matter of routine — read sections 6.2 and 6.3 rather than the abstract, because the cultural fight is the part everybody skips. On what a number can do with no rule anywhere near the people acting on it, the post-WHI prescribing literature is the cleanest case I found, and the NIH release announcing the stop [20] is worth reading beside the prescribing data [21][22] for how much has to be true at once before a number does that. Huang, Damalapati and Wu [18] is the paper to read before trusting a semantic layer, with the dbt benchmark [25] as the strongest case against my reading of it, and Airbnb's Minerva write-up [17] is the best public account of a company paying to fix one. For governance, go straight to the NIST framework [5] rather than to a vendor's reading of it.

An earlier version of this piece — written before this publication rebuilt how its voices are generated — is published unedited at The Number Was Never Doing the Work: AI, BI, and the Limits of Better Evidence (original), so the two can be read side by side. The account of why is at The Voice Problem.

References

[1] Gartner, "Gartner Identifies Top 10 Data and Analytics Technology Trends for 2020," press release, June 22, 2020 — Trend 3, Decision Intelligence, quoted verbatim. https://www.gartner.com/en/newsroom/press-releases/2020-06-22-gartner-identifies-top-10-data-and-analytics-technolo (Gartner has since withdrawn its public IT glossary, where this definition also appeared; the press release carries it and is live.)

[2] Fast Company, "Why Google defined a new discipline to help humans make decisions" (on Cassie Kozyrkov and decision intelligence). https://www.fastcompany.com/90203073/why-google-defined-a-new-discipline-to-help-humans-make-decisions

[3] Gartner, "Gartner Identifies Top 10 Data and Analytics Technology Trends for 2020," press release, June 22, 2020 — Trend 2, Decline of the Dashboard, on augmented analytics and streamed insight; same release as #ref-1, different claim. https://www.gartner.com/en/newsroom/press-releases/2020-06-22-gartner-identifies-top-10-data-and-analytics-technolo (Gartner's separate "Augmented Analytics" glossary entry, cited in earlier drafts, has been withdrawn along with the rest of the public IT glossary.)

[4] Gartner, "Gartner Predicts 75% of Analytics Content Will Use GenAI for Enhanced Contextual Intelligence by 2027," press release, June 18, 2025. https://www.gartner.com/en/newsroom/press-releases/2025-06-18-gartner-predicts-75-percent-of-analytics-content-to-use-genai-for-enhanced-contextual-intelligence-by-2027

[5] National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1, January 2023 — core functions: Govern, Map, Measure, Manage. https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf

[6] AtScale, "What Is a Semantic Layer? Definition, Benefits, Types & More," AtScale Glossary. https://www.atscale.com/glossary/semantic-layer/ (Vendor source — AtScale sells a semantic layer. Cited for the definition of the category, not as evidence of its value.)

[7] Feldman, Martha S., and James G. March, "Information in Organizations as Signal and Symbol," Administrative Science Quarterly 26, no. 2 (1981): 171–186.

[8] Klein, Gary, Sources of Power: How People Make Decisions (Cambridge, MA: MIT Press, 1998) — the recognition-primed decision model.

[9] Kahan, Dan M., Ellen Peters, Erica Cantrell Dawson, and Paul Slovic, "Motivated Numeracy and Enlightened Self-Government," Behavioural Public Policy 1, no. 1 (2017): 54–86.

[10] Wood, Thomas, and Ethan Porter, "The Elusive Backfire Effect: Mass Attitudes' Steadfast Factual Adherence," Political Behavior 41 (2019): 135–163.

[11] Webb, Thomas L., and Paschal Sheeran, "Does Changing Behavioral Intentions Engender Behavior Change? A Meta-Analysis of the Experimental Evidence," Psychological Bulletin 132, no. 2 (2006): 249–268. DOI 10.1037/0033-2909.132.2.249 — a medium-to-large change in intention (d = 0.66) produced only a small-to-medium change in behaviour (d = 0.36). See also the authors' later review, Sheeran and Webb, "The Intention–Behavior Gap," Social and Personality Psychology Compass 10, no. 9 (2016): 503–518, DOI 10.1111/spc3.12265, which summarizes it.

[12] Business-intelligence adoption has held at roughly a quarter to a third of employees and stayed broadly flat despite sustained tooling investment. The lower figure is BARC and Eckerson Group (2022), which put penetration at 25%; the upper is Gartner (2019), which put it at 35%. Both reported via Eric Avidon, "BI adoption poised to break through barrier — finally," TechTarget, February 1, 2023. https://www.techtarget.com/data-technologies/news/365530077/BI-adoption-poised-to-break-through-barrier-finally Gartner's "Survey Analysis: Why BI and Analytics Adoption Remains Low and How to Expand Its Reach" (doc 3753469) is the paywalled primary and was not read directly.

[13] Lichtenstein, Sarah, Baruch Fischhoff, and Lawrence D. Phillips, "Calibration of Probabilities: The State of the Art to 1980," in Daniel Kahneman, Paul Slovic and Amos Tversky, eds., Judgment Under Uncertainty: Heuristics and Biases (Cambridge University Press, 1982), ch. 22, pp. 306–334. DOI 10.1017/CBO9780511809477.023. The authors report that only 72–83% of items answered with complete certainty were correct, and gloss the 0.90 band as "when they are 90% certain, they are only 75% right." They also stress that calibration depends on task difficulty, reversing to underconfidence on easy material.

[14] U.S. Securities and Exchange Commission, Office of Investor Education and Advocacy, "Beginners' Guide to Financial Statements," January 12, 2014 (last reviewed February 6, 2017) — "this top line is often referred to as gross revenues or sales… At the bottom of the stairs, after deducting all of the expenses, you learn how much the company actually earned or lost." https://www.sec.gov/about/reports-publications/beginners-guide-financial-statements For a fuller treatment, including the several names the top line goes by on real statements, see Julie Dahlquist and Rainford Knight, "5.1 The Income Statement," Principles of Finance (OpenStax, Rice University, March 24, 2022). https://openstax.org/books/principles-finance/pages/5-1-the-income-statement

[15] Liautaud, Bernard, and Jean-Michel Cambot (Business Objects S.A.), "Relational database access system using semantically dynamic objects," U.S. Patent 5,555,403, application 07/800,506 filed November 27, 1991, issued September 10, 1996 — the origin document for the business-semantic layer. https://patents.google.com/patent/US5555403A/en

[16] SAP, "Describing the SAP BusinessObjects BI Semantic Layer," Analytics with SAP Solutions (SAP Learning), page rendered and read August 16, 2026. https://learning.sap.com/courses/analytics-with-sap-solutions/describing-the-sap-businessobjects-bi-semantic-layer_a8ee309a-8118-466f-851c-ffd94530cb81 (Vendor documentation, cited for what the product does — not as evidence of its merits.)

[17] Chang, Robert, et al., "How Airbnb Achieved Metric Consistency at Scale (Part I: Introducing Minerva)," The Airbnb Tech Blog, April 30, 2021. https://medium.com/airbnb-engineering/how-airbnb-achieved-metric-consistency-at-scale-f23cc53dea70

[18] Huang, Zezhou, Pavan Kalyan Damalapati, and Eugene Wu, "Aggregation Consistency Errors in Semantic Layers and How to Avoid Them," in Proceedings of the Workshop on Human-In-the-Loop Data Analytics (HILDA '23) (ACM, 2023). arXiv:2307.00417. https://arxiv.org/abs/2307.00417

[19] Kohavi, Ron, Thomas Crook, Roger Longbotham, Brian Frasca, Randy Henne, Juan Lavista Ferres and Tamir Melamed, "Online Experimentation at Microsoft," Microsoft ThinkWeek paper, 2009. https://robotics.stanford.edu/~ronnyk/ExPThinkWeek2009Public.pdf (that host declined automated retrieval on 2026-09-04; the identical file served from the author's other Stanford path, https://ai.stanford.edu/~ronnyk/ExPThinkWeek2009Public.pdf, is what was read.) The one-third figures and the internal-resistance quotation are the authors' own, reporting on Microsoft's own experiment platform; read in full rather than from the abstract. Cited as the counter-example to the article's opening claim; sections 6.2 and 6.3 record the internal resistance the body describes. Liveness and the quoted passages (the one-third finding, the MSN home page experiment, the dismissal of the first statistics) re-checked 2026-09-26 through a rendering fetch of the ai.stanford.edu copy.

[20] National Institutes of Health, National Heart, Lung, and Blood Institute, "NHLBI Stops Trial of Estrogen Plus Progestin Due to Increased Breast Cancer Risk, Lack of Overall Benefit," news release, July 9–10, 2002; the release text was read end to end as carried by ScienceDaily, July 10, 2002. https://www.sciencedaily.com/releases/2002/07/020710081413.htm The trial-arm size, the 5.2-year average follow-up, the effect sizes, and the pre-specified monitoring boundary are all the sponsor's own statements. Garnet Anderson, the biostatistician who led the analysis, is quoted directly on the pre-specification: “We pre-specified that the change in cancer rates did not have to be that large to warrant stopping the trial.” The peer-reviewed principal results are Writing Group for the Women's Health Initiative Investigators, "Risks and Benefits of Estrogen Plus Progestin in Healthy Postmenopausal Women," JAMA 288, no. 3 (2002): 321–333, DOI 10.1001/jama.288.3.321 — cited to the published abstract for provenance, since the publisher's site answered with a bot check and the full text was not reached.

[21] Parente, Lynn, Catherine Uyehara, Wilma Larsen, Bradford Whitcomb and John Farley, "Long-term impact of the women's health initiative on HRT," Archives of Gynecology and Obstetrics 277, no. 3 (2008): 219–224. DOI 10.1007/s00404-007-0442-1. https://link.springer.com/article/10.1007/s00404-007-0442-1 Observational cohort of 71,592 hormone-therapy prescriptions dispensed at Tripler Army Medical Center, July 1999 to July 2005. Cited to the published abstract, read at the publisher 2026-09-04; the body is paywalled and was not read, so every figure quoted here is an abstract figure. The abstract states the 1,272-to-493 monthly fall across the study window, not across the post-July-2002 period alone, and separately reports that obstetrician-gynaecologists began decreasing their prescribing before July 2002 — which is why this article says part of the decline predates the result.

[22] Kim, Nancy, Cary Gross, Jeptha Curtis, Glen Stettin, Stephen Wogen, Nami Choe and Harlan M. Krumholz, "The impact of clinical trials on the use of hormone replacement therapy: a population-based study," Journal of General Internal Medicine 20, no. 11 (2005): 1026–1031. DOI 10.1111/j.1525-1497.2005.0221.x. https://link.springer.com/article/10.1111/j.1525-1497.2005.0221.x Roughly 340,000 women aged 55 and over in the Medco Health database, May 1998 to May 2003. Cited to the published abstract, read at the publisher 2026-09-04; the body is paywalled and was not read. The persistence-of-variation finding is the authors' own conclusion, quoted closely rather than inferred.

[23] Matthew Miller, "Tableau Metrics and Natural Language Query Evolve with Tableau Pulse," Tableau blog, February 21, 2024. https://www.tableau.com/blog/tableau-metrics-and-natural-language-query-evolve-tableau-pulse Read 2026-09-26 through a rendering fetch; extracts stored at documents/wi/redrafts/sources/AIBI_v2/Miller_2024_Tableau_Ask_Data_Retirement_VENDOR_CAPTURE.txt. Vendor source, cited for what the vendor did and said about its own product: Ask Data and Metrics no longer available from Tableau Cloud at the 2024.1 upgrade and from Tableau Server at 2024.2; customer feedback that it "isn't 'natural' enough"; and "the declining adoption rates for Ask Data on-premises made us confident that this is the correct plan for the future"; the post presents Tableau Pulse as the experience that "replaces Ask Data starting in Tableau 2024.1." A product withdrawal, not a study of pilot outcomes.

[24] Lorien Pratt, "Decision Intelligence has left the lab: lessons learned from 10 years of evangelism," lorienpratt.com, June 21, 2019. https://www.lorienpratt.com/decision-intelligence-has-left-the-lab-lessons-learned-from-10-years-of-evangelism/ Read 2026-09-26 through a rendering fetch; extracts stored at documents/wi/redrafts/sources/AIBI_v2/Pratt_2019_Decision_Intelligence_Has_Left_The_Lab_CAPTURE.txt. Self-published by one of the field's longest-standing evangelists; cited for her own account of the name (first "Decision Engineering," which "just wouldn't sell") and for "Decision Intelligence means different things to different people," not as evidence about outcomes.

[25] Jason Ganz and Benoit Perigaud, "Semantic Layer vs. Text-to-SQL: 2026 Benchmark Update," dbt Developer Blog, April 7, 2026. https://docs.getdbt.com/blog/semantic-layer-vs-text-to-sql-2026 Read 2026-09-26 through a rendering fetch; extracts stored at documents/wi/redrafts/sources/AIBI_v2/Ganz_Perigaud_2026_dbt_Semantic_Layer_vs_Text_to_SQL_VENDOR_CAPTURE.txt. Vendor benchmark: dbt Labs sells a semantic layer, and accuracy is scored against its own modeled definitions. Cited for the modeled-data accuracy figures (90.0% to 98.2% and 84.1% to 100.0%), 100.0% within the layer's scope, and the quoted sentences on failure modes.