Another Set of Eyes: How a Data Steward Should Work With What the AI Proposes

Weaving Intelligence (House AI Style), with Jeff Shabel’s Experiential Commentary

When the AI backstop in a master data pipeline has done its work, what it produces should land in one place: a data steward's queue, as a suggestion, with the evidence it used laid beside it, and nothing written to a master record until the steward has approved it. That is a narrower job for the model than most of what is sold as AI-augmented stewardship, and I hold to it for a reason that has less to do with the model than with the steward. Asked where the backstop's output should land in a steward's day, I answered:

AI supports the decision making process of the Data Steward and helps them to increase their throughput. The output of AI, as well as the context used by the AI to make the decision, should be able to be quickly reviewed by the Steward and approved.

There are two requirements in that answer, and the second is the one designs drop. The first is that the model supports the decision and the steward makes it: the output is reviewed and approved, not applied first and audited later. The second is that the steward sees the context the model used, and sees it quickly. A suggestion without its context leaves the steward two bad options. Redo the model's work from the start, and the throughput the model was meant to buy is gone. Take the suggestion on trust, and the review is gone. Everything in this piece is about keeping both.

Putty, Not Body Panels argued where the model belongs in a master data pipeline: not in the default path of every record, but beside it, as a backstop for the records standard processing cannot resolve, with its use measured, tracked and controlled. This piece starts where that one stopped, at the moment the backstop has looked at a record and a person has to decide, which is also the question Who Owns the Data, Really? set aside: what the steward should see, what the research says about a person reviewing a machine's suggestions all day, how to configure the queue, and how to introduce it.

The review queue is older than the model

The steward's queue was not invented for AI. Ivan Fellegi and Alan Sunter's 1969 theory of record linkage, the statistical basis of most probabilistic matching since, gives every compared pair one of three outcomes: a link, a non-link, or a possible link [1]. The third is the queue. Of the possible-link decision they wrote that it "will require expensive manual linkage operations" [1], and their optimal rule is defined by what it does to that region: for the error rates the user has chosen to tolerate, it "minimizes the probability of failing to make a positive disposition" [1]. Put plainly, it sends as few pairs to people as the tolerated errors allow, and no fewer. William Winkler's overview of the field for the US Census Bureau uses the region's working name, the "clerical review region" [2], and one of his examples shows what automation has done to it: in the 1992 Census of Agriculture, computerised matching cut the review from 75 clerks over three months to 6,500 hours [2]. The queue shrank. It did not go away.

Master data platforms carry the same split today, and since what follows assesses the vendor we work with, a disclosure first. The practice behind this publication is a Profisee partner. Clients pay us for deployment work, a licence sale we refer earns us a referral fee from the vendor, and our services revenue depends on master data management being bought at all. That interest runs toward praise of Profisee and of the category, so weigh anything favourable I say about either below what I say against them.

Profisee describes "an ML-powered matching engine to automatically match duplicate records", tells buyers to "keep a human in the loop for manual review of inconclusive results", and shows candidate matches "side-by-side in a match group for data stewards to review" [3]. Inconclusive results are Fellegi and Sunter's possible links under a newer name. A platform's own trained matcher sits outside the rule I hold about models, which concerns calls to a hosted model priced by the token, not the matching and scoring a platform already runs as part of its processing. It matters here for a different reason: the steward was already working a queue before any model arrived, and the AI backstop adds to that queue rather than creating it.

What it adds is different in kind. A possible link from the matcher arrives as two records and a score. An item from the backstop arrives as a record that standard processing could not handle, with a proposal attached: a value filled in, a category assigned, a field extracted from free text, a merge suggested on grounds the rules could not see. The proposal is the new thing. It is what makes the item faster to decide, and it is also what makes it easier to wave through.

There is a way to shorten the queue without any model at all, and a team under pressure is tempted by it: lower the auto-approve threshold and let the matcher decide more of the borderline pairs. I have said before what that shortcut costs. A threshold set too low does not take work away from the stewards; it guarantees them more of it later, unpicking false merges after the business has already decided and reported on the merged records. The length of the queue is set by the errors the organisation will tolerate, which is how Fellegi and Sunter framed it more than fifty years ago. What a model can change is the cost of each decision in it.

The decision budget

The rest of my answer gave the reason the cost of each decision matters:

The objective is to give the Data Steward the ability to filter their ‘decision budget’ to the records that matter rather than needing to manually review every element and make a decision. There’s some substantial psychological research on decision fatigue and AI helps Data Stewards avoid or minimize it.

The decision budget is my name for what a steward actually spends. A steward has a finite number of good decisions in a day. Spent on every element of every record, the budget runs out on the routine ones, and the records that genuinely need judgement arrive at the end of the queue, when there is least left to give them. That decline over a working session is what the field studies below measure; why it happens is a separate question. The point of putting a model in front of the steward is to change where the budget goes: let the model gather the evidence and turn the routine call into a confirmation, so that the steward's real attention goes to the records that matter.

I called the research on decision fatigue substantial, and the field studies usually cited under that name are. The best known followed experienced parole judges through their working days and found that "the percentage of favorable rulings drops gradually from ≈65% to nearly zero within each decision session" and "returns abruptly to ≈65% after a break" [4]. In primary care, a study of 21,867 visits for acute respiratory infection found that antibiotic prescribing "increased throughout the morning and afternoon clinic sessions", with adjusted odds in the fourth hour of a session 1.26 times those in the first, and its authors called that "consistent with the hypothesis that decision fatigue progressively impairs clinicians' ability to resist ordering inappropriate treatments" [7]. Among securities analysts, "forecast accuracy declines over the course of a day as the number of forecasts the analyst has already issued increases", and the analysts lean more on heuristics such as herding toward the consensus [8]. That study is the closest to a steward's day, because it counts decisions rather than hours.

What I did not say, and should have, is that the mechanism usually offered for those patterns is in serious dispute. The parole study was challenged within months by critics who argued that prisoners without legal representation were heard at the end of each session; the authors re-ran their analysis with representation included and replied that "case order and the meal break remain robust predictors of the parole decision" [5]. Andreas Glöckner later simulated the data and concluded, in his title, that "the magnitude of the effect is overestimated": because favourable rulings take longer, a judge who simply avoids starting a long case shortly before a break would produce much of the same pattern without any fatigue at all [6]. The original authors had been candid from the start: "we do not have a direct measure of the judges' mental resources" [4].

Decision fatigue is usually explained as ego depletion, the claim that self-control and choice draw on "a common resource" that acts of will use up [9], and two large preregistered replications have since failed to find it: 23 laboratories and 2,141 participants produced d = 0.04 [10], and 36 laboratories and 3,531 participants a nonsignificant d = 0.06, with only an exploratory d = 0.08 found after the fact [11]. The decline in field decisions over a session is reasonably well documented; the claim that it comes from a mental resource that decisions drain is not settled, and I would not rest a stewardship design on it.

The design does not need it. The case for spending a steward's decisions carefully survives without the depletion theory, for two reasons. A steward's hours are finite whatever the psychology, and every element reviewed by hand is time not spent on a record that needs judgement. And the second reason is better evidenced than depletion and bears directly on AI. Kate Goddard, Abdul Roudsari and Jeremy Wyatt's systematic review of automation bias, the tendency to accept a computer's advice with too little scrutiny, retrieved 13,821 papers and kept 74. Among the conditions that made people likelier to defer to the machine, it lists "workload, task complexity, and time constraint, which pressurized cognitive resources" [12]. An overloaded queue does not only tire a steward. It makes the steward likelier to wave through exactly the items the model proposed. A decision budget, filtered so that each item gets the time it needs, is a defence against over-reliance as well as a way to raise throughput, and that is the stronger reason to keep one.

What the steward sees

If the steward is to approve the model's output, the queue item has to make approval a real decision that can be made quickly, and the research on that cuts against the obvious design.

The obvious design shows the model's reasoning: a fluent paragraph explaining why the merge is right. Gagan Bansal and colleagues tested explanations of that kind with people making decisions alongside an AI. The pair did better than either alone, but the explanations did not add to the gain; instead they "increased the chance that humans will accept the AI's recommendation, regardless of its correctness" [13]. A persuasive rationale is a sales document for the suggestion. Helena Vasconcelos and colleagues found the condition under which explanations reduce over-reliance rather than feed it. People weigh the cost of checking the AI against the cost of trusting it, and earlier studies that found no benefit "could be due in part to the explanation not sufficiently reducing the costs of verifying the AI's prediction" [14]. The context a steward needs is not the model's argument. It is the evidence, laid out so that checking it is cheaper than trusting it.

That is what "quickly reviewed" has to mean in configuration. In a queue item for a master data steward, I would put five things on one screen.

• The proposal, as the change it would make. The field, its current value and the proposed one; or the records to be merged and the values that would survive.
• The evidence the model was given, not a summary of it. The source records side by side, the free text a value was extracted from with the extracted span marked, the reference list a category was taken from. If the model was handed only the raw record, the steward should see that too. A model knows nothing outside what it was given, and it will classify whatever it is asked to classify with the same confidence; what it was not given is often where the error lies.
• Why the record was routed here. The named check in standard processing that failed and sent the record to the backstop: the entry rule Putty, Not Body Panels argued for. Showing it lets the steward see when the rules, not the record, are the problem.
• What else changes if it is approved. The downstream systems that consume the field, or the child records a merge would move.
• How sure the model was, but only if that has been tested. A model's confidence can be surfaced, thresholded and used to send the weak cases to a person, which was the analysis in Putty, Not Body Panels. A confidence figure that has not been checked against the organisation's own records is one more persuasive number. Goddard's review lists updated confidence levels attached to a system's output among the design factors that mitigated automation bias [12]; the warning about a figure nobody has tested is my own reading, not the review's.

John Lee and Katrina See's review of trust in automation supplies the test for all five. The aim is calibration, "the correspondence between a person's trust in the automation and the automation's capabilities" [15], not more trust. A queue item that shows the evidence lets the steward's trust follow the evidence. A queue item that shows only the answer asks for trust that follows nothing.

The decision controls matter as much as the display. In many businesses stewards carry duties beyond stewardship, so the work has to fit into the day: they step into the queue, make the call there and then, and go back to what they were doing. Approve, edit and reject should each be one action, inline, wherever the steward meets the item. Each decision should be recorded with the evidence the steward saw and, on an edit or a rejection, a short reason chosen from a list. That record is the audit trail, and it is also the most valuable data the queue produces: Sunita Sarawagi and Anuradha Bhamidipaty showed in 2002 that a deduplication system which brings the hard pairs to a person and learns from the answers "significantly reduces the number of instances needed to achieve high accuracy" [17], and a steward's edits and rejections are exactly those hard pairs. David Madras, Toniann Pitassi and Richard Zemel go further and train a model to "Pass" a case to the person downstream, arguing that "learning to defer can make systems not only more accurate but also less biased" [18]. Their setting is a single trained classifier, aimed at fairness as well as accuracy, not a rules engine with a backstop, but the principle carries over: sending an item to the steward is a designed output, not a failure.

One configuration decision overrides the rest. The approval has to be enforced by the system, not requested of the user. If there is any path by which a model's proposal can reach a master record without a steward's approval, the approval is advice, and Every Generation Moved the Work has already shown what a gate that is not enforced turns into. The model's output is a suggestion. The approval is not.

That is where the vendor we partner with deserves a careful reading, in both directions. Profisee's page for Aisey, its AI assistant, says that "Stewards use Aisey to review match proposals, apply survivorship rules and interrogate records for violations", and in the same sentence offers to "let those tasks run autonomously" [16]. For administrators it automates routine tasks such as attribute mapping and file imports, "giving admins full control to review and approve" [16], and the page says that "every change is reviewable" [16]. It also describes agents that "monitor your environment, detect conditions, and act automatically", and lists among its templates one named "Edit or Update Attributes Automatically" [16]. Both modes are offered to stewards, so the choice is the implementer's; the page does not say how that choice is set, or whether an agent's edits to attribute values can be held for a steward's approval before they are written. For changes a model proposes to master values, the route I would choose is approval before writing, every time. A change that is reviewable after it is written can be audited; a change approved before it is written was decided by a person. I have not tested either mode, but I look forward to doing so soon.

Throughput, and the rubber stamp

The objection is plain: if every suggestion waits for a person, where does the throughput come from? It has evidence behind it, and it deserves an answer from the evidence.

The throughput can be real. Erik Brynjolfsson, Danielle Li and Lindsey Raymond studied the staggered introduction of "a generative AI-based conversational assistant" to 5,179 customer-support agents, people who still did the job with the tool beside them. Issues resolved per hour rose "by 14% on average", including "a 34% improvement for novice and low-skilled workers", with "minimal impact on experienced and highly skilled workers" [19]. Customer support is not master data, but the pattern is worth looking for in a stewardship queue's own numbers: a model may raise the floor of a queue worked by newer stewards more than the ceiling of one worked by experts.

The same body of research shows the cost. Fabrizio Dell'Acqua and colleagues, in a field experiment with 758 knowledge workers run with Boston Consulting Group, gave people tasks inside and outside what the model did well. Inside, those using AI completed 12.2% more tasks and finished 25.1% more quickly; on a task outside it, they were about 19 percentage points less likely to produce a correct solution than those without AI [20]. Michelle Vaccaro, Abdullah Almaatouq and Thomas Malone's meta-analysis of more than a hundred experiments goes further: "on average, human-AI combinations performed significantly worse than the best of humans or AI alone", with "performance losses in tasks that involved making decisions" [21]. A steward approving a merge is making a decision. On the average result, the combination I am recommending is the one that loses.

The average hides the condition that decides the question, and the meta-analysis states it: "when humans outperformed AI alone, we found performance gains in the combination, but when AI outperformed humans alone, we found losses" [21]. So the case for steward approval turns on which side of that line the records fall. My own record puts them on the human side. I have yet to see an AI system that reviews and corrects master data as well as a steward who knows the domain, at a cost an organisation would accept, which is why I treat the steward's review as core to mastering, as Proving the Platform Is Delivering argued. That is experience, not a measurement, and it carries its cost condition. But it places stewardship where the meta-analysis finds that the pair beats either party alone.

It also rules something out, and I would rather say so than leave it implied. If an organisation measures on its own records that the model is right more often than its stewards on some class of records, then on that class the approval step costs accuracy, and this design has no accuracy defence for keeping it there. The answer is still not to let the model write unattended. It is to take what the model reliably gets right on that class and turn it into a rule the standard flow applies, tested like any other code, which takes the model out of those records altogether and stops paying for it on them. Where no rule can capture what the model gets right on a class, the approval stays, and the accuracy it costs there is the price of keeping the decision with a person. Taking models out of the pipeline wherever possible was the argument of the earlier piece; the steward's queue is where the evidence for doing it collects.

The other danger is the rubber stamp, and expertise does not cure it. Raja Parasuraman and Dietrich Manzey's review found automation bias "in both naive and expert participants", and found that it "cannot be prevented by training or instructions" [22]. Designs that force a person to engage before accepting do help. Zana Buçinca, Maja Malaya and Krzysztof Gajos found that such interventions "significantly reduced overreliance compared to the simple explainable AI approaches", and also that "people assigned the least favorable subjective ratings to the designs that reduced the overreliance the most" [23]. Friction works, and stewards will dislike it, which is a reason to spend it where the stakes are, on the merges that feed finance and regulatory reporting rather than on every address correction.

So the queue needs measures of its own, kept as a series and read the way any other flow is read. How many items arrive, and by which routing reason. How many are approved, edited and rejected. How long each takes. And, most important, how many approved suggestions are later reversed. A falling rejection rate on its own tells you nothing: it is what improving suggestions look like, and it is also what a tired steward looks like. A reversal rate that climbs while rejections fall is the rubber stamp showing up in the numbers.

Another set of eyes

Up to here the weight has rested on published evidence; from here, on introducing the queue to stewards, it rests on my own position and on what I know of how others went about it, not on a rollout I ran. Asked in a follow-up how the AI had been introduced to the stewards and what had told me it was safe to give it more, I answered:

I didn't do the introduction of AI to the stewards, other early adopters did and they tended to sell it as a panacea--an approach I strongly reject. Were I to do it, I would characterize it as another set of eyes and a suggestion, to be taken or left based upon their best judgement.

So what follows is a prescription, not a report: what I would do, and why the evidence supports it.

The panacea pitch fails on its own terms, and the research explains why. Berkeley Dietvorst, Joseph Simmons and Cade Massey found that "people more quickly lose confidence in algorithmic than human forecasters after seeing them make the same mistake", and that this held "even among those who saw the algorithm outperform the human" [24]. A tool introduced as the answer to everything will make its first visible mistake soon enough, and the stewards will mark it down harder than a colleague who made the same error. In Lee and See's terms, overtrust, where "trust exceeds system capabilities" [15], invites misuse, and the correction overshoots into disuse: an oversold model gets rubber-stamped until it is caught out, and then it gets ignored.

"Another set of eyes and a suggestion" is the framing that avoids both, and on the evidence it is also the framing that gets an imperfect tool used. In a follow-up study the same three authors found that people were "considerably more likely to choose to use an algorithm" when they could modify its output, "even when they are severely restricted in the modifications they can make" [25]. Approve, edit and reject on every item is that control, and leaving each suggestion to the steward's judgement is not a concession to soothe the stewards; it is what keeps them using it.

Fred Davis's work on technology acceptance adds the plainest test for a rollout: in his studies, "usefulness had a significantly greater correlation with usage behavior than did ease of use" [26]. The honest way to introduce a queue is to show its usefulness on the stewards' own records: run the backstop in the background on a sample first, let the stewards compare its suggestions with the decisions they made without it, and start the live queue with the classes of record where the comparison shows it helps.

Measurement is where this rollout needs the most care, and here my own rule needs a qualification. For any stewardship tool, I hold that the one reliable way to get stewards into it is to make it part of the daily work, and that the lever is moving their performance and reporting measures onto its use. Applied to an AI queue, that lever needs care, because the obvious measure of use is the wrong one. Measure the queue being worked: items decided, reasons recorded, time to decision, reversals. Do not measure agreement with the model. Marilyn Strathern's statement of Goodhart's law is exact on the point: "When a measure becomes a target, it ceases to be a good measure" [27]. A steward scored on acceptance rate will accept, and the acceptance rate will stop telling anyone whether the suggestions are any good.

The other half of that follow-up question, what tells you it is safe to give the AI more, does not arise under this framing, because the AI is never given more than a suggestion. What can widen is which records the backstop sees and which items the queue puts in front of the steward first. Both should widen on the queue's own counts, one class of record at a time, when the reversal rate shows that the suggestions hold up. Nothing on that path ends with the model writing to a master record on its own authority.

There is an outside statement of the same two requirements. The European Union's AI Act requires that the people overseeing a high-risk AI system be able "to remain aware of the possible tendency of automatically relying or over-relying on the output", which it names "automation bias", and be able "to otherwise disregard, override or reverse the output of the high-risk AI system" [28]. On my reading of Annex III, the Act's list of high-risk areas, master data stewardship does not generally fall within them, so this is not an obligation for most of it [28]; I cite it because a regulator, starting somewhere quite different, arrived at the two things this design protects: a reviewer who knows the pull of the machine's answer, and one who can say no to it.

The decision stays with the steward

Who Owns the Data, Really? traced how responsibility for fixing data has moved over the years, from the people the errors hurt, to IT, and on, until the present attempt to move it to AI. The design in this piece is not that attempt. It does not move the decision. It changes what the decision costs: the model gathers the evidence and proposes, the steward sees the evidence and decides, and the steward's budget of good decisions is spent on the records that need them.

The research gives that design a firmer footing than the phrase it is usually sold under. Decision fatigue, as a depleting mental resource, is contested, and I would no longer lean on it. What holds up better is narrower and more useful: workload and time pressure make people defer to machines, explanations can sell a wrong answer, and a person and a model together beat either alone on average, where the person is the better judge. In my experience, no AI system at a cost an organisation would accept yet reviews as accurately as a steward with domain knowledge. While that holds, the steward approves. Where an organisation measures that it has stopped holding for some class of records, the right move is a tested rule, not an unattended model. And the way to introduce any of it is the way I would have done it: as another set of eyes, offering a suggestion the steward is free to take or leave.

References

[1] Ivan P. Fellegi and Alan B. Sunter, "A Theory for Record Linkage," Journal of the American Statistical Association 64(328), December 1969, pp. 1183–1210. Read 2026-10-12 through a rendering fetch of a scanned copy (OCR); a note of what was read is stored locally. Cited for the three decisions, the sentence on the possible-link decision and the definition of the optimal rule, quoted in the body. https://cs.cornell.edu/~shmat/courses/cs6434/fellegi-sunter.pdf

[2] William E. Winkler, "Overview of Record Linkage and Current Research Directions," Research Report Series (Statistics #2006-2), Statistical Research Division, U.S. Census Bureau, 8 February 2006. Read 2026-10-12 through a rendering fetch; note stored locally. Cited for the term "clerical review region" and the 1992 Census of Agriculture example. https://census.gov/content/dam/Census/library/working-papers/2006/adrm/rrs2006-02.pdf

[3] Profisee, "Data Matching and Record Linkage Software," profisee.com, last modified 2 June 2025. Read 2026-10-12 through a rendering fetch; note stored locally. Vendor page (the vendor we partner with). Cited for the three phrases quoted in the body; the page does not describe thresholds. https://profisee.com/solutions/initiatives/matching-and-survivorship/

[4] Shai Danziger, Jonathan Levav and Liora Avnaim-Pesso, "Extraneous factors in judicial decisions," Proceedings of the National Academy of Sciences 108(17), 26 April 2011, pp. 6889–6892. Read 2026-10-12 through a rendering fetch; note stored locally. Cited for the two findings and the caveat quoted in the body. https://www.pnas.org/doi/10.1073/pnas.1018033108

[5] Shai Danziger, Jonathan Levav and Liora Avnaim-Pesso, "Reply to Weinshall-Margel and Shapard: Extraneous factors in judicial decisions persist," Proceedings of the National Academy of Sciences 108(42), 18 October 2011, p. E834, read in a Tel Aviv University copy that also holds the original article. Read 2026-10-12 through a rendering fetch; note stored locally. The critique it answers (Keren Weinshall-Margel and John Shapard, "Overlooked factors in the analysis of parole decisions," PNAS 108(42), p. E833) could not be retrieved; it is described in the body as the reply summarises it. https://en-coller.tau.ac.il/sites/coller-english.tau.ac.il/files/RP_190_Danziger.pdf

[6] Andreas Glöckner, "The irrational hungry judge effect revisited: Simulations reveal that the magnitude of the effect is overestimated," Judgment and Decision Making 11(6), November 2016, pp. 601–610. Read 2026-10-12 through a rendering fetch; note stored locally. Cited for the phrase from the title quoted in the body and the simulation argument, paraphrased. https://dlab.sauder.ubc.ca/sjdm/journal/16/16823/jdm16823.html

[7] Jeffrey A. Linder, Jason N. Doctor, Mark W. Friedberg, Harry Reyes Nieva, Caroline Birks, Daniella Meeker and Craig R. Fox, "Time of Day and the Decision to Prescribe Antibiotics," JAMA Internal Medicine 174(12), December 2014, pp. 2029–2031 (research letter). Read 2026-10-12 through a rendering fetch; note stored locally. One health system; scheduled time used as a proxy for visit time. https://jamanetwork.com/journals/jamainternalmedicine/fullarticle/1910546

[8] David Hirshleifer, Yaron Levi, Ben Lourie and Siew Hong Teoh, "Decision fatigue and heuristic analyst forecasts," Journal of Financial Economics, 2019 (NBER Working Paper 24293, February 2018). The abstract read 2026-10-12 through a rendering fetch of the NBER page; the full paper was not read. Note stored locally. https://www.nber.org/papers/w24293

[9] Roy F. Baumeister, Ellen Bratslavsky, Mark Muraven and Dianne M. Tice, "Ego depletion: Is the active self a limited resource?," Journal of Personality and Social Psychology 74(5), 1998, pp. 1252–1265. Abstract and introduction read 2026-10-12 through a rendering fetch; note stored locally. https://faculty.washington.edu/jdb/345/345%20Articles/Baumeister%20et%20al.%20%281998%29.pdf

[10] Martin S. Hagger, Nikos L. D. Chatzisarantis and others, "A multilab preregistered replication of the ego-depletion effect," Perspectives on Psychological Science 11(4), 2016, pp. 546–573. Read 2026-10-12 through a rendering fetch; note stored locally. Cited for the sample and the effect size. https://nature.berkeley.edu/garbelottoat/wp-content/uploads/hagger-chatzisarantis-2016.pdf

[11] Kathleen D. Vohs and others, "A Multisite Preregistered Paradigmatic Test of the Ego-Depletion Effect," Psychological Science 32(10), 2021, pp. 1566–1581. The abstract read 2026-10-12 through a rendering fetch of the PubMed record (34520296) and re-read 2026-10-13 on the publisher's page; the full paper was not read. Note stored locally. Cited for the confirmatory and exploratory results reported in the body. https://journals.sagepub.com/doi/10.1177/0956797621989733

[12] Kate Goddard, Abdul Roudsari and Jeremy C. Wyatt, "Automation bias: a systematic review of frequency, effect mediators, and mitigators," Journal of the American Medical Informatics Association 19(1), 2012, pp. 121–127. The abstract read 2026-10-12 through a rendering fetch; the full review was not read. Note stored locally. Focused on healthcare decision support, drawing on several research fields; not data stewardship. https://academic.oup.com/jamia/article/19/1/121/732254

[13] Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro and Daniel S. Weld, "Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance," CHI 2021 (arXiv:2006.14779). The abstract read 2026-10-12 through a rendering fetch; note stored locally. https://arxiv.org/abs/2006.14779

[14] Helena Vasconcelos, Matthew Jörke, Madeleine Grunde-McLaughlin, Tobias Gerstenberg, Michael Bernstein and Ranjay Krishna, "Explanations Can Reduce Overreliance on AI Systems During Decision-Making," arXiv:2212.06823, December 2022, revised January 2023. The abstract read 2026-10-12 through a rendering fetch; note stored locally. https://arxiv.org/abs/2212.06823

[15] John D. Lee and Katrina A. See, "Trust in automation: Designing for appropriate reliance," Human Factors 46(1), 2004, pp. 50–80. Read 2026-10-12 in a pre-print copy, through a rendering fetch, to about 85% of its length (the remainder is the bibliography and the closing discussion); note stored locally. Cited for the definitions of calibration and overtrust quoted in the body. https://csel.eng.ohio-state.edu/productions/intel/research/trust/Lee%20%26%20See%20Trust%20Review.pdf

[16] Profisee, "Aisey: The AI Assistant Redefining Enterprise MDM," profisee.com, last modified 11 September 2026. Read 2026-10-12 through a rendering fetch; note stored locally. Vendor page (the vendor we partner with). Cited for the phrases and the template name quoted in the body; the page does not describe audit trails or behaviour under load. https://profisee.com/platform/ai-assistant-aisey/

[17] Sunita Sarawagi and Anuradha Bhamidipaty, "Interactive deduplication using active learning," Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '02), 2002, p. 269. The abstract read 2026-10-12 through a rendering fetch of the repository record; note stored locally. https://repository.ias.ac.in/128419

[18] David Madras, Toniann Pitassi and Richard Zemel, "Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer," arXiv:1711.06664 (version of September 2018; NeurIPS 2018). The abstract read 2026-10-12 through a rendering fetch; note stored locally. https://arxiv.org/abs/1711.06664

[19] Erik Brynjolfsson, Danielle Li and Lindsey Raymond, "Generative AI at Work," The Quarterly Journal of Economics 140(2), 2025, pp. 889–942 (NBER Working Paper 31161, April 2023, revised November 2023). The abstract read 2026-10-12 through a rendering fetch of the NBER page; the full paper was not read. Note stored locally. Customer support, not data stewardship. https://www.nber.org/papers/w31161

[20] Fabrizio Dell'Acqua, Edward McFowland III, Ethan Mollick, Hila Lifshitz-Assaf, Katherine C. Kellogg, Saran Rajendran, Lisa Krayer, François Candelon and Karim R. Lakhani, "Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality," Harvard Business School Working Paper 24-013, September 2023. The abstract read 2026-10-12 through a rendering fetch of the HBS page; note stored locally. The page reports the outside-the-frontier result as "19% less likely"; the working paper's body (SSRN 4573321) reports it as an average decrease of 19 percentage points across the AI conditions, the reading used in the body, since the abstract's wording reads as a relative drop. https://www.hbs.edu/faculty/Pages/item.aspx?num=64700

[21] Michelle Vaccaro, Abdullah Almaatouq and Thomas Malone, "When combinations of humans and AI are useful: A systematic review and meta-analysis," Nature Human Behaviour 8(12), December 2024, pp. 2293–2303. The abstract read 2026-10-12 through rendering fetches of the IDEAS record and the arXiv page (arXiv:2405.06087); the full paper was not read. Note stored locally. https://ideas.repec.org/a/nat/nathum/v8y2024i12d10.1038_s41562-024-02024-1.html

[22] Raja Parasuraman and Dietrich H. Manzey, "Complacency and Bias in Human Use of Automation: An Attentional Integration," Human Factors 52(3), 2010, pp. 381–410. The abstract read 2026-10-12 through a rendering fetch of the TU Berlin repository record; the full paper was not read. Note stored locally. https://depositonce.tu-berlin.de/items/0c787d91-6021-480b-a333-009e0d1d0092/full

[23] Zana Buçinca, Maja Barbara Malaya and Krzysztof Z. Gajos, "To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making," Proceedings of the ACM on Human-Computer Interaction (CSCW), 2021 (arXiv:2102.09692). The abstract read 2026-10-12 through a rendering fetch; note stored locally. https://arxiv.org/abs/2102.09692

[24] Berkeley J. Dietvorst, Joseph P. Simmons and Cade Massey, "Algorithm aversion: People erroneously avoid algorithms after seeing them err," Journal of Experimental Psychology: General 144(1), 2015, pp. 114–126. The abstract read 2026-10-12 through a rendering fetch; note stored locally. https://faculty.wharton.upenn.edu/wp-content/uploads/2016/11/20-Dietvorst-Simmons-Massey-2015.pdf

[25] Berkeley J. Dietvorst, Joseph P. Simmons and Cade Massey, "Overcoming Algorithm Aversion: People Will Use Imperfect Algorithms If They Can (Even Slightly) Modify Them," Management Science 64(3), 2018, pp. 1155–1170. The abstract read 2026-10-12 in the authors' working version through a rendering fetch; the published version was not read. Note stored locally. https://marketing.wharton.upenn.edu/wp-content/uploads/2020/07/Dietvorst-Overcoming-Algorithm-Aversion.pdf

[26] Fred D. Davis, "Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology," MIS Quarterly 13(3), 1989, pp. 319–340. The abstract read 2026-10-12 through a rendering fetch of a library record; the full paper was not read. Note stored locally. https://docs.edtechhub.org/lib/W4SUF2B4

[27] Marilyn Strathern, "'Improving ratings': audit in the British University system," European Review 5(3), 1997, pp. 305–321. Read 2026-10-12 through a rendering fetch; note stored locally. Cited for the sentence quoted in the body, which the paper itself attributes, through Keith Hoskin, to Goodhart. https://www.gwern.net/doc/statistics/decision/1997-strathern.pdf

[28] Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 (Artificial Intelligence Act), Article 14, "Human oversight." Title and date read 2026-10-12 on EUR-Lex (https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng); the text of Article 14 read the same day on an unofficial rendering of the Act, through a rendering fetch; note stored locally. Cited for paragraph 4, points (b) and (d), quoted in the body. Annex III, the high-risk areas referred to in Article 6(2), read 2026-10-13 on the same unofficial rendering (https://artificialintelligenceact.eu/annex/3/); the reading of it in the body is the author's, not the Act's. https://artificialintelligenceact.eu/article/14/