DMS.LABS / Ideas / ReedoEngineering & transformation
Optical study03 / Refraction

Ideas

Notes and essays written alongside the work. Earlier pieces are still available in Korean, and new ones are being added in English.

17 English editionsA space where images and words meet
Several sheets of paper with fine, looping pencil lines overlap on a worn wooden table by a window, with small circles and short diagonal strokes in the corners and margins. A sharpened pencil and a small eraser lie on the right, and a ceramic jug holding eucalyptus sprigs stands on the left in soft morning light. No text is legible.
Generated image. A desk scene for illustration, not a real record.View full image ↗

Essay / 2026. 11. 19

After the AI Gets One Wrong

Two studies on how people use an algorithm once they have seen it make a mistake

Even when an algorithm was more accurate than a person, participants who watched it err chose its forecasts less often. This essay follows what two studies, from 2015 and 2018, did and did not show, then offers a DMS.Labs suggestion for settling in advance what to do when an AI gets something wrong.

Once people have watched an AI get something wrong, do they use it less? Five experiments published in 2015 said yes. Even though the algorithm was more accurate than the human on average, participants who saw it make mistakes were reluctant to tie their bonus to its forecasts. A 2018 follow-up reported that letting people adjust the forecasts, even slightly, closed much of that gap. But both studies used tasks that ask for a number. Neither looked at generative AI that writes or summarizes. This essay follows what the two studies showed and, as a DMS.Labs suggestion, offers a way to decide in advance how to handle an AI’s mistakes when it enters your work.

Based on sources checked on November 19, 2026. I read the full text of both papers from the PDFs on a Wharton faculty page. I could not read the selection rates in Figure 3 of the 2015 paper because they are in an image, and I did not read the supplementary materials or later work by other researchers. The language-school scene below is an invented illustration, not a DMS customer case. The last section is a DMS.Labs suggestion that no paper has tested.

Where did the term “algorithm aversion” come from?

It comes from a 2015 paper in the Journal of Experimental Psychology: General by Berkeley Dietvorst, Joseph Simmons, and Cade Massey, titled “Algorithm Aversion: People Erroneously Avoid Algorithms After Seeing Them Err.” In the paper, an algorithm means any evidence-based forecasting formula or rule, which includes a statistical model.

The authors begin with decades of evidence that algorithms forecast better than people. Meehl reviewed 20 studies in 1954, and a 2000 meta-analysis by Grove and colleagues covering 136 studies found algorithms ahead by 10% on average. Yet people often picked the human forecast anyway. The authors asked when and why that happens.

In five experiments, participants forecast real outcomes from real data. In three of them they predicted how well MBA applicants performed after admission, as a percentile. In two they predicted the rank of U.S. states by the number of airline passengers departing. The researchers built a statistical model from the same data, and at the end each participant chose whether to tie a bonus to their own forecasts (in one experiment, another participant’s) or to the model’s. What varied was what they had seen first: the model’s forecasts only, their own forecasts and the right answers only, both, or neither. The final samples ranged from 206 to 1,036 people per experiment.

The model really was better. In the MBA task the human forecasters made 15% to 29% more error than the model, and in the airline task 90% to 97% more. In every condition, participants would have earned more by tying their bonus to the model.

What changed once people saw the algorithm make a mistake?

Participants who watched the model forecast chose it less. The difference was clear in all four experiments that manipulated experience with the model. The most telling group saw both the model and the human. Of the 741 participants in that condition across the five experiments, 610 (83%) saw the model beat the human with their own eyes. Even so, this group was among the least likely to choose the model. Looking only at those who saw the model ahead, they still chose it less than people who had not seen the model perform. In the first airline experiment (3a), though, that comparison was not statistically clear.

The other side looked different. Seeing the human forecast did not clearly reduce how often people chose the human, even though the human’s errors were larger.

The confidence ratings pointed the same way. In the first experiment, confidence in the model (on a 1 to 5 scale) was 3.04 for people who had seen nothing, and 2.49 and 2.63 for the two groups that saw the model forecast. Confidence in one’s own forecasts ranged from 2.47 to 2.70 across conditions, with no meaningful difference. Most people who rated the model and the human equally confident chose the human, and this was especially true after they had seen the model perform. People chose the model mostly when they trusted it more.

The authors’ explanation is that we forgive the same mistake differently. A driver who takes a different route to work and arrives late usually shrugs it off, they say, but if a GPS makes the same call and errs, many of us hesitate to trust it again. Their analysis also found that a drop in confidence in the model explained the choice, and confidence in the human did not.

Expectations were high, too. In the second airline experiment, participants who had seen nothing expected humans to get 16.7 of the 50 state ranks exactly right (33%) and the model to get 30.4 (61%). In reality, humans got 2.2 right (4%) and the model 6.0 (12%). Even after seeing the model forecast, people still expected about 15 exact hits, far above what happened.

The authors state some limits themselves. They could not say whether people would still use a model after seeing it err if it were many times more accurate than humans. A fair share of participants who had not seen the model err also chose not to use it, so seeing a mistake is not the whole story. And there were only two kinds of task.

What happens if people can adjust it a little?

The follow-up paper, by the same team in Management Science in 2018, asked whether that distance can be reduced. Participants knew the algorithm was imperfect; would they use it more if they could modify its forecasts a little? The first experiment had 288 participants in the Wharton lab, who forecast the standardized math-test percentiles of 20 high school students. They were told in advance that the model was off by 17.5 percentiles on average.

In one condition, people could only use the model as is or not at all. In another, if they chose the model, they could adjust every forecast by up to 10 percentiles. In a third, they could change 10 of the 20 forecasts by any amount. When nothing could be changed, 32% chose the model. When they could adjust by up to 10 percentiles, 76% did, and when they could change 10 forecasts, 73% did.

Performance differed too. Average absolute error was 22.97 when nothing could be changed, against 19.97 to 20.44 in the three conditions where changes were allowed. The bonuses were $1.14 against $1.46 to $1.53. The authors explain that very large errors were more common among those who did not use the model.

The second experiment, with 816 online participants, varied the limit to 2, 5, or 10 percentiles. With no changes allowed, 47% chose the model. With limits of 10, 5, and 2, the figures were 71%, 71%, and 68%. A tighter limit barely changed the choice. The authors read this as a wish for some control, not for more control.

The third experiment looked at reactions after trying it. People made their first ten forecasts using the model as is, adjusting by up to 10 percentiles, or adjusting freely, and were then asked how to make the second round. Those who had to use the model as is were less satisfied with the process than those who could modify it. In every condition more than 80% used the model in some way the second time, and mixing it with their own judgment was the most common choice (51% to 74%).

These results come with conditions. The authors note that when people hold important information the algorithm lacks, large adjustments may help. They tested limits only from 2 to 10 percentiles, and they say results might change if the algorithm were much less accurate. Participants were students and online workers, and they call for research with professional forecasters. They also cite earlier studies in which people’s adjustments to an algorithm often made forecasts worse. In the first experiment, the groups that could adjust did better than the group that could not, and the authors report that in the third experiment the group held to a limit did better still.

What do these experiments say about using AI at work?

Not much directly. The tasks asked for a forecast whose accuracy shows up as a number, and participants saw the size of each error as a number. A draft or summary from a generative AI is harder to check, and the size of its error does not come as a number. The participants were students and online workers in bonus-paid experiments, not people using AI on the job.

One passage in the papers speaks for itself, though. The authors say that posing the choice as all or nothing is likely to be counterproductive. They also warn that forcing people to use an imperfect algorithm exactly as it is may breed dissatisfaction and a push for change. The papers answer this with number-forecasting tasks, and they cannot tell us whether the same holds for an AI that writes.

How could you settle in advance what a mistake means?

Write down your rules for mistakes before you start using an AI. From here on this is a DMS.Labs suggestion, not a research result. We have not tested it and cannot promise an effect.

Picture a language school with four staff members that hands draft replies to parents’ questions to an AI. The first week goes smoothly. In the second week, someone catches a draft that gives the class start time an hour late, just before it goes out. The director says this will not work and returns to writing every reply by hand. That one draft stands out, while the record of all the other drafts that went out without trouble does not. This school is an invented illustration.

Instead, write down three things before you start. First, where mistakes are likely: pick the items that cause trouble when wrong, such as dates, amounts, and names. Second, how far a person may change the draft: reword sentences freely, but change dates and amounts only after checking the source. This is DMS.Labs’ translation of the 2018 study’s “let people adjust a little” into daily work. Third, what happens after a mistake: do not drop the tool the moment it errs. Note the mistake and what you fixed in one line each, count after a month how many drafts out of how many were wrong, and decide again.

Editing has risks of its own. Earlier studies cited in the 2018 paper found that adjustments people make to an algorithm’s forecasts often make them worse. So record what you changed and why, and count whether the person’s version was better than the AI’s original.

A large sheet of paper lies spread out like a map on a wooden table, with a winding pencil line that has been corrected to take a short detour in the middle. A small brass compass with its lid open and a short pencil rest on the right, and the leaves of a potted vine show at the window in soft natural light. No text is legible.
Generated image. A tabletop scene for illustration, not a real record.View full image ↗

Can anyone say this works?

Not yet. None of the studies above tested this approach with an AI that writes, used at work. There is no evidence for where the limit should sit or whether a month is the right period. What I hope for is modest: that one error does not become the entire decision about whether to keep using the tool, because the criteria were settled beforehand.

There is a way to check. Use it on a single task for one month, and first record how many mistakes you counted the same way during a few weeks before you started. You need that record to compare with the period after you start using the AI.

Who checks AI output, and by what standard, is in “Naming a Reviewer for AI Output.” How to test a tool before you rely on it is in “Acceptance Tests for an AI Skill.”Naming a Reviewer for AI OutputAcceptance Tests for an AI Skill

This week, pick one thing an AI made, and write down three places where it is likely to go wrong and how far a person may change it. The next time a wrong result appears, open that note.

If you want to review how your team uses AI, you can reach us through the contact page.Contact

Sources

Next: DMS Generative AI Education (in Korean)

A lined notebook lies open on a worn wooden table by a window, with a few faint lines of pencil handwriting on the right page. A pencil and a small teacup with a leaf pattern sit beside it, and a closed laptop rests behind. Leaves of a houseplant show at the window on the left in soft morning light. No text is legible.
Generated image. A desk scene for illustration, not a real record.View full image ↗

Essay / 2026. 11. 16

When Notes Write Themselves

What three studies say about taking notes and understanding

A 2014 experiment found that handwritten notes beat typed ones for understanding, but later replications did not find the same result consistently. This essay follows what three studies confirmed and what they did not, then offers a DMS.Labs suggestion for reading AI-written meeting notes.

If an AI writes down your meeting, does the way you understand it change? The research we have cannot say. A 2014 experiment found that handwritten notes beat typed notes for understanding, and it became widely cited. Replications in 2019 and 2021 did not find the same result consistently. That does not show that transcribing is harmless. These studies measured students who wrote their own notes while watching a talk and then took a quiz. None of them looked at notes an AI wrote for the reader. This essay follows what the three studies confirmed and, as a DMS.Labs suggestion, offers a way to read AI meeting notes afterward.

Based on sources checked on November 16, 2026. I read the full text of Mueller and Oppenheimer’s 2014 paper and its 2018 corrigendum. For the 2019 study by Morehead and colleagues I read only the journal abstract. For the 2021 study by Urry and colleagues I read the abstract and a summary from the British Psychological Society’s Research Digest, not the full paper or appendices. The language-school meeting below is an invented illustration, not a DMS customer case. The last section is a DMS.Labs suggestion that no paper has tested.

Where did “handwriting beats typing” come from?

It comes from a 2014 paper in Psychological Science by Pam Mueller and Daniel Oppenheimer, titled “The Pen Is Mightier Than the Keyboard.” Across three experiments, students who typed their notes scored lower than students who wrote by hand on questions about concepts.

The first experiment had 67 Princeton students, and two were dropped from the analysis. Each watched one of five TED talks of a little over 15 minutes and took notes the way they normally would. The laptops were offline. About 30 minutes after the talk, and after other tasks, they answered factual questions and conceptual-application questions. The two groups did about equally well on facts. On concepts, the laptop group scored lower.

The notes looked different too. Handwritten notes averaged 173.4 words and typed notes averaged 309.6. The share of three-word chunks in the notes that also appeared in the lecture transcript was 8.8% for handwriting and 14.6% for typing. The authors read this as typists transcribing what they hear, with less chewing it over. Across participants, longer notes went with higher scores, and notes that overlapped more with the lecture went with lower scores.

In the second experiment, the final sample was 151 UCLA students. Some laptop users were told in advance to write in their own words rather than word for word. The authors reported that this instruction did not stop the verbatim habit. Laptop users who got the instruction did not differ significantly on conceptual questions from handwriters or from laptop users who did not get it.

The third experiment had 109 UCLA students in the final sample and tested them a week later. They watched four lectures of about seven minutes each. Half could look over their own notes for 10 minutes before the test, and half could not. The handwriting group that reviewed its notes scored higher than the other three groups. Where there was no chance to review, typing and handwriting did not differ significantly.

The authors published a corrigendum in 2018. It recalculated some scores and fixed errors in how the statistics were reported, and it said the overall direction of the results held. The score details in this essay follow the corrected version.

What happened when others ran the experiment again?

The result did not hold up consistently. A close replication of the first experiment appeared in Psychological Science in 2021. Heather Urry and colleagues ran it with 142 Tufts students, 74 typing and 68 writing by hand. As in the original, the typists wrote more words and copied more of the lecture verbatim. Their quiz scores, however, were about the same as the handwriters’. A small meta-analysis of eight studies with similar designs, run by the same team, also showed no handwriting advantage on conceptual questions.

The replication differed from the original in some ways. According to the BPS summary, more of its participants said they normally take notes by hand, and about 80 took part outside a class, watching the talk on a laptop the team provided, and many of them described those sessions as prone to distractions and errors. The short talks also sit far from a real university lecture. The authors acknowledged that such differences could have affected the results.

In 2019, Morehead, Dunlosky, and Rawson repeated the original work and added a group that used electronic writing devices and a group that took no notes. According to the abstract, scores did not differ consistently between groups, including the group that took no notes. After students studied their notes, group differences shrank further. A meta-analysis of the direct replications showed a small effect favoring handwriting that was not statistically significant. The authors wrote that it is premature to say which method is better.

What do the three studies still agree on?

Not much holds firmly, but a few clues remain. People who wrote more notes tended to score higher. In both the original study and the 2021 replication, a higher word count went with a better quiz score. That is an observed pattern, though, not a result of making people write more.

The link between overlap with the lecture and the score is unstable. In the original, more overlap went with lower scores, but the replication did not find this robustly. The replication’s authors concluded that their results do not support the idea that handwriting improves immediate learning by encoding information better.

The role of reviewing notes is mixed. In the original third experiment, only the handwriting group that reviewed stood out. In the Morehead experiment, group differences shrank after students studied their notes. Which way the same step helped varied from study to study.

What does this say about AI-written meeting notes?

Nothing directly. The participants were students who wrote their own notes, and nobody read a record an AI had made. The test was a quiz on a short talk. What a meeting needs is memory of what was decided and who does what by when, and these studies did not measure that.

There is a resemblance. AI meeting notes are close to transcription at its extreme: nearly every word ends up on the page with almost no effort from the reader. That matches the worry in the original study, a record that grows longer while the chewing-over shrinks. But the replications weakened the grounds for that worry, so resemblance alone does not let us predict the same result. That is as far as I can check.

What could you do before opening the AI summary?

Write three lines in your own words before you open the AI summary. From here on this is a DMS.Labs suggestion, not a research result; we have not tested it and cannot promise an effect.

Picture a language school with five staff members and a weekly meeting. A recording app transcribes the meeting and shares a summary. The teacher in charge reads it, thinks it is well organized, and closes the window. Three days later someone asks who agreed to change next week’s class schedule, and the staff give different answers. The summary probably said it. Reading something and remembering it are separate things, which is how such a scene can happen. This is an invented case for illustration.

Here is the method. After the meeting, before opening the AI summary, spend three minutes writing three lines in your own words: what was decided, what you are responsible for and by when, and what you did not understand. Then open the summary and compare. Whatever is in your three lines but not in the summary, and whatever is in the summary but not in your lines, becomes your list to check.

A long strip of white paper printed with dense, fine lines unrolls from a paper roll and spills over the edge of a wooden table by a window. To the right lie a palm-sized blank card and a green pencil in warm natural light. No text is legible.
Generated image. A tabletop scene for illustration, not a real record.View full image ↗

Can anyone say this works?

Not yet. None of the studies above tested this sequence. There is no evidence that three minutes or three lines is the right size. What I would expect is that gaps become easier to notice. Whether understanding deepens needs its own check.

A check is possible. Use it for one type of meeting for a month. Three days after each meeting, ask everyone to write down separately what they own, and count how often the answers match. You need the same count from a few weeks before you started, so write that record down first.

Before you switch on an AI note-taker, the guide on notifying attendees and file retention is “Before the AI Note-Taker Joins.” Who reviews the output, and by what standard, is in “Naming a Reviewer for AI Output.”Before the AI Note-Taker JoinsNaming a Reviewer for AI Output

This week, pick one meeting and write three lines before opening the AI summary. Any line that does not appear in the summary is your first question to check.

If you want to review how your team approaches AI use and work automation, you can reach us through the contact page.Contact

Sources

Next: DMS Work Automation (in Korean)

A pair of reading glasses rests on a printed sheet of paper on a worn wooden table by a window. One line on the sheet is marked with pale yellow highlighter. A pencil, a mug, and a leafy plant sit nearby in soft morning light, and no text is legible.
Generated image. A desk scene for illustration, not a real record.View full image ↗

Essay / 2026. 11. 13

When the AI Says “I’m Not Sure”

What a hedge does to the person reading it

When an AI answer opened with “I’m not sure, but...”, people agreed with it less often and scored higher. This essay follows a 2024 FAccT experiment to see how far that goes, then suggests what a team could try when its AI sounds unsure.

Does opening an AI answer with “I’m not sure, but...” change what the reader does? In one experiment it did. People who saw hedged answers agreed with the AI less often and scored higher on the questions. The harm from wrong answers did not go away, though. The study appeared at FAccT in 2024, and its task was eight yes/no questions from a medical domain where lay people rarely know the answer. This one study cannot tell us the same holds for drafting a memo or making a business call. Below I follow what the experiment actually changed, then set out a few things DMS.Labs suggests trying at work when an AI sounds unsure.

This reflects sources checked on November 13, 2026. For the paper by Kim, Liao, Vorvoreanu, Ballard, and Vaughan (FAccT ’24, June 2024), I read the methods, results, discussion, and limitations in the arXiv version (v2), but not the appendices. For the second study, by Steyvers and colleagues, I read only the abstract. The office scene near the end is a hypothetical for illustration, not a DMS client case. The closing suggestions are DMS.Labs’s own ideas and have not been tested in any study.

How was the hedge attached to the answers?

The participants were U.S. workers on Amazon Mechanical Turk. Of 656 complete responses collected over two weeks in September 2023, the authors removed 252 (38.4%) under five pre-registered criteria and analyzed 404. People were split into four groups. A total of 104 saw AI answers with no hedge, 92 saw first-person hedges (“I’m not sure, but it seems...”), 94 saw impersonal hedges (“There is uncertainty, but it seems...”), and 114 answered without any AI.

The task was eight yes/no questions from the medical domain, chosen so that lay people would struggle to know the answer and could not find it by pasting the question into a search engine. The AI was presented as a fictional search engine, “AI system A,” but its answers were taken, with minor edits, from what Copilot in Bing actually returned in July 2023. Four of the eight answers were right and four were wrong, so the AI was correct 50% of the time. Always following it was no better than a coin flip, by design.

A hedge was added to half of the eight answers, and which questions got one was randomized. Apart from the opening phrase, the three versions carried the same information. So the experiment does not show what happens when an AI correctly warns about the answers most likely to be wrong. It shows what happens to people when one phrase in the answer changes. The authors say as much in their limitations: the AI in the study had low accuracy and expressed uncertainty often, in a poorly calibrated way.

What changed in the reader when the same answer came with a hedge?

Without a hedge, 80.9% of participants gave the same answer as the AI. People who answered on their own happened to land on the AI’s answer 58.4% of the time, so the AI’s pull was clear. With first-person hedges, agreement fell to 74.8%, a statistically significant drop. With impersonal hedges it was 77.6%, which was not a significant difference.

Accuracy moved the other way. It was 63.9% for the no-hedge group, 72.8% for first-person hedges, 67.9% for impersonal hedges, and 74.2% for people with no AI. Only the first-person gap from the no-hedge group was significant. The paper does not report a significant difference between the two hedge styles, so I would not rank them on this evidence alone.

Confidence in the AI’s answer, on a five-point scale, was 3.95 with no hedge, 3.66 with first-person hedges, and 3.80 with impersonal hedges. A separate score for willingness to use the system was lowest for first-person hedges at 2.91, against 3.25 with no hedge and 3.36 with impersonal ones. One participant called the system “very non-committal” and said that made it hard to trust. Yet the score for whether the system was trustworthy at all did not differ between groups. The hedge made people less keen to use the AI without overturning their overall rating of it.

What happened when the AI was right, and when it was wrong?

The authors split the questions into the four the AI got right and the four it got wrong. This was an exploratory analysis, not one of the pre-registered tests. In the no-hedge group, accuracy on questions the AI got right was 88.5%, above the 77.9% of people working alone. On questions it got wrong, accuracy was 33.0%, about half the 64.7% of people working alone. The AI helped when it was right and pulled people down with it when it was wrong.

Within the hedge groups, comparing questions with a hedge to questions without one pointed in two directions. On questions the AI got wrong, accuracy rose from 43.6% to 52.0% in the first-person group and from 32.8% to 48.0% in the impersonal group. On questions the AI got right, accuracy slipped from 92.2% to 89.2% in the first-person group and from 94.8% to 83.1% in the impersonal group. The hedge slowed people down in front of wrong answers, and it slowed them a little in front of right ones too.

Even so, accuracy on wrong-answer questions in the hedge groups stayed well below the 64.7% of people with no AI. The hedge cushioned the effect. It did not erase it.

How did people read the hedge?

The 186 participants in the two hedge groups were asked when and why they thought the AI expressed uncertainty. A majority, 102, took it to mean the system could not answer the question: it found no answer, found conflicting sources, or could not make sense of what it found. Five read it as an invitation to do their own research. A small number read it as the AI trying to seem human, protect its credibility, avoid liability, or follow limits on medical advice. The authors note that those readings do not necessarily reduce overreliance. Eleven people wrote that the hedge made them open the linked sources, and thirteen wrote that it made them search on their own.

One participant described the other side of this: “If the AI seemed confident I assumed he was correct most of the time.” Reading that as the flip side of a hedge is my interpretation, not the paper’s: an answer with no hedge can work as a signal to stop checking, whether or not the answer is right.

What if the wording matched what the model actually knows?

Another line of work looks at exactly that. Steyvers and colleagues, writing in Nature Machine Intelligence (volume 7, issue 2, published January 2025), report that users who received default explanations overestimated how accurate the AI’s answers were. Longer explanations raised user confidence even when the extra length did not improve accuracy. When the explanations were adjusted to reflect the model’s internal confidence, both the gap between human and model confidence and the gap in telling right answers from wrong ones narrowed. I read only the abstract, so I leave out the design and the numbers.

Read together, the two studies fit like this. In the first, hedges were placed regardless of whether the answer was right. We see how a phrase can make people pause, but not how much it helps when it lands only on the answers likely to be wrong. The second points toward matching the wording to what the model actually has. The tasks, models, and measures differ, so neither confirms the other.

What can a team try when its AI says “I’m not sure”?

From here on this is a DMS.Labs suggestion, not a research result. We have not tested it and cannot promise an effect.

Picture a six-person interior-design firm. A staff member asks an AI to tidy up material prices, and one line in the answer opens with “I’m not sure, but this size seems to have gone up since last quarter.” If the team treats that line as a disclaimer, the AI’s weakest spot goes straight into the quote. Checking that line against the supplier’s price list first points the same way the hedge pushed participants in the experiment. This is a made-up case for illustration.

The other side needs checking too. A line with no hedge is not a guarantee. The AI answers the no-hedge group saw were wrong half the time. Picking one line with a number or a date from the unhedged part and checking it against the source is the safer habit.

In practice, you could add two columns to an AI output review card: sentences the AI said with doubt, and sentences it said firmly that you checked anyway. Check all of the first kind, and one or two of the second kind that hold numbers, dates, or proper names. Try it on a small task before settling on a ratio for your team.

A tape measure is unrolled partway across a wooden workbench next to a pencil, a small wooden spirit level, and a few wood shavings. Hand tools hang on the wall behind, and warm window light from the right falls across the wood grain.
Generated image. A workbench scene for illustration, not a real record.View full image ↗

Who reviews AI output, and by what standard, is covered in “Naming a Reviewer for AI Output.” What an AI skill should pass before a team relies on it is in “Acceptance Tests for an AI Skill.”Naming a Reviewer for AI OutputAcceptance Tests for an AI Skill

This week, take one AI answer and mark the hedged sentences in one color and the firm ones in another. Then check one line of each against the source.

If you want to review how your team approaches AI training and use, you can reach us through the contact page.Contact

Sources

Next: DMS Generative AI Training (in Korean)

A wooden table by a window with two open books, loose sheets of paper covered in handwritten notes, a pencil and a mug. A green vine grows by the window and soft morning light falls across the table.
Generated image. A desk scene made for illustration, not a record of a real event.View full image ↗

Essay / 2026. 11. 10

Answers Found, Answers Handed

Between reading a summary and following the links

People who learned from an AI summary and people who followed search links reported learning differently, even when they saw the same facts. This essay follows seven experiments published in PNAS Nexus, marks how far the results reach, and suggests three steps for after reading a summary at work.

People handed a summary reported learning less than people who gathered the same facts by following links, and the advice they wrote afterward was shorter and more alike. That is the pattern in seven experiments published in PNAS Nexus in October 2025. The task, though, was writing advice for a friend, and the topics were everyday ones such as vegetable gardening and healthy living. This study alone does not show the same result for looking up a single fact or for learning a specialized field. Below I follow what the experiments actually compared, say how far the results reach, and end with suggestions from DMS.Labs for what to do after reading an AI summary at work.

Based on sources checked on November 10, 2026. I read the full text of Shiri Melumad and Jin Ho Yun's paper in PNAS Nexus, volume 4, issue 10 (published October 28, 2025), a peer-reviewed journal article. I did not read the supplementary material, so for the replications described there I rely on the summaries in the main text. The office scene near the end is a hypothetical for illustration, not a DMS client case. The suggestions at the end are my own and were not tested in the paper.

What differed between two people who asked the same question?

The group handed ChatGPT answers spent less time with the results and said they learned less. The first experiment split 1,136 online panelists at random into two groups. After 32 people who failed an attention check were removed, 1,104 remained. Everyone imagined a friend asking how to plant a vegetable garden. One group searched on the real Google; the other used a ChatGPT (GPT-3.5 turbo) that the researchers had built into the survey. Each searched as much as they liked and then wrote advice for the friend. The Google group could open any links and change their query as often as they wished.

The number of queries was nearly the same: 2.15 on average for Google and 2.06 for ChatGPT. The time spent on the search task was not. The Google group averaged 742.81 seconds (about 12 minutes 23 seconds) and the ChatGPT group 585.41 seconds (about 9 minutes 45 seconds). The paper reads this gap as less time spent with the results that came back, not as fewer questions asked.

Learning was measured by survey, from 1 (strongly disagree) to 5 (strongly agree). On "I learned new things from the results," the Google group scored 3.86 and the ChatGPT group 3.43. On feeling a sense of ownership over what they learned, the scores were 3.55 and 3.36. The advice differed in shape too: 94.64 words on average for Google and 84.58 for ChatGPT. When the researchers computed how much the pieces of advice overlapped in topic, using cosine similarity, the Google group scored 0.057 and the ChatGPT group 0.159. A higher number means the texts resemble each other more, so the advice from people who learned through ChatGPT was more alike.

These are group averages. The word count differs by about ten words, and the result does not mean that everyone who used ChatGPT learned shallowly.

What if both people were given the same facts?

The differences mostly remained. The real Google and the real ChatGPT can return different content to begin with, so the second experiment fixed the information. The researchers asked GPT-4o for vegetable-garden advice and got a 291-word answer with seven suggestions. Participants in the ChatGPT condition saw that answer as the summary. For the Google condition, the researchers kept the same seven suggestions and had GPT-4o write six articles in the styles of outlets such as Better Homes and Gardens, Martha Stewart and The New York Times, shown as six links. Whether a participant opened one link or all six, the facts they met were the same. Those articles were also written by GPT-4o, so the setup does not reproduce the variety of the real web. There were 1,979 participants.

The ChatGPT group took longer to type their query (59.45 seconds against 49.42) but spent less time with the results: 83.65 seconds against 124.32. On "I learned new things," the scores were 3.96 and 3.71. Advice averaged 74.22 words against 64.49, and the similarity among the pieces of advice was 0.072 against 0.224. The sense that the information was comprehensive was about the same, 4.25 against 4.30.

I read this as the most striking part of the study. The facts were identical and only the format differed, yet the feeling of having learned and the shape of the advice changed. The authors explain that a summary does the choosing, reading and combining for you, which makes learning more passive. A statistical model built from survey answers and time records fit that account, but the process itself was not observed directly.

Did a Google-made summary, or a summary with links, behave the same way?

Largely yes, in the same direction. The third experiment ran in a laboratory with 253 members of a university behavioral-research panel; after 3 failed an attention check, 250 remained, with an average age of 22.75. Everyone used Google. One group's computers had the AI Overview extension installed and the group was told to look only at the AI summary at the top of the results. The other group skimmed the links without an AI summary. The topic was how to lead a healthier life.

Search time was 67.37 seconds for the links group and 55.50 for the AI summary group. The item "the results gave comprehensive information" showed the widest gap in this experiment, 4.18 against 3.32, and advice averaged 90.17 words against 65.17. Holding the search engine fixed at Google produced the same direction. The sample was a small university panel, so it is hard to carry over directly to working adults.

The main text also summarizes two supplementary experiments. One was a replication with 2,402 people who said healthy living mattered to them, adding a condition in which the AI summary came with real-time web links. The same direction appeared even then. The paper notes that only 26% of those participants actually clicked the links. The other was a replication with 1,976 people on what to do after becoming the victim of a financial scam, and it pointed the same way. Having links attached did not mean people opened them.

How did the advice from summary readers look to the people who received it?

Slightly worse, on every rating. The fourth experiment looked at the receiving side. 1,493 people each read two pieces of advice from the third experiment. One was drawn at random from 125 written by people who learned through links, the other from 125 written by people who learned through the AI summary, and the readers did not know which was which. They rated each piece from 1 to 5.

On all six items the advice from the links group scored higher: helpful, 3.816 against 3.553; informative, 3.575 against 3.224; the writer seemed to have put in effort, 3.421 against 3.024; likely to recommend, 3.821 against 3.481; trustworthy, 4.171 against 3.906; likely to adopt, 3.897 against 3.711. The gaps run from about 0.2 to 0.4 points on a five-point scale. That is hard to call large, but it is consistent in direction. What was measured is willingness to adopt the advice, not whether anyone did.

A single printed sheet of paper lies on a wooden desk beside an open lined notebook with nothing written in it. A pencil rests across the notebook, and a closed laptop and a small plant sit nearby. Soft light from the window falls across the desk.
Generated image. A desk scene made for illustration, not a record of a real event.View full image ↗

How far can these results be carried?

The paper lists its own limits. The task was to learn something and then write advice for another person. For looking up one fact, such as a date, the authors see no obvious advantage in web links over an AI. The topics were everyday questions that need little background knowledge, and the authors write that in highly specialized fields such as advanced science or math, an AI summary might help people understand more. Learning was gauged by participants' self-reports, the length and factual content of their advice, and readers' impressions, not by standardized test scores. For goals like test preparation, the authors say the AI may do better by pointing to the key facts.

I would add three points. First, as noted above, the link articles in the second experiment were also written by an AI. Second, the tools were GPT-3.5 turbo, GPT-4o and the 2025 AI Overview, so it cannot be assumed that today's tools give the same result. Third, the authors speculate that deep research tools, which investigate many sources for the user, might widen the gap, but that was not tested here.

Nor is the paper evidence that thinking ability is shrinking. The authors discuss the possibility that leaning on AI weakens skills, yet call it still an open empirical question. Their conclusion is careful: learning can shift from an active to a passive activity, so people should know how much they rely on a tool, and one may be better off not letting an AI do the Googling.

What can you do after reading a summary?

From here on, the content is DMS.Labs's suggestion and assumption rather than something the paper tested, and its effect has not been checked. As the results above show, adding links to a summary did not make the pattern disappear, so this is not a suggestion to attach links.

Take a four-person facilities company as an assumption. A new hire reads a revised safety inspection standard through an AI summary and writes a customer notice. The sentences are smooth, but the notice cannot be told apart from a senior colleague's, and when a customer asks about one item, the new hire has no answer. The only part of this scene grounded in the paper is the tendency for advice written after a summary to be more alike. The rest is imagination.

After reading a summary you can do three things. First, close the screen and write three lines in your own words: one main point, one piece of evidence, and one thing you still do not know. Second, pick one of the original sources the summary points to, actually open it, and read only the relevant passage. Third, compare your three lines with the summary and fix whatever is wrong or missing. This gives back a little of the choosing and combining the authors describe, but there is no evidence yet that it deepens learning. For looking up a single fact, such as a date or a specification, I do not think the routine is needed.

If your team handles AI training, two other pieces cover what a workplace prepares after training and what trainees can check in the first two weeks.AI training should start with 'Should I hand this task over?'What the Workplace Does After AI Training

This week, pick one document you read through a summary, close the screen and write three lines. If any line will not come, go back to the original for that part only.

If you want to review how your team approaches AI training, you can reach us through the contact page.Contact

Sources

Next: DMS Generative AI Training (in Korean)

An open weekly planner on a wooden desk by a window, every box empty, with a single pencil resting across the page. A closed laptop sits at one edge, with a mug and a green plant behind. Soft morning light falls across the desk.
Generated image. A desk scene made for illustration, not a record of a real event.View full image ↗

Essay / 2026. 11. 7

Where the Saved Time Goes

Between the hours a tool gives back and the day that remains

If AI makes work faster, does the time it saves become rest? Following an eight-month observation of one technology company and a randomized experiment with 7,137 workers at 66 large firms, this essay looks at where saved time can go and suggests three things a workplace can write down.

Whether saved time turns into rest on its own depends on which study you read. In an eight-month observation of one U.S. technology company, employees worked faster, took on a wider range of tasks and stretched work into more hours of the day. In a randomized experiment with 7,137 workers at 66 large firms, people who used the AI tool spent about two fewer hours a week on email and a little less time in work apps outside regular hours, yet the number and kind of tasks they completed did not visibly change. The methods and tools differ, so the two do not refute each other. My reading is that a workplace should write down where the saved time went.

Based on sources checked on November 7, 2026. I read the full text of Ranganathan and Ye's Harvard Business Review article of February 9, 2026, which the authors describe as in-progress research and which is not a journal paper. I read the full November 2025 revision of the NBER working paper by Dillon and colleagues (2025); its cover states that it has not been peer-reviewed. The training company scene near the end is a hypothetical for illustration, not a DMS client case. The suggestions at the end are my own and are not procedures defined by these studies.

Does faster work mean fewer working hours?

In an eight-month study of one technology company, it did not. Aruna Ranganathan and Xingqi Maggie Ye followed a U.S. company of about 200 employees from April to December 2025. They observed work in person two days a week, tracked internal communication channels and held more than 40 in-depth interviews across engineering, product, design, research and operations. The company did not mandate AI use and only offered enterprise subscriptions to commercially available tools. The researchers report that employees worked at a faster pace, took on a broader scope of tasks and extended work into more hours of the day, often without being asked.

The change took three forms. The first was task expansion. Once AI could fill gaps in knowledge, product managers and designers began writing code, researchers took on engineering tasks, and people tried work they would once have handed off or put off. Engineers, in turn, spent more time reviewing AI-assisted code from colleagues and finishing partly complete pull requests, which are requests to merge code changes.

The second was a blurring of work and breaks. Because AI made starting a task easy, employees sent prompts at lunch, during meetings and while waiting for a file to load. Some sent a “quick last prompt” before leaving their desks so the AI could keep working. These moments rarely felt like extra work, but over time the day had fewer natural pauses. Some workers realized, often in hindsight, that once prompting during breaks became a habit, downtime no longer restored them in the same way.

The third was running several things at once: writing code by hand while AI produced an alternative version, running multiple agents in parallel, and reviving long-deferred tasks because AI could “handle them” in the background. Having a partner gave a feeling of momentum. In practice, the researchers describe constant switching of attention, frequent checking of AI output and a growing number of open tasks.

One engineer's words appear in the article: “You had thought that maybe, oh, because you could be more productive with AI, then you save some time, you can work less. But then really, you don’t work less. You just work the same amount or even more.” The authors see a self-reinforcing loop. AI speeds up some tasks, which raises expectations for speed; faster work makes people lean on AI more; heavier reliance widens what they attempt. Several participants said they felt more productive but no less busy, and some felt busier than before.

How far can this observation travel?

Not far without care, because it is a qualitative study of one company in one period. The article gives no figure for how many hours of work were added. The authors call it in-progress research, and it appeared in a management magazine rather than a journal. The company let employees use generative AI tools widely, so this piece alone cannot say the same pattern would appear where AI is rarely used or in other industries.

The authors propose what they call an “AI practice”: intentional pauses, sequencing of work, and protected time for human conversation. That is their recommendation, not something this study tested.

A small round wooden table in an office break area around midday. A half-eaten sandwich sits on a paper wrapper beside a glass of water and a smartphone propped against a book. A sheer curtain and a green plant are softly out of focus behind.
Generated image. A break-area scene made for illustration, not a record of a real event.View full image ↗

What happens when you measure the hours?

Email time fell, while the amount and kind of work stayed about the same. Eleanor Dillon, Sonia Jaffe and Nicole Immorlica of Microsoft Research and Christopher Stanton of Harvard Business School ran a field experiment at 66 large firms between September 2023 and October 2024. Of 7,137 participating workers, 3,684 were randomly chosen to get Microsoft 365 Copilot, built into the programs they already used for email, meetings and writing. The rest worked as before, and workers decided for themselves whether to use the tool. The researchers analyzed usage records from Microsoft products. They could not see the content of anyone's work or any measure of performance.

The main analysis centers on the period at least 12 weeks after assignment, once use had settled. Use peaked at 55% of treated workers in the first weeks, then settled at a weekly rate just under 40%. Averaged over everyone offered the tool, time in Outlook fell by 1.4 hours a week, or 12%. Estimated for workers who actually used it, the drop is about two hours, or 17%. Email work was also consolidated into fewer stretches, opening about 1.5 extra hours a week without Outlook activity. Average time in video meetings and in writing documents did not change significantly.

The amount and kind of work did not change either. Treated and control workers replied to the same number of email threads, attended the same number of Teams meetings and completed the same number of Word documents. Time spent in digital applications outside regular working hours fell by about 0.25 hours a week, or 9%, which the authors read as a sign that some of the time saved on email returned to the workers.

The paper has its limits. It observes only activity inside Office programs and has no productivity or performance measure. The participating firms were large companies willing to adopt new tools early, and the authors caution that the workers may not represent knowledge workers more broadly. Three of the four authors are at Microsoft Research, and the tool studied is a Microsoft product. Because the tool was given to individuals, the authors conclude that team-level and firm-level changes need further research.

What shows up when the two studies sit side by side?

The two look at different places, so they do not contradict each other. The first is a long record of what people did and how they felt inside one company. The second is a comparison of usage logs between people who got the tool and people who did not. The tools differed too. In the first company, employees used commercial AI tools widely, including for coding and running agents. The experiment studied one tool embedded in email, meetings and documents, and measured only activity there. So “the saved time went into more work” and “email time fell” can describe two sides of the same workplace. These two sources cannot settle which is more common.

Both studies show that the hours that disappeared went somewhere, and the tool did not decide where. In one study the hours reappeared as new responsibilities and prompts slipped into breaks. In the other, part of the time showed up as less use outside working hours, and the records do not say where the rest went. That is my reading.

How could a workplace handle the time it saves?

One way is to decide where the time goes first and write down where it went afterwards. From here on, these are DMS.Labs' own suggestions and assumptions, not procedures from either study, and their effect has not been tested. Suppose a small training company with six staff starts drafting replies to customer inquiries with AI on a Monday. A month later someone says replies are faster. Whether the saved time went into revising course material, into extra drafts at lunch or back into fixing the AI's drafts cannot be known without a record.

I suggest keeping three lines. First, for two weeks before adoption, record how long the task takes and how often it is done outside working hours. Second, on the first day of use, write one line: this saved time goes to such-and-such, and the rest is left empty. Third, after a month, put new tasks, time spent checking and correcting AI output, and lunch-hour or evening use in the same table and compare it with the start. There is no evidence yet that this method reduces work. Its purpose is to make it possible to see where the time went.

The measurement side can be read alongside the DMS guide to measuring AI adoption, which sets task baselines and review costs first. This essay only suggests recording where saved time goes. It does not report that DMS training or this method has reduced anyone's time.Measuring AI adoption: revenue, costs and review time

This week, pick one task you often do with AI and write one line on where the time it saved went over the past seven days. If there is nothing to write, you have learned that no record existed.

If you want to review how work flows after introducing AI, you can reach DMS.Labs through the contact page.Contact

Sources

Next: DMS Workflow Automation (in Korean)

A closed laptop pushed to one side of a wooden desk by a window, with an open notebook and a pencil in front of it. Handwritten lines of calculation are faintly visible on the page. A mug and a green plant sit behind, and warm morning light falls across the desk.
Generated image. A desk scene made for illustration, not a record of a real event.View full image ↗

Essay / 2026. 11. 4

A Day Without the Assistant

Between the score a tool helped produce and the skill you keep

If your score went up with AI's help, did your own ability go up too? Following an experiment with about 1,000 high school students and an observational study of 19 endoscopists, this essay looks at how performance with help and skill without it can part ways, and suggests ways to measure them separately at work.

A higher score with AI's help does not show that your own ability has risen. In an experiment with about 1,000 high school students, those who practiced with GPT-4 did 48% better on practice problems, but on an exam taken without AI they scored 17% lower than students who had practiced without it. In an observational study of 19 endoscopists in Poland, the adenoma detection rate in colonoscopies done without AI fell from 28.4% to 22.4% after the clinics began using an AI polyp-detection tool. Both findings come from narrow settings, so they cannot be carried over directly to office documents or automation.

Based on sources checked on November 4, 2026. For Bastani et al. (2025, PNAS) I read the methods, results and limitations in the full text. For Budzyń et al. (2025, The Lancet Gastroenterology & Hepatology) I read the abstract, a journal-society summary and news coverage, not the full text. The record shows a correction to that paper, and I did not check what was corrected. The new hire scene is a hypothetical for illustration, not a DMS client case. The suggestions near the end are my own and are not procedures defined by these studies.

Does a higher practice score mean a higher exam score?

Hamsa Bastani, Osbert Bastani, Alp Sungu and colleagues published a field experiment in PNAS in 2025, run at a high school in Turkey. About fifty classes in grades 9 to 11, nearly 1,000 students in all, took part in four 90-minute sessions. Each session had three parts: the teacher reviewed an earlier topic, students worked through practice problems, and then everyone sat a closed-book, closed-laptop exam. Only the middle part, the practice period, was randomized.

Students were split into three groups. The control group practiced without AI. The GPT Base group practiced with a GPT-4 interface that looked like ChatGPT. The GPT Tutor group used a GPT-4 interface designed with teachers' input to give hints instead of handing over answers. On the practice problems, the GPT Base group scored 48% higher than the control group and the GPT Tutor group 127% higher. On the exam without AI, the GPT Base group scored 17% lower than the control group, a drop of 0.054 on a one-point scale. In the GPT Tutor group that drop had largely disappeared, though the group did not beat the control group either.

What did the students do with the AI?

The chat logs offer a clue. Many GPT Base students asked for a full solution and copied it. The authors describe this as using the tool as a 'crutch.' GPT Tutor students were more likely to ask for help or to try an answer themselves first. The underlying model was the same GPT-4. The interface and instructions differed, and so did the way students used it and the exam results.

What students felt did not match what happened. The GPT Base group did not think they had performed worse or learned less. The GPT Tutor group felt they had done significantly better, even though their exam scores were no higher than the control group's. The authors note that the study covers one school and short-term outcomes only, and that other tutor designs and settings need further study. Whether the same gap remains months later cannot be read from the paper.

Did something similar show up in working adults?

A study from medicine is worth a second look. Krzysztof Budzyń and colleagues published data from four endoscopy centres in Poland in The Lancet Gastroenterology & Hepatology in August 2025. The centres introduced an AI polyp-detection tool at the end of 2021, and afterwards colonoscopies were randomly assigned, by examination date, to be done with or without AI. The researchers compared standard colonoscopies done without AI in the three months before the tool arrived and the three months after. The sample was 1,443 patients: 795 before, 648 after.

The main measure was the adenoma detection rate. An adenoma is a type of polyp that can develop into colorectal cancer, and the rate is the share of colonoscopies that found at least one. For examinations done without AI, it fell from 28.4% (226 of 795) before to 22.4% (145 of 648) after. The difference was 6.0 percentage points (95% CI -10.5 to -1.6; p = 0.0089). In a regression that also accounted for factors such as patient sex and age, exposure to AI had an odds ratio of 0.69 (0.53 to 0.89).

A few more details. There were 19 endoscopists, reported to have performed more than 2,000 colonoscopies each, and 15 of them showed a numerical decrease. The 734 examinations done with AI had a detection rate of 25.3%, which is not higher than the 28.4% before the tool arrived. My reading is that figures from the days with help cannot, on their own, show how the clinician's own eye has changed.

This was a retrospective observational study. It compared the same centres before and after, rather than assigning doctors to different levels of AI exposure, so it is hard to say that AI exposure caused the drop. The authors themselves wrote that continuous exposure 'might' reduce the rate. Douglas Rex, who reviewed the paper for the ASGE journal scan, called it the first large-scale formal evaluation of whether AI causes physician deskilling, described the results as very concerning, and added that more studies are warranted. The task here was finding polyps by eye, not writing documents or answering customers.

A pair of training wheels removed from a child's bicycle lies on a wooden workbench next to a wrench. Soft natural light comes from a window behind, and a rag and a jar of screws are blurred in the background.
Generated image. A workbench scene made for illustration, not a record of a real event.View full image ↗

What shows up when the two studies sit side by side?

A practice score and an exam score measure different things, and so do a detection rate on days with AI and on days without. The first of each pair is what a person and a tool achieve together. The second is what remains with the person once the tool is taken away. In these two studies the pairs did not move together, and the high school students did not feel the difference. If feeling cannot confirm it, the two have to be measured separately. That reading is mine.

The design of the tool deserves a look too. With the same GPT-4, the version built to give hints instead of answers largely avoided the exam drop. How the AI helps may matter as much as whether it is used at all.

How could you measure the two separately at work?

What follows are my suggestions, not procedures validated by research. Picture a scene. A new hire who answers product inquiries has spent three months editing AI-written draft replies before sending them. One day the system goes down, and the replies have to be written without AI. The sticking point could be the terminology, the tone, or knowing where to start checking. You only see where on the day it happens. The scene is hypothetical and predicts no result.

First, set aside one day a month to work without AI. Pick one small task you normally do with AI, do it without, and write two or three lines on how long it took and where you got stuck. The aim is to find the sticking point, not to grade anyone. The exam played that role in the high school study, and most workplaces have no equivalent.

Three notes are enough for the record: how long it took, where you got stuck, and whether anything is missing compared with how it usually goes with AI. If you get stuck at the same step repeatedly, mark that step as something a person should learn once before handing it to AI. If you did not get stuck, that is one piece of evidence that the task can be handed over. A single record does not judge anyone's ability. What you watch for is the pattern that builds up over repeated records.

Second, separate the tasks a person needs to learn from the tasks that only need to be finished. If the goal is just to get the work done, handing it to AI is fine. If a person needs to learn the task, set the tool to give hints before answers. That was the GPT Tutor design, and it largely removed the exam drop, though it did not beat the control group, which is worth remembering too. I wrote earlier about how to decide which tasks to hand over and which a person should learn.AI training should start with 'Should I hand this task over?'

Third, for work where judgment stays with a person, check accuracy without AI now and then. What the endoscopy study suggests is that numbers recorded with AI on hand do not show how a person's own eye is changing. I do not know how often such checks should happen. My own view is that higher-stakes work calls for shorter intervals.

This week, pick one task you often do with AI and do 15 minutes of it without. Writing down the first step where you got stuck is enough for a first record.

If you are thinking about AI training or applying AI to your work, you can reach us through the contact page.Contact

Sources

Next: DMS Generative AI Education (in Korean)

On a wooden desk by a window, an open notebook with nothing written in it has a pencil resting across the page. To the right lies a printed page of text, and a cup, a plant and a few books sit at the back. Warm natural light falls on the desk.
Generated image. An illustrative desk scene, not a real record.View full image ↗

Essay / 2026. 10. 31

When the Draft Arrives First

Where your own view sits when you edit what AI wrote first

If you edit a draft that AI wrote first, do you end up with the piece you would have written yourself? Following five writing experiments, this essay looks at how a first draft tilts wording and opinion, how much of your own voice returns while you revise, and how to change the order of work.

If you take a draft that AI wrote first and edit it, do you end up with the same piece you would have written yourself? Experiments suggest not necessarily. Writers who received AI ideas produced stories that raters scored higher on average, but those stories resembled one another more. In other experiments, the opinions in people's writing and in their later survey answers moved toward the assistant's position. Most of this evidence comes from short writing tasks, and a recent preprint finds that the outcome depends on how people are rewarded for revising.

This essay relies on sources checked on October 31, 2026. I read the abstract and parts of the full text for Doshi and Hauser (2024) and Padmakumar and He (2024), and only the abstracts for Jakesch et al. (2023) and Williams-Ceci et al. (2026). The Jo and Raghavan paper is an arXiv preprint, and I could not confirm that it has been peer reviewed. The office scene is a hypothetical for illustration, not a DMS client case. The suggestions near the end are my own ideas, not procedures set by these studies.

How does a first draft change the writing?

Writers who got AI ideas scored higher on average, but their stories were more alike. Anil Doshi and Oliver Hauser published an online experiment in Science Advances in 2024. A total of 293 writers each wrote an eight-sentence story for a teenage and young adult audience. They were split into three conditions: no help, the option to get one story idea from a generative AI model, or the option to get up to five. Separately, 600 evaluators read the stories without being told which condition each came from.

Stories written with AI ideas were rated as more creative, better written and more enjoyable. With up to five ideas, novelty was 8.1% higher and usefulness 9.0% higher than in the no-help group. Usefulness here is an index built from items such as fit with the target audience and whether the story could be developed into a book. The gains were larger for writers who scored low on a creativity test: in the five-idea condition, novelty rose 10.7% and usefulness 11.5%. Among the writers who could get AI ideas, 88.4% asked for at least one.

There was a cost. Stories written with AI ideas were more similar to one another than stories written alone. The increase in similarity amounted to 10.7% of the range of similarity scores among no-help stories in the one-idea condition and 8.9% in the five-idea condition. The authors compare this to a social dilemma. Each writer is better off, but if everyone uses the same tool, the range of new content narrows. The study covered one kind of short story and relied on online raters.

Vishakh Padmakumar and He He looked at argumentative essays in a 2024 ICLR paper. Writers recruited on Upwork, all native English speakers with writing or copyediting experience, answered prompts taken from the New York Times student opinion series. They worked in one of three conditions: alone, with a base language model (GPT-3), or with a model tuned on human feedback (InstructGPT). Only the InstructGPT condition made different authors' essays more alike and reduced lexical and content diversity. The authors trace this mainly to the text the model contributed, which was less diverse. The text the writers contributed themselves was not affected. These were the models available to the study, so the paper alone cannot tell us whether the same holds for the models people use today.

Does it carry over to opinions?

In two experiments it did: people's writing and survey answers moved toward the assistant's position. At CHI 2023, Maurice Jakesch and colleagues asked 1,506 participants to write a post on whether social media is good for society. The treatment group used a writing assistant built on a language model configured to argue that it is good, or that it is bad. The opinions in participants' writing leaned toward the assistant's side, and so did their answers on a later attitude survey. Independent judges (500 of them) rated the opinions expressed in the writing.

A 2026 follow-up in Science Advances, from several of the same researchers, ran two preregistered experiments with 2,582 participants. People wrote about societal issues while a biased assistant offered autocomplete suggestions, and the attitudes they reported afterward converged toward the AI's position. Most participants were unaware that the suggestions were biased or that they had been influenced. The influence was stronger than when similar content was shown as static text. The authors note that they measured short-term effects and that it is unclear how long those effects last.

Does knowing about the bias protect you?

According to the same paper, warning participants about the assistant's bias, before or after they wrote, did not reduce the shift. Carried over to the workplace, this suggests that a reminder to read AI drafts carefully may not be enough by itself. That reading is mine. The experiments dealt with opinions on social issues, and these papers do not test whether the wording or positions in business documents shift in the same way.

Can your own voice come back while you revise?

Partly: in one preprint, writers drifted away from the AI's draft as they edited, but did not fully recover their diversity. A preprint by Nathanael Jo and Manish Raghavan (arXiv, first posted in April 2026 and revised on October 28) paints a more mixed picture. In two preregistered randomized trials, participants wrote while chatting freely with an AI assistant, OpenAI's GPT5-Mini. Study 1 had 200 participants writing a short story of 250 to 350 words, and Study 2 had 400 writing a product pitch of 100 to 125 words. Of the participants, 91% in Study 1 and 83% in Study 2 asked for at least one complete draft.

Over the session, the authors report, participants' writing moved away from the initial AI drafts and recovered much, though not all, of the diversity those drafts had suppressed. The incentives differed too. In one condition, tiered bonuses went to the most original submissions relative to peers, covering roughly the top 2% to 25%. In the other, bonuses depended on meeting a technical quality rubric. Writing rewarded for originality was more diverse. Those participants kept using the AI but copied less of its text verbatim. The effect was clear in Study 2 and pointed the same way, without statistical significance, in the smaller Study 1. Even with the originality bonus, AI-assisted writing was not as diverse as writing done alone. The bonuses are an experimental device and do not stand for a pay or review system at a real company.

What does this look like at work?

Picture a made-up training company with six employees. A staff member has AI write the first version of an introduction to send to a new corporate client. The draft is smooth and contains no errors. The staff member fixes a few awkward sentences and sends it. If another company in the same field had used the same tool, the two introductions could be close enough to swap by changing the company name. The cases this company has actually taught, the points where its trainees tend to get stuck, and the group sizes and schedules it can handle were never in the draft, so the editing never brings them up. This scene is a hypothetical drawn from the direction of the studies above. I have no figure for how alike such documents would be.

On a printed page, a hand draws a pencil line through several lines of text, and a few short pencil marks that are not legible slant across the margin. The edge of a cup shows in the corner, and soft natural light falls on the paper and the hand.
Generated image. An illustrative desk scene, not a real record.View full image ↗

How could you work with AI instead?

These are suggestions of mine, not procedures any study has validated. The aim is to change the order of work so that a draft does not settle in before your own thinking does.

First, write your own point down before opening the AI. Two sentences are enough: what the document must say and what the reader needs to know. Once a draft arrives, it becomes the reference point for every later judgment. The Jakesch and Williams-Ceci experiments show suggestions received during writing moving opinions, and this order is an attempt to keep the reference point on your side. I could not find evidence on how much it helps.

Second, do not settle for a single draft. Ask separately for two or three versions with different positions or structures, and if none fits, return to your two sentences. In the Doshi and Hauser study, the five-idea condition still produced stories more alike than the no-help condition (8.9% of the range, against 10.7% with one idea), so more options do not remove the problem by themselves. Comparing versions at least makes the first one visible as one choice among several.

Third, add one question to review: would this paragraph still make sense in another company's version of the same document? If it would, fill it with something only you have, such as a case, a number, a condition, or something you have actually done. This borrows from the originality-bonus result in Jo and Raghavan. The review then asks for more than correctness: does the paragraph say something only we could say? No study has tested this procedure itself.

Fourth, mark the passages in the final version that remain in the AI's wording. I know of no standard ratio. Marking lets the reviewer read those passages with extra care for facts and positions, and it shows whether the same phrasing keeps appearing across documents. I covered the record-keeping side in an earlier article.Does an AI-Made Product Have Copyright?

Fifth, build it into the process rather than relying on warnings. If telling people about a bias did not reduce its effect, a note saying "be careful" is a weak safeguard. The four steps above are all written as order of work or review items for that reason.

Our Work article on deciding whether to accept an AI's output covers the testing side.Acceptance Tests for an AI Skill

What could you try this week?

Pick one document you made with AI this week. Check whether you wrote down your own main point before opening the AI, and compare it with the main point of the final version. If they differ, ask whether the difference came from something new you learned or from following the draft's direction. If you wrote nothing down, starting the next document with two sentences is enough.

If you are thinking about AI training or applying AI to your work, you can reach us through the contact page.Contact

Sources

Next: DMS Generative AI Education (in Korean)

A long row of identical blank paper slips on a wooden desk, with a hand holding a small rubber stamp paused above them. An ink pad sits nearby and soft window light falls across the desk and paper.
Generated image. A staged desk scene, not a record of real work.View full image ↗

Essay / 2026. 10. 27

When Approval Prompts Pile Up

How checking gets thinner when automation keeps asking

When an automated tool asks permission at every step, the person answering may end up checking less, not more. Following hospital alarm research and data on AI agent approvals, this essay looks at how the number of questions and the quality of checking come apart, and proposes asking fewer questions and making the remaining ones readable.

If an automated tool asks permission at every step, does the person on the other end check each step more carefully? Not necessarily. Reports from hospital alarms and from AI agent approval prompts point the same way: the more often people are asked, the less attention each question gets, and the prompt slowly stops working as a safeguard. The studies disagree about why this happens.

This essay is based on sources as of October 27, 2026. I read the abstracts of Cvach (2012) and Ancker et al. (2017), not the full papers. The figures on AI agents come from two public posts by Anthropic and from the tallies of a browser game built by Alex Wauters. The Anthropic numbers appear in posts that describe its own product and have not been independently verified. The office scene below is a made-up illustration, not a DMS client case. The design suggestions near the end are my own, not standards set by these sources.

What happened in hospitals when alarms multiplied?

A bedside monitor sounds an alarm when a patient's readings leave a set range, and a nurse is expected to respond. When alarms come too often, the ones that matter can get buried. In a review published in Biomedical Instrumentation & Technology in 2012, Maria Cvach called alarm fatigue the number one medical device technology hazard of that year. The review synthesizes 72 research and non-research articles published between January 2000 and October 2011.

The abstract says desensitization to alarms is multifaceted. It ties the problem to a high false alarm rate, poor positive predictive value (how often an alarm points to a real problem), a lack of alarm standardization, and the number of alarming devices in hospitals. The non-research evidence falls into two themes: strategies to reduce desensitization, and alarm priority and notification systems.

Is it the workload, or the same alert showing up again?

Jessica Ancker and colleagues asked exactly this in a 2017 paper in BMC Medical Informatics and Decision Making. It is a retrospective cohort study of electronic health record data from 112 ambulatory primary care clinicians, covering January 2010 through June 2013, with both drug alerts and clinical practice reminders. On average, one quarter of the drug alerts a clinician received, and one third of the reminders, were repeats for the same patient within the same year.

The results cut against a simple story of overwork. Acceptance was linked to work complexity and to repeated alerts, but not to the amount of work. The likelihood of accepting a reminder dropped by 30% for each additional reminder received per encounter, and by 10% for each five percentage point increase in the share of repeated reminders. Newly deployed reminders did not show acceptance declining over time, which is what desensitization would predict. The authors conclude that clinicians became less likely to accept alerts as they received more of them, particularly repeated ones, and suggest that cutting within-patient repeats is a promising target.

It would be a mistake to read this as people getting lazy. The abstract does not say whether the declined alerts were wrong or the accepted ones were right. What I can say from what I read is that acceptance fell as alerts grew more numerous and more repetitive.

Does the same thing happen with AI agent approvals?

According to Anthropic, its coding agent Claude Code started out allowing reads and requiring approval for writes, shell commands and network access. In a post published on May 25, 2026, Anthropic reported that users approved roughly 93% of permission prompts, and wrote that the more approvals a user sees, the less attention they pay to each one. A post from August 7 gives the figure as 97%. I could not confirm from either post whether the two numbers were measured the same way.

The August post also contains a contrast. When Claude presents a plan for approval, users reject 39% of them, while the rejection rate for individual permission requests is 3%. One reading is that people stop to think when shown the whole picture and stop stopping when small questions arrive in a stream. That reading is mine, and the post does not say whether the two kinds of request carried the same level of risk.

Anthropic also ran an experiment with 1,053 paid testers. Partway through each session, one permission prompt was swapped for a clearly dangerous command (approving it never ran anything dangerous). The testers blocked it 13.6% of the time, 143 of 1,053. Early in a session they blocked about 17% of dangerous commands, and after 50 or more prior prompts about 5%. The post reporting this argues for making Anthropic's own automatic approval feature the default, and it states that participants knew they were being evaluated and worked in a test environment rather than on their own projects. I would not carry those percentages over to a real workplace.

A source with no tie to Anthropic is a browser game by Belgian developer Alex Wauters, in which players approve or deny simulated permission requests for 60 seconds. According to his write-up, it has been played more than 40,000 times, producing about 409,000 decisions, and the average player missed one in three threats. The most-missed command, npm run analyze, was approved by about two thirds of players. Anyone could play and the clock was short, so this says little about the rate at an actual workplace.

Why does asking feel safe?

I think asking feels safe because responsibility looks clear. Once a person has clicked, the record shows that a check took place. In the IANS article about the game, security expert George Gerchow said human-in-the-loop was never really a control and was closer to a checkbox. That is one expert's opinion, not evidence for the claim.

Picture an imaginary estimates office with seven staff. They add an automation that drafts and sends quotation emails, and it asks 'Proceed?' at every step: recipient, attachment, send, save, file. In the first week the person in charge reads the recipient line and the attachment before clicking. After a few weeks the same window appears dozens of times a day, and nearly every one has ended fine. The window stops being something to read and becomes something to get past. One day a quotation goes out with a wrong line in the recipient field, and the approval log shows an ordinary click. This scene illustrates the direction the sources point in. The 'dozens' and the 'few weeks' are assumptions with no data behind them.

A single brass key hangs on a wooden board, with a few folded blank sheets of paper on a wall shelf below. A potted plant sits by the window and soft light falls across the wall.
Generated image. A staged scene, not a real office.View full image ↗

What would better asking look like?

What follows is my proposal, not a procedure validated by research. The aim is to ask fewer questions and make the remaining ones readable.

First, sort the steps by how much could be lost. Do not ask before steps that only read. Log steps that can be undone. Stop a person only before steps that are hard to take back, such as external sending, deletion and payment. Anthropic says that adding a sandbox to Claude Code, one that allows writes inside the workspace and denies network access by default, cut permission prompts by 84%. Tightening the boundary first can remove the need to ask.

Second, ask about the plan, not each step. The 39% and 3% above do not come from a controlled comparison, but they are a clue that people pause more when they see the whole. In the estimates office, instead of asking 'Send this email?' twelve times, ask once: 'Today twelve quotations go to these recipients at this time.' The twelve is only an example.

Third, put what changes into the question. 'Run it?' gives the reader nothing to check. If the sentence says who receives what and how many go out, a person has something to find fault with. The Ancker study found that repeated alerts were especially likely to be passed over, so I would also avoid building a prompt that repeats itself.

Fourth, make a wrong click cost little by narrowing what the tool can reach. Anthropic's May post separates supervising what an agent does from limiting what it is able to do, and says a tight perimeter lets you relax oversight. The same post says its auto mode catches roughly 83% of overeager behaviors before they execute, and adds that any probabilistic defense has a non-zero miss rate. Handing out access in small pieces is needed whatever the approval prompt looks like.

Fifth, count approvals. A prompt that is approved nearly 100% of the time has blocked almost nothing, so it is worth asking whether it needs to exist. That does not mean there is no risk. Where such a prompt stays, read a few of last week's approved items at random afterward. How many to read is an assumption, sized to what a person can read, with no research figure behind it.

How to split permissions into steps is covered in a Work piece.Giving AI automation permission in three steps

What to count this week

If you use an automation or AI tool today, count over one week how many times a person clicked yes and how many times no. Finding the prompts that never got a no shows where human attention is being spent, and where it is only being used up.

If you are considering automation for your own work, you can reach us through the contact page.Contact

Sources

Next: DMS Business Automation (in Korean)

A hand-pressed metal tally counter and a neat stack of blank paper slips on a wooden desk. The top slip has one corner folded halfway, and soft window light falls across the counter's surface.
Generated image. A desk scene made for illustration, not a record of a real place.View full image ↗

Essay / 2026. 10. 24

When the Number Becomes the Goal

The gap that opens when one figure stands for automation results

If you measure automation by one number and ask a team to raise it, the number can climb while the work gets worse. Once a measure becomes a target, it tends to lose its power to say whether the work is going well. The number is worth keeping, with what it misses placed beside it.

Does raising one automation metric make the work better? Not always. Once a measure becomes a target, it tends to lose its power to say whether the work is going well, so the figure can climb while the work gets worse.

The economist Charles Goodhart made an observation in 1975 that Manheim and Garrabrant quote this way: any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes. In 1997 the anthropologist Marilyn Strathern put the idea into a single sentence: "When a measure becomes a target, it ceases to be a good measure." Her example was the 2.1 mark in British university examinations. The more a 2.1 becomes an expectation, she wrote, the poorer it becomes as a discriminator of individual performances. In the same passage she notes that Hoskin calls this 'Goodhart's law'.

The scenes in this essay are assumptions made for illustration. They are not DMS client cases or measured results. I based it on Strathern's openly available 1997 paper, the paper by Manheim and Garrabrant, and the abstract of the paper by Gao and colleagues, all checked on October 24, 2026, and the essay reflects them as of that date. I read only the abstract of the Gao paper and did not read Goodhart's original text, so his wording reaches me through Manheim and Garrabrant and through Rodamar's 2018 summary. The practices suggested near the end are my own proposals, not standards set by these studies.

A day when the count goes up

Picture an imaginary office that handles customer inquiries. An AI sorting tool has been added, and the team's result is defined as the number of inquiries closed each day. At first the figure works well. The faster inquiries are handled, the higher the count, and customers generally get answers sooner too.

Once the count becomes the team's goal, behavior shifts a little at a time. A reply that says only 'received' can close an inquiry. One person's question can be split in two and closed twice, which doubles the count. Slow cases drift to the back and quick ones move up. None of this needs bad intent. When the easiest way to raise the count and the way to resolve the inquiry part ways, the people judged by the count will see the first road more clearly.

I think there is a reason numbers end up in the goal's seat. They are easy to compare, easy to report, and easy to hand to a tool. When someone asks what changed after automation arrived, a count is the quickest answer to give. Preventing the slide takes some thought on the day the number is chosen.

The abstract of Strathern's paper widens the picture a little. Audit, it says, does more than monitor: it has a life of its own that jeopardizes the life it audits. I read that as saying that once a number enters a reporting system, matching the number becomes a job in its own right. That reading is mine, and I did not draw on anything beyond the abstract for it.

How does the gap open?

It opens in several ways, and the literature has sorted at least four. Manheim and Garrabrant define a Goodhart effect as optimization collapsing the statistical relationship between the goal an optimizer intends and the proxy used for it, and they split the causes into four kinds. They add that their categories do not exactly match what was originally discussed. The office scenes below are my own translation of each kind into an assumed setting.

The regressional kind appears when you pick the top of an imperfect proxy. You select not only people who did the job well but also the gap between the proxy and the goal. The agent with the highest count may owe part of it to skill and part to a run of easy inquiries. The paper calls this the most fundamental Goodhart effect and says it cannot be avoided: whatever measure is chosen, an inexact metric diverges from the goal in the tail.

The extremal kind appears when the proxy takes a value far from the usual. A relationship between proxy and goal that held in the normal range need not hold in the region you reach by pushing the proxy to an extreme. Demanding a daily count far above the usual comes close to this.

The causal kind appears when you act on the proxy directly. The link that let the count stand in for resolved inquiries is altered by the very actions meant to raise the count. Splitting inquiries before closing them is an example. The paper offers a classroom parallel. When two tests are correlated and a teacher trains general test-taking skill, one of the traits behind good scores on both tests changes, and the correlation between the two scores is likely to fall.

The adversarial kind involves an agent whose goal differs from the evaluator's and who breaks the relationship. If matching the number pays off, a person or a tool can find that path.

A brass kitchen timer and a few loose cards on a pale wooden counter. The top card carries a single pencil tick, and a ceramic cup of cold tea sits beside it.
Generated image. A desk scene made for illustration, not a record of a real place.View full image ↗

Does using AI widen the gap?

One paper argues that it may. Manheim and Garrabrant write that the importance of Goodhart effects depends on the amount of power directed at optimizing the proxy. Because artificial intelligence increases that power, they judge the problem especially critical in that field.

The 2022 paper by Gao, Schulman, and Hilton measures the problem in model training. A reward model trained to predict human preferences is an imperfect proxy. According to the abstract, optimizing its value too much can hinder ground-truth performance, in line with Goodhart's law. The authors use a synthetic setup in which a fixed gold-standard reward model plays the role of humans, compare reinforcement learning with best-of-n sampling, and report that the relationship follows a different functional form for each method, with coefficients that scale smoothly with the number of reward model parameters.

That experiment concerns training a model, which is a different layer from using an AI tool in an office. The structure still looks similar. If a tool is given one number as its goal, it may move toward raising that number, and the gap between number and work may widen just as fast. This link is my own reading, not a conclusion the paper draws about workplace tools.

Should we stop using numbers?

No. The problem Manheim and Garrabrant point to is a metric that could help improve a system being used past the point where pushing it further is ineffective or harmful. Counts and handling times are still useful for seeing the work. Their usefulness shrinks when a single number starts standing in for the goal and the reward together.

What to put beside the number

What follows is a proposal from this essay, not a procedure any study has tested. On the day a number is chosen, write down the easiest bad way to raise it first. If entries such as closing after only a receipt, or splitting inquiries, show up, that is a sign you need a second number to guard against them. For example, put the count of reopened inquiries next to the count of closed ones.

Apply that to the imaginary office. The goal is for inquiries to be resolved, and the closed count is the proxy that stands in for it. Closing after only a receipt and splitting inquiries came up as easy bad methods, so the paired number is the count of reopened inquiries, and each week five closed inquiries are opened and read to see whether the customer's problem was actually solved. The number five has no evidence behind it. It is an assumed amount that a person can read.

On reporting days, open a few of the underlying records and read them. What the number missed only shows up in the records. This will not close the gap between number and work. It gives you a chance to notice early that the gap is widening.

A separate Work article sets out how to read adoption results split into revenue, costs, and review time.Measuring AI adoption: revenue, costs and review time

One question before the meeting

The next time I sit down to choose an automation number, I want to ask first whether the easiest way to raise it is to do the work well. If the answer is no, I will keep the number but not in the goal's seat, and I will decide together what number sits beside it and which records get read.

How to set a baseline when choosing a first task to automate is covered in this earlier piece.Choosing the First Task to Automate

Sources

Next: DMS AX transformation design (in Korean)

An open notebook and a pencil on a wooden desk. The notebook holds a sketch of a zipper with arrows that stop partway, and a canvas bag with an open zipper sits beside it.
Generated image. A desk scene made for illustration, not a record of a real place.View full image ↗

Essay / 2026. 10. 21

What Explaining Reveals

A small test to run before you hand work to automation

Before handing a task to automation or AI, ask the person who has been doing it to explain why each step exists. The places where the explanation stalls are the places nobody understands yet, and they are worth asking about before any tool is attached.

Before you automate a task or hand it to an AI, ask the person who has been doing it to explain, out loud, why each step is there. The points where the explanation stalls are the points nobody understands yet. Those are the questions to ask before a tool is attached. The feeling of knowing does not reveal them very well.

The longer someone has done a task, the earlier their hands move. They open the report, copy the figures across, and send it to the right person without deciding who that is. At that moment they feel they know the work. But what the hands know and what the person can explain may not be the same thing. The difference rarely matters day to day. It shows up when the work is handed to someone else, or to a machine.

The scenes in this essay are assumptions made for illustration. They are not DMS client cases or measured results. My starting point was the abstract of Rozenblit and Keil (2002) and parts of the openly available manuscript, which I read on October 21, 2026. I did not read the tables or the supplementary material. The sequence I suggest for applying this to work is my own proposal, not a standard set by that study.

People who think they understand a zipper

Researchers at Yale looked at how well people feel they understand everyday objects. Participants first rated their own understanding. They then wrote out, step by step, how the object works, and finally rated themselves again. The items in the paper include a speedometer, a zipper, a piano key, a flush toilet, a cylinder lock, a helicopter, a quartz watch, and a sewing machine.

The abstract says that people feel they understand complex phenomena with far more precision, coherence, and depth than they really do. The authors named this the illusion of explanatory depth. In the first studies, which used mechanical devices, self-rated understanding dropped after the attempt to explain. The authors did not read that drop as participants simply becoming modest. They read it as participants judging their own knowledge more accurately.

The same paper says the illusion is much stronger for explanatory knowledge than for knowledge of facts, procedures, or narratives. In the excerpts I read, there was also a result in which attempting to explain a procedure did not lower people's self-ratings. That distinction deserves attention in a work setting, because knowing the order of steps and knowing why the steps are in that order may be different kinds of knowledge.

The order can be written down, the reason is where it stalls

Handover documents are usually lists of steps. First open the file, second paste the figures, third get a sign-off. The steps go down quickly. Slowing down begins when you try to write, beside each line, why that step exists. I suggest reading that slowdown as a signal.

Take a five-step process for a month-end report. Say the third step is that a manager reviews the figures and then corrects them. From the step list alone you cannot tell whether that review exists to catch errors, whether it is a habit left over from one past problem, or whether the manager assumes nobody looks at the result anymore. Automation could delete the step. It could also copy the step exactly without knowing its reason. Either way, the decision is made without the reason.

That study was not about work procedures. It tested mechanical devices, and nothing in this essay shows that the same effect appears at the same size in report writing or approval flows. What I am borrowing is not the result but the shape of the question: ask someone who feels they know to explain it, and the gap between feeling and fact starts to show.

Marking the places where it stalls

The proposal starts by asking three things about each step. What happens if this step is missing? Who uses its output? Where would we find out if it went wrong? If the answers connect within two or three sentences, leave the step as it is. If they stall or contradict each other, mark it. Until the explanation is filled in, I would keep a marked step out of the automation or leave it with a person.Before you ask AI to do the job, write the one-page job sheet

A brass cylinder lock taken apart on a linen cloth. The body, plug, pins, and springs are laid out in rows, with a key, a pencil, and a blank sheet of paper nearby.
Generated image. A scene made for illustration, not a record of a real place.View full image ↗

After marking, there are roughly three things to do. Find someone else who can actually explain the step. Check whatever records remain, such as logs or documents. If there is still no answer, write down that it is unknown and let a person keep handling that step. Writing down an unknown looks like a small act, but it becomes the reference point later, when you look back at how a tool treated that spot.

One thing I would be careful about is asking an AI to fill the gap. An AI can produce a smooth, plausible reason, and a person may then come away with only the feeling that an explanation exists. This part is my own judgment, not something taken from the study, and I have not tested it. So I would record an AI-written explanation as a hypothesis until a person has checked it.

Visible steps and hidden steps

The paper's discussion raises the possibility that the ratio of visible parts to hidden parts may be an important factor in how strong the illusion is. An object like a zipper, whose moving parts are exposed on the outside, is easy to feel you understand. That discussion cannot be carried over to work as it stands. One question does carry over: in our own work, is a step that shows up on a screen as easy to explain as a step that sits behind another team or another system?

I suggest adding that question as the last column of an automation review list. For each step, note whether it happens on a screen or is handled by another person or system, and look separately at how well the second kind has been explained. Whether this column actually helps has not been checked.

One line to leave before the meeting ends

In the next automation review, I would add one more column beside the list, called steps where the explanation stalled. If that column is empty, you cannot tell whether every step was explained or nobody tried. So I would start with something small: one person explains the work aloud for five minutes, and a listener writes down only the places where it stalled.

This essay does not calculate whether the method saves time or reduces mistakes. An increase in the number of steps that have been explained would not be enough to say the work improved. What to look at next is how many steps actually stalled, and how many of those later got an answer. Only after that record builds up can anyone say whether the proposal was useful.DMS AX transformation design (in Korean)

Sources

Next: DMS AX transformation design (in Korean)

Paper-filled and empty document trays on a wooden desk
Generated image illustrating a hypothetical desk scene.View full image ↗

Essay / 2026. 10. 18

Subtract Before You Add

Considering the work a new tool would preserve

An improvement proposal can start with a new feature before anyone asks whether the underlying task is still needed. Considering removal gives that question a place, without treating safeguards as expendable.

Before adding a tool to improve a workflow, consider whether part of the existing work is still needed. I would put a list of tasks we might stop beside the list we plan to automate. The possibility of doing less does not authorize removing a safeguard or information someone else depends on.

Imagine papers covering a desk. Another tray would give them somewhere to go. Choosing a tray does not require deciding which papers are still useful. Keeping everything after examining it is different from keeping everything because removal never entered the discussion. The desk might look the same at the end, although the decisions behind it differ.

The desk and reporting situations in this essay are hypothetical, not DMS customer cases or measured results. The starting point is the abstract and publicly available experiment descriptions of a 2021 Nature paper, checked on October 18, 2026. The workflow questions that follow are my proposals, not a professional standard established by that study.

An option that never entered the comparison

Gabrielle Adams and colleagues investigated how people search for changes that add or remove components from objects, ideas and situations. Across eight experiments, participants were less likely to identify advantageous subtractive changes without a cue to consider subtraction, with only one opportunity to recognize the shortcomings of additive search, or under a higher cognitive load. Each comparison concerns the tasks and conditions studied. It is not a rule describing every decision.

One publicly described task involved stabilizing a structure with a small block supporting one corner of its upper platform. A participant could add supports or remove that block so the platform rested flush on the layer below. Adding blocks cost money; removal was free. Removing the support was advantageous under the task's particular conditions. Those conditions matter when interpreting the example.

What interests me happens before the ranking of solutions. If nobody has considered removing anything, addition and subtraction have not been compared. We have selected from the options that occurred to us. A discussion can feel thorough while leaving an entire kind of change outside its scope.

Moving a cue from an experiment into a work meeting does not guarantee the same effect. Organizations have contractual duties and permissions, as well as commitments between people. With those differences in view, I would still like to alter one meeting question. Before specifying the feature to build, ask whether any of the work could stop.

Does this report need to be automated?

First establish who uses the report and what decision it supports. Suppose someone transfers the same information into two documents each week. One goes to a team meeting; the other is retained separately. Before finding software to copy the information, we could ask whether the documents have different readers and whether those readers require different information.

If the meeting document is no longer read and no duty requires its production, stopping it becomes an option. A retention document subject to an audit or contractual requirement cannot be removed merely because its contents look similar. The actual obligations have not been investigated in this hypothetical example, so it offers no instruction to delete either document.

Perhaps both readers need the same facts in different formats. Generating both views from a shared source might then be worth considering, and that could require new automation. The comparison should include the copying that disappears and the maintenance that begins. Someone will need to keep the connection working and investigate errors. That responsibility belongs in the proposal.

I would write the proposed addition and leave a space beside it for work that might no longer be necessary. That space may remain empty. A review that finds no justified removal is still a completed review. Sacrificing useful work to fill the space would defeat its purpose.

It helps to imagine the day after a task has stopped. What will people open in the meeting? Whom will they ask when a fact is missing? Removing a document may produce repeated requests to the person who used to prepare it. A smaller document count would not, on its own, establish an improvement. The requests that appear belong beside the item that disappears.

Some inconvenience has a purpose

A confirmation step takes time. If it prevents an unintended external message or a change that is difficult to reverse, speed alone is a poor reason to remove it. A proposed deletion may also change who bears the risk. Convenience for the person running a workflow does not settle what protection its recipients need.

Even repeated checks require a closer look. Are they examining new evidence each time or rereading the same observation? Two checks that catch different errors may not be duplicates. Repeating the same check against the same evidence at the same moment could be a candidate for consolidation. I would establish who accepts the responsibility before reducing the number of checks.

A task can also disappear from one person's view while moving to somebody else's desk. Fewer fields on a form may mean a colleague has to collect the missing details later. The benefit to the requester and the burden on the processor both matter. If the information remains necessary after the field has gone, there must be a way to obtain it.

Before choosing an item to remove, I would ask the last person who used it to explain its purpose. Something that looks like an old custom from a distance might matter in a rare but consequential situation. Recent non-use does not establish that an item will never be needed. Where usage records or reasons for retention are missing, I would record the uncertainty rather than infer permission.

Paper cards spaced across a broad worktable, with one card set aside
Generated image of a hypothetical paper-card workflow.View full image ↗

A way to examine removal

1. Describe the required result. Instead of starting with an instruction to automate a report, establish who needs which information, for what decision, and when. Alternatives can be compared if they preserve that result. A goal defined as adopting a named tool excludes solutions that finish the work without it.

2. Write down an additive option and a subtractive option. Retain the option of leaving the process unchanged. This is a review, not a deletion instruction: keep the original files and do not stop a live procedure. If it is not yet clear that the alternatives deliver the same result, turn the difference into a question to investigate.

3. Establish the purpose of the item under consideration. Identify its reader, the error a check prevents, or a duty to retain a record. An unknown purpose is different from an unnecessary purpose. If nobody remembers the original reason, the decision lacks evidence; it has not acquired authorization.

4. Review with the person who has authority, then compare within a limited scope where appropriate. Preserve the original and a recovery method when possible, and examine whether users lose information they need. Even a trial may require permission when duties or hard-to-reverse changes are involved. These are the boundaries I propose for a review, not the paper's experimental procedure.

The additive option may be better. A missing capability can be a sound reason to build something. Giving removal a place in the proposal creates no obligation to choose it. What I want to preserve is the opportunity to compare. It also makes the explanation for keeping an existing task more specific.

How does this relate to DMS guidance?

It can be read alongside the DMS task boundary map guide, which starts by separating work that AI can take on from work a person should check. This essay proposes noting tasks whose necessity has not yet been established separately. It does not report that DMS training has produced time savings through this method.The task boundary map

When describing a reduction in work, I would include the review and correction that remain. The DMS guide to measuring AI adoption develops that perspective. No time saving has been calculated here. The disappearance of one report, by itself, is insufficient evidence that time has been saved.Measuring AI adoption

Space on the next meeting page

An addition is easy to name. A proposal can include a feature title and an expected screen. A photograph of the deliverable may leave the discontinued work invisible. In a review record, I would therefore describe what remains and whose responsibilities change, alongside what has been removed. Subsequent observation must establish whether the change helped.

The next improvement proposal would still have room for new tools. Beside that list, I would leave space for an output we might no longer need to produce. If the right person has not been consulted, the note would say that confirmation is required. Before the meeting ends, I would record the next action needed to answer that question.

Sources

Next: DMS AX transformation design (in Korean)

Four short paper strips and one long strip on a wooden desk
Generated image, not a photograph of client work or measurement records.View full image ↗

Essay / 2026. 10. 15

Waiting Outside the Average

The individual record beside the headline number

A set of tasks can average six minutes without any one task taking six. Mean and median offer a starting point for an essay on the waits a summary leaves out.

A lower average does not establish that everyone waited less. Quick tasks and a long unresolved task can share the same summary. When choosing a tool for work, I would keep a record of the task that took longest beside the headline number.

An estimate of six minutes sounds useful. You can put it into a schedule. Yet the task you are waiting for may take a very different amount of time without anyone having calculated the average incorrectly. The person reporting the figure is describing a group. You are waiting for one particular result, so that group summary may not answer your question.

The durations and work situations below are constructed examples, not measurements from DMS clients or operations. The definitions come from the US National Institute of Standards and Technology (NIST) statistical handbook, checked on October 15, 2026. The suggestions about explaining waiting are my own, rather than a standard prescribed by that source.

Nobody waited six minutes

Suppose there are five independent tasks. Four take two minutes each, and one takes twenty-two. Their durations add up to thirty minutes. Divide by five and the mean is six minutes. No task in the group took six minutes. The arithmetic is correct, although it describes none of the individual experiences exactly.

Put the durations in order and the median is two minutes, the middle observation. That seems closer to the four quick tasks. Replace the mean with the median, however, and the twenty-two-minute task becomes even harder to notice. Choosing a different central value has not supplied an account of the long wait.

For comparison, imagine five tasks that each take six minutes. Their total and mean match the first group, and their median is six minutes too. The first group contains four quick tasks and one long one; the second has no variation. A reader given only the mean cannot distinguish the groups. That is a specific reason to retain information beyond the headline.

NIST explains that the mean and median can differ for a skewed distribution. Extreme values affect the mean, while the median, which depends on rank, does not respond in the same way. The shape of the data and the question determine which description is useful. I read this as a reason to ask what a number is supposed to represent before accusing it of being inaccurate.

Someone estimating the combined amount of work may need the thirty-minute total. Here that is the sum of the individual durations, not elapsed time for a system running tasks in parallel. Someone waiting for the next result has a different concern: how much the individual durations vary. The same five records support different answers because the questions differ.

The inconvenient long task

Suppose there was a particular reason for the twenty-two-minute task. Perhaps input was missing, or the task paused for a person to review it. Those are possibilities in this example; a real explanation would require a record. A long duration alone is not evidence of an error, or evidence that the event will never recur.

If the question concerns computation alone, measuring the active processing interval separately may be appropriate. If the question concerns how long a requester waits for a usable result, the review delay belongs in the duration. Change the start or end of the measurement and the same task receives a different number. Put an unlabeled number beside another and readers may compare intervals that have little in common.

Removing a long task to improve a report also removes information needed by whoever must handle it next time. Blaming every delay on the tool is unhelpful too. I would retain the duration and state how much of its cause has actually been established. Unknown is a sufficient entry when the cause remains unknown.

For the five invented tasks, reporting a mean of six minutes alongside a longest duration of twenty-two changes what the reader can see. The average remains unchanged; the reader now has an additional observation. A person planning the work can ask about the long task before relying on the summary.

A way to read the waiting

1. Match the start and end. Did timing begin when the request arrived or when processing began? Did it stop at the first draft or when someone delivered a reviewed result? If different intervals were measured, explain the difference before comparing their averages.

2. Read the count with the central values. This example has five tasks, a mean of six minutes and a median of two. Five observations do not guarantee the duration of the next task. Keep the small sample attached to the numbers rather than turning them into a promise about all future work.

3. Reopen the long record. If nobody has established whether the twenty-two minutes came from preparation or review, leave that cause unresolved. Different causes call for different people and changes. Nor does the longest observed duration establish the worst duration that could occur on a future run.

4. Compare similar work. If familiar short tasks were mixed with tasks containing complicated material, consider inspecting them separately. Examine the assignments before judging the people who received them. These steps are my proposed reading method, not an automation assessment standard issued by NIST.

A card beside a paper tray and an empty chair in afternoon light
Generated image, not a photograph of client work or measurement records.View full image ↗

Does the median explain the wait?

The median does not tell you how long the slow tasks took or why. In this example it is two minutes, although one task took twenty-two. Use the median to describe the middle, and inspect the long tasks through their original records or a separate summary.

Sometimes the uncertainty around a wait matters as much as the duration. Consider a hypothetical requester who cannot begin another piece of work until a result arrives. An average without a current status leaves that person checking repeatedly. Even when a finish time is unknown, knowing which stage is pending gives them different information from silence.

That is not a reason to invent a completion estimate for every request. An unverified prediction makes the explanation less reliable. A status can report what is known, such as whether material has arrived or a review is pending. Recalculating the average more often will not answer those questions.

How is this different from the DMS measurement guide?

The existing DMS.Labs guide on measuring AI adoption connects faster processing with costs and actual work outcomes. This essay asks what disappears when several tasks become one number. It offers neither a rule to replace every mean with a median nor a comparison of particular tools.Measuring AI adoption

The guide proposes including review in total completion time and noting task difficulty and working conditions. Keeping individual records alongside that approach lets a reader investigate whether a certain kind of task remains slow after the overall average improves. Whether such a pattern exists must be established from observations. The invented numbers here cannot establish it.

The task boundary map guide separates work that can go to AI from work a person should check. Recording a delay against those stages could help distinguish slow generation from a review waiting for an available person. Before calling the review wasted time, the reader should also examine which errors it is there to catch.The task boundary map

In a training session I would leave time to examine one slow task rather than retain only the fastest output. The learner would first need a way to check the answer and material that does not expose sensitive information. Explaining the delay could then help separate necessary assistance from avoidable waiting and inform the next exercise. This is a teaching preference, not a reported training outcome.

Returning the number to a day

An average makes unlike records easier to read together. Someone receiving a report benefits from that compression. I would keep the convenience, with a route back to the observations. Finishing a summary is not the same as having understood every task inside it.

If I rewrote the five-task example, I would keep all five rows and place the average underneath. Beside the twenty-two-minute row I would record only what had been established about the cause. In a conversation with the person responsible for that task, we could begin with the stage that remained pending in that row, rather than ask them to account for a six-minute experience they never had.

Sources

Next: the DMS guide to designing AI training

A pencil and eraser beside faint pencil traces on paper on a wooden desk by a window
Generated image, not a photograph of a particular task or actual record.View full image ↗

Essay / 2026. 10. 12

Does Undo Erase What Happened?

Restoring a screen and reversing its effects

Undo may restore a document without reversing what someone has already read or acted on. A reflection on the limits of cancellation and the review time worth keeping in automated work.

Undo can return part of a screen to its earlier state. If someone has already read the content or begun work because of it, that effect may remain. Before handing a task to a tool, I think it is worth asking exactly what can be restored, rather than only looking for an Undo button.

Recovering a sentence deleted from a document is familiar work. The earlier state is still available, so the editor brings it back. It is easy to carry that expectation into other tasks: try something now, cancel it later if it is wrong. That is often a reasonable approach while a draft stays private. Once the document reaches another person, there is more to check.

The document and message situations below are hypothetical illustrations, not accounts of a client incident or something I experienced. I checked the Nielsen Norman Group explanation of user control and freedom and the W3C WCAG 2.2 error-prevention guidance on October 12, 2026. I distinguish what those sources say from the judgments I draw about automated work.

Having somewhere to return to

Nielsen Norman Group separates going back a step, cancelling an unfinished task, and undoing a change of state. Back, Close, Cancel and Undo help people leave unwanted situations, but they do not necessarily do the same thing. People need to be able to anticipate what will remain after using each control.

Consider closing an input window. In one application, the unfinished text is retained; in another, it disappears. The same-looking control leaves you with different work. Can you reopen the window and continue, or do you need to begin again? When that is unclear, the name of the button is not enough to describe the state it will leave behind.

Knowing how to recover is different from permission to make any mistake. A visible way to correct an error reduces some of the burden of exploring a choice. NN/g connects correction and backtracking with exploration and learning. I am not claiming a measured effect size here, or that an identical effect has been established for AI tools.

Erasing a pencil sentence can leave a faint mark on the paper. A digital editor may restore the earlier wording more cleanly. Yet if someone has already acted on the document, restoring the file is only part of the work. We need to separate the saved state from the events that followed from it.

What happened beyond the screen

Editing a draft incorrectly and distributing an incorrect draft may begin with the same sentence error. In the first case, comparing versions and restoring the wording may be enough. In the second, you may also need to find out which version the recipient read, whether it was forwarded, and what decisions it informed. Whether sending a correction finishes the job depends on the situation.

Removing a public post or changing a value back likewise does not establish that every intervening effect has disappeared. A reader may retain the information, and a copy may exist independently of the current screen. How far the material travelled is a fact to investigate. Neither assuming a wide spread nor assuming that nobody saw it helps.

My proposal is to restore the parts that can be restored and address the remaining effects separately. If a record has returned to its earlier value, that restoration can be reported as complete. If a document based on the wrong value is still in circulation, correcting the document remains another task. Naming what is finished and what is unknown makes the next action more specific.

The same distinction applies to a success message in an automation tool. A stopped status may mean that future actions have been prevented, or it may mean that earlier actions have been reversed. You cannot choose between those meanings without checking the tool. Pressing Stop and verifying the result should not be collapsed into a single completion record.

When should review happen?

I would keep a review point before creating an effect that is hard to reverse. Attaching a confirmation dialog to every click can turn review into a formality. It makes more sense to choose the review points by what changes and whom the change affects, without treating every task as if it carried the same consequences.

WCAG 2.2 Success Criterion 3.3.4 covers web pages that create legal commitments or financial transactions, modify or delete user-controllable stored data, or submit test responses. At least one of its conditions must hold: submissions can be reversed; entered data is checked for errors with an opportunity to correct them; or a mechanism allows review, confirmation and correction before the submission is finalized.

The choice among those conditions matters. This is not a requirement to make every action perfectly reversible afterwards. Review before finalization is another approach. The explanation also says that it is not intended to require confirmation for every save or for ordinary document creation and editing. Reading the scope too broadly would add friction to everyday revisions.

An accessibility criterion is not a universal certification rule for AI workflows. I use its explanation as a way to think about where review belongs. For an outgoing document, the recipient and attachment should be visible before sending. For a data change, the target should be visible before execution. Where recovery afterwards is incomplete, I would make the review beforehand more specific.

A sequence for checking what can be restored

1. Locate the earlier state. An original document or previous value has to exist before there is anything to restore. A backup said to exist is different from a backup you have opened and checked. If the material needed for recovery cannot be found, it is difficult to classify the task as reversible.

2. Check who can see the changed material. A private copy, a shared team document and content being sent outside the team have different possible effects. Permission to read is not evidence that someone actually read it. An unverified view does not need to become a fact in the record.

3. Identify the people and time needed for recovery. Selecting an earlier version on your own is different from restoring something with another operator’s help. I suggest no universal recovery-time figure. Start with the procedure and conditions established for the particular task.

4. Decide what to verify after restoration. Check separately whether the previous value has returned, whether an output already produced needs correction, and whether future execution has stopped. If one control handles several actions, define completion by the resulting states, not just by the button’s name.

An open cream envelope, paper and a small paper clip on a shelf in soft daylight
Generated image, not a photograph of a particular task or actual record.View full image ↗

How does this extend the DMS permission guide?

The DMS.Labs guide “Giving AI automation permission in three steps” separates reading, drafting and acting. This essay looks more closely at what can be reversed after the action. Keeping a task at reading or drafting leaves room to correct material that has not yet been distributed. Granting execution rights can still leave review in place for actions with substantial external effects.Read the permission guide

The related guide “Before you ask AI to do the job, write the one-page job sheet” also sets out the target of a task and its completion conditions. Including the restoration state in those conditions makes it easier to distinguish stopping execution from recovering the earlier state. This essay reports no project savings or client results; it offers a way of extending the guides’ reasoning.Read the job-sheet guide

Does an Undo button justify automatic execution?

The presence of the button is not enough. Check which state it restores, how long restoration is available, and whether anyone was affected in the meantime. A task can remain at drafting when those answers are unclear. Automated actions can run at different speeds and under different review conditions.

A useful starting exercise is to choose one cancellation feature and write one sentence describing what it restores. “The earlier document version is back” names something you can inspect. “The information already distributed has disappeared” requires much more evidence. Check whether the person delegating the work and the tool executing it use those descriptions to mean the same thing.

Sources

Next: automation, development and training enquiries

Five white toggle switches in a row on a wooden plate set into a beige wall. Four point the same way and one is flipped the other way.
Generated image. Not a record of a real place.View full image ↗

Essay / 2026. 10

Who Chose the Default?

기본값은 누가 정했을까

Open a new tool and the settings are already in place. Following studies of organ donor forms and retirement-plan enrollment, this essay looks at how the first value tilts a choice, and what to ask about the default settings of an AI tool.

Install a new program and a settings screen opens once. Most of us close it without reading. Notifications stay on, autosave stays on, and the switch that sends usage data stays where it was first put. This is not laziness. There is no time to weigh every item, and even with time it is hard to know which side is better. Meanwhile the value that happened to be there first hardens into something that feels like our own choice.

This essay follows a few studies of that first value, the default. At the end it turns to the default settings of AI tools at work, and how to look at permissions and records. I do not draw on my own experience as evidence. I say only what published papers support.

One line on a form

The most quoted case is organ donation. In 2003 Eric Johnson and Daniel Goldstein published a two-page piece in Science, “Do Defaults Save Lives?” They compared a form that says “check here if you want to be a donor” with one that says “check here if you do not want to be a donor.” The first is called opt-in, the second opt-out.

Their first source was an online experiment. 161 respondents each received one version. The share who said they would donate was 42 percent on the opt-in form and 82 percent on the opt-out form. A neutral form with no preset answer produced 79 percent. The neutral number is the interesting one. When the choice was left blank, answers landed near the opt-out result. One reading is that many people were open to donating, and the opt-in form did not turn that openness into a mark on the page.

Their second source compared countries. The paper plots “effective consent rates” side by side. The four countries that require explicit consent ranged from 4.25 to 27.5 percent. The seven that presume consent unless a person objects were far higher. A later meta-analysis summarizes the gap as the high nineties for opt-out countries and the tens for opt-in countries. For the flip of a single line on a form, that is a wide gap.

These are consent rates, not transplant counts. Johnson and Goldstein themselves note that a default can misclassify people in two directions. Some people willing to donate go unrecorded, and some who never wanted to donate end up recorded as donors. When a default speaks for you, what it says is not always right.

Savings that leave before you see them

Retirement saving shows a similar pattern. In 2001 Brigitte Madrian and Dennis Shea published a paper in the Quarterly Journal of Economics, “The Power of Suggestion: Inertia in 401(k) Participation and Savings Behavior.” It analyzed 401(k) records at one large American company, before and after the firm began enrolling new hires automatically. Employees could opt out whenever they liked.

A later meta-analysis sums up the finding this way: employees were roughly 50 percent more likely to take part in the retirement plan when enrollment was the default. The result does not show that people were unaware of the benefits of saving. It shows that filling in a form was easy to put off. Once the form disappeared, people who had been putting it off ended up in the same plan.

Johnson and Goldstein also explain, briefly, why a default lasts. Every policy needs some value that applies when nobody does anything, and a person who wants to change it has to pay costs in effort and thought. For something as emotional as donation, there is an emotional cost too. Finding the form, reading the items, signing and handing it in are each small, but together they make most people put it off. A default sits closest to that urge to wait.

How strong, and when it weakens

Default effects are not known from a handful of cases alone. In 2019 Jon Jachimowicz and colleagues published a meta-analysis in Behavioural Public Policy covering 58 default studies and 73,675 participants in total. The overall effect size was d = 0.68, with a 95 percent confidence interval of 0.53 to 0.83. A pre-selected option raised the chance of it being chosen by about 0.68 standard deviations.

The same paper stresses the spread. Most studies found a positive effect, several found no significant effect, and two found a negative one. Defaults worked better in consumer settings and less well in environmental ones. The authors account for part of the variation with two factors. Defaults were more effective when people read them as the designer's recommendation (endorsement), and when they read them as the current state of affairs (endowment).

The first factor deserves a second look. A default is not received as a neutral blank. It reads as a sign that whoever built the form thinks this value is right. The judgment of the person who arranged the options is already inside the first value.

None of this leads to the conclusion that defaults are bad. Some, like automatic enrollment, can help many people, and on a screen with dozens of options a starting point is often better than none. The trouble is less the value itself than the fact that users cannot see for whose circumstances it was chosen. What cannot be seen cannot be reconsidered. People need to know that a value can be changed before they can think of changing it.

A winding stone path around a dewy lawn, and a narrow dirt path cutting straight across the grass between the same two points, in low morning light.
Generated image. Not a record of a real place.View full image ↗

Defaults when you bring in an AI tool

The studies above cover donor forms, retirement enrollment and consumer choices. I did not find research that tests an effect of this size on AI tool settings, so this section is a cautious extension. The structure still looks familiar. Open a tool for the first time and a starting value already sits behind each question: which files it can reach, how long it keeps conversation history, how far it connects to outside services, whether it acts without a person's confirmation. Given the research above, it is likely that many of these values will be used as they came.

Whoever set those values is usually the company that made the tool. Whether they suit your team's work is a separate question. A vendor might start with wide permissions to make the first step easy, or with narrow ones to put safety first. Either way, someone's judgment is already sitting inside the settings screen.

With permissions the problem is more concrete. People usually learn what a feature touches only after they have started using it. By then the work is already running on that setting, and changing it means stopping the flow. That deciding once at first opening is cheaper than fixing later is my estimate, carried over from the logic of the studies above.

Three questions at the settings screen

This is not an argument for changing every default. Defaults exist to spare us the effort of choosing, and most of them can stay as they are. But a few that matter for the work are worth setting by hand once. These questions help.

First, who set this value? Separate what the product shipped with from what your team discussed and decided. Second, what does it take to change? Whether it is one click or a request to an administrator shapes how often people actually change it. Third, who pays if the value is wrong? As the donation paper listed errors in both directions, the cost of permissions that are too wide falls on different people than the cost of permissions that are too narrow.

The questions work at team scale too. For each tool, pick around ten settings, mark each “left as default” or “changed by us,” and write one line of reasoning. Months later, anyone who looks can see where the product's judgment ends and ours begins. This is not a conclusion from the research. It is a suggestion of my own.

If those three have answers, a default can stay a default. What changes is that the value is now somebody's decision. Where there is a person who decided, there is also a person who can fix it later.

A first value is easy to overlook, and so it tends to last. Reading the screen once more before closing it is often enough. Even when a path is already laid, you can still choose which one to walk once you are standing on it.

Sources

A paper planner pinned to the wall above a wooden desk. Colored paper strips and small blank notes are lined up neatly, with a pencil resting nearby.
Generated image. Not a record of any real place.View full image ↗

Essay / 2026. 10

Why a Careful Schedule Still Runs Late

꼼꼼히 짠 일정이 늘 늦는 이유

The more detail we put into a schedule, the safer it feels, and that feeling can pull the forecast off course. Following research on the planning fallacy and on reference class forecasting, this essay asks how to talk about the duration of a new piece of work.

Filling in a schedule feels good. Every box has a task, each box hands off to the next, and the last one carries a deadline. Nothing looks missing. Yet such schedules rarely finish on the date they promise. The cause is not sloppy planning. Careful planning is often exactly what makes the forecast wrong.

It happens every time a new piece of work arrives. Bringing an AI tool into a company, automating a repetitive task: the less familiar the job, the more optimistic the schedule tends to be. This essay follows a handful of studies from psychology and project management to see why. It does not lean on personal experience. It says only what the published material supports.

We shorten our own work, and only our own

The phenomenon has a name: the planning fallacy. It describes a tendency to underestimate the costs, durations and risks of planned actions while overestimating their benefits. The groundwork comes from Daniel Kahneman and Amos Tversky's 1979 papers on forecasting. The term itself was later popularized in a 2003 article by Lovallo and Kahneman.

The classic experimental paper is “Exploring the ‘planning fallacy’” by Roger Buehler, Dale Griffin and Michael Ross, published in 1994 in the Journal of Personality and Social Psychology. It combines five studies with 465 undergraduates in total. Predictions of completion time were too optimistic across a range of academic and nonacademic tasks. Two details stand out. People underestimated their own completion times but not other people's. In the fifth study, participants who acted as observers actually overestimated the time others would need, and they drew more on relevant past experience.

So the optimism is not simply a matter of sunny temperament. The same person is fairly realistic when checking someone else's schedule. The view changes only when the schedule is their own.

The more we picture the plan, the less we remember the past

In the same paper the researchers asked participants to think aloud while making predictions. The transcripts show that people mostly pictured how the work would unfold. Few recalled how long similar tasks had taken before. Making a plan means stringing scenes together, and the scenes usually contain only the smooth path.

Add the habit of planning carefully. Breaking work into small steps and estimating each one is standard advice from management tools. But the method only puts the steps we already know on the schedule. Waiting for approval, a changed specification, an urgent request from another project, an unfamiliar error from a tool used for the first time: none of these has a box, so none has a place. The denser the boxes, the more finished the plan looks, and the harder it is to suspect that something is missing. This reading is my own inference from the paper's findings. It was not measured directly.

The same paper also contains a hint at what can change direction. In the fourth study, participants who were told to connect relevant past experiences to their current prediction lost the optimistic bias. The problem, in other words, is not that people do not know their past. They do not pull it out at the moment they predict.

The view from inside, the view from outside

Kahneman separated the two stances as the inside view and the outside view. The inside view studies the details of the job at hand and pictures what comes next. The outside view sets the details aside and asks how similar jobs actually ended. In his 2006 article Bent Flyvbjerg lays out the distinction and notes that forecasts made from the inside were overly optimistic even when they were conservative. Placing the current job inside a distribution of outcomes from comparable jobs, by contrast, turned out much more accurate in the cases he describes.

The method has a name and a procedure: reference class forecasting. First choose a class of similar past projects. Then build a distribution of the quantity you want to forecast for that class. Finally, place the current project within that distribution. That is all. Its central move is to leave out the particulars of the present job on purpose, skipping the very place where the conviction that this one is different works hardest.

A wooden shelf by a window holding a row of cloth-bound ledgers and archive boxes of different thickness. One box is pulled partway out, and the spines carry no writing.
Generated image. Not a record of any real place.View full image ↗

What it looks like in numbers

Flyvbjerg's article includes a practical case. In October 2004 Ove Arup reviewed the business case for Edinburgh Tram Line 2. The base cost was £255 million. On top of it sat £64 million, 25%, for contingency and optimism bias, bringing the total to about £320 million. Using the UK Department for Transport's optimism-bias uplifts, the review then calculated two figures. For an 80% chance of staying within budget, total capital cost came to £400 million, which is 1.57 times the base cost, or a 57% uplift. For a 50% chance, it came to £357 million, a 40% uplift.

A note on how to read this. An uplift is not a prediction that costs will rise by that much. It says how much to allow in order to have a given chance of staying within budget, so the number depends on how safe you want to be. The review itself said its estimate was probably on the low side. The uplifts are meant to be applied to the budget at the time of the decision to build, and this project had not yet reached that stage, so more risk remained. The same article gives another example. A metro rail project that wants 80% certainty adds 57% to capital costs, so a £300 million budget becomes £504 million. At 50% certainty it becomes £420 million.

The format matters more than the size of the numbers. The method does not produce a single date. It produces a range with probabilities attached. When two numbers sit side by side, the decision-maker has to change the question, from when it will finish to how likely a given finish date has to be.

The method has limits too

None of this makes reference class forecasting a cure-all. A 2026 review by Cantarelli is titled “Reference class forecasting: promises, problems, and a way forward,” which tells us its weaknesses are a subject of their own. I will not restate that paper's detailed arguments here. The title does point to one thing: the accuracy of the method depends heavily on which projects are grouped into the class.

Carrying it over to work like AI adoption or process automation is harder still. Tools change quickly and every company's tasks differ, so there may not yet be a trustworthy class of comparable past projects. Bringing out a table of precise uplifts in that setting risks creating precision with nothing behind it. The studies above come from construction and transport projects and from student assignments, so applying them to AI rollout schedules is a cautious extension, not a verified conclusion.

Small things worth trying

One idea is still worth borrowing. The way Buehler's team removed the optimistic bias was modest. They made people bring up past experience at the moment of prediction. Translated into working habits, that becomes a few questions. How long did a similar job actually take last time? How far apart were the first plan and the finish date? What came up and knocked it off course? If no record exists, that absence is itself a result.

Another habit is to state a schedule as a range instead of a single line: the earliest date, the usual date, the late date, written side by side. I cannot claim this is a statistically validated method. It shrinks two findings from above to everyday size: our own work looks shorter than it is, and looking at a distribution improves forecasts.

For a job you have never done, it can also help to leave one box empty. It stands for whatever you cannot yet name, and it gives time to risks that have no label. An empty box does not mean the plan is sloppy. It records on the schedule that the plan does not know everything.

A careful plan is still needed. Splitting work and ordering it is how you know what to do today. You just do not have to believe that the same plan also says when it will end. The tool for deciding what to do and the tool for estimating how long it takes are different tools. Use the first from the inside, and borrow the second from the outside.

Sources

An open notebook and pencil on a wooden desk by a window, with a half-finished cup of tea and soft overcast morning light.
Generated image. Not a record of any real place.View full image ↗

Essay / 2026. 10

Saved, So Safe to Forget?

적어 두었으니 잊어도 된다는 말

Writing something down brings relief. Handing memory to the outside world has a cost and a benefit. Following the 'Google effect' paper and the research after it, this essay asks what to hand over and what to keep.

Writing something down brings relief. You put the appointment in the calendar, hand the password to a manager app, record what was said in the meeting, and a corner of your head seems to clear. It is hard to call that lightness laziness. People have stored memory outside themselves for a very long time: clay tablets, knotted cords, ledgers, pocket notebooks, and now phones and cloud documents.

Lately the old habit has picked up a new question. There are tools that answer in seconds with something already organized. What we hand over seems to be widening from memory to a piece of thinking itself. This essay accepts that worry only halfway. Handing things over has a real cost and a real benefit. I want to follow a few studies that separate the two and ask what to hand over and what to keep. I will not tell it as a personal story. I will lean on published papers.

What the expectation of finding it again leaves behind

In 2011 Betsy Sparrow, Jenny Liu and Daniel Wegner published “Google Effects on Memory” in Science. Across four studies, the paper reports two things. When people face difficult questions, they are primed to think about computers. And when people expect to have future access to information, they recall the information itself less well and recall where to find it better.

Wegner was already known for describing, in the 1980s, how close partners share the work of remembering, a pattern he called transactive memory. One partner keeps the family calendar and the other keeps track of taxes and repairs. The paper argues that the internet has become one more store of this kind. We used to lean on other people for memory. Now we lean on a search box too.

The finding spread as the “Google effect,” and it is often cited as proof that the internet is ruining memory. The later story deserves equal attention. Several subsequent studies questioned the result, and the 2018 replication project by Camerer and colleagues, which re-ran 21 social-science experiments from Nature and Science, is often cited in that discussion. Across that whole project, 13 of the 21 studies showed a significant effect in the same direction as the original, and the average effect size was about half the original. How any single study fared needs checking on its own, but the broader lesson is clear: one striking experiment should not carry a large conclusion.

So what should we take from the paper? Not a flat claim that the internet damages memory. A smaller possibility is enough: expecting to find something again may shape what we hold and what we let go, and we may end up remembering the location instead of the content. Even read that modestly, it leaves plenty to think about.

One experiment in the paper shows the scale of the claim. Participants typed short factual statements into a computer, and those told the statements would be erased remembered them better than those told they would be saved. The reading is that the head spends less effort when it believes the material will stay available. So the effect is less a story about memory getting worse than about people adjusting where to spend their memory when they know something will remain.

When saving made remembering better

On the same topic there is a result that points the other way. In 2015 Benjamin Storm and Sean Stone reported three experiments in Psychological Science. When participants saved one file before studying a new one, they remembered the new file better than when they had not saved. Their reading is that saving cleared out content that was not needed at that moment, so the new material suffered less interference.

The conditions in that paper matter. The effect did not appear when people judged the saving process unreliable. It also did not appear when the saved content was not substantial enough to interfere with the new file. So the result does not say that any note helps at any time. The benefit shows up when there is a store you trust and the thing you hand over is heavy enough to occupy your head. It is a benefit with conditions attached.

Handing things over has a name

In 2016 Evan Risko and Sam Gilbert published a review in Trends in Cognitive Sciences that gathers these behaviors under the name cognitive offloading. Counting on your fingers, writing a shopping list and asking a map app for the route all fit. The term covers any action that moves some of the processing load from the head to the body or a tool.

The usefulness of the frame is that it treats handing over as a trade, not as a good or bad habit. Offloading lowers the load right now. In exchange you may remember the offloaded material less well, or be stuck if the store disappears. The two studies above each show one side of that trade. Sparrow's team saw a cost, and Storm and Stone saw a benefit. They read less as rivals than as two views of the same exchange.

A box of index cards standing neatly on a wooden table with a few handwritten cards scattered beside it. The writing is not legible.
Generated image. Not a record of any real place.View full image ↗

When the tool moves from facts to process

Search mostly tells you where a fact lives. Tools that summarize, compare and draft go a step further. They take over the reading and organizing, not just the finding. What we hand over moves from what to how. The studies above mostly dealt with storing facts and files. Nothing in them supports saying the same results hold when we hand work to generative tools. It is more honest to say we do not know.

What we can do is turn the conditions those studies offer into questions. Before handing something over, ask a few. Can I open this store and trust it when I need it? After handing it over, do I still know where it is? Is the thing I am handing over material I will need for a later judgment, or is it a weight that has nothing to do with judgment?

The last question seems the most important. A password or an appointment time can be handed over with little risk to judgment. Which clause of a contract is risky, or why this figure differs from last month, is different. That is material you should still be able to check after handing it over. If the habit becomes keeping only the summary and never looking at the source, then when the summary is wrong, nobody is left to notice. This is not a research finding. It is an inference from carrying the studies' conditions over to work.

Keeping the list of what you handed over

In practice it is modest. Keep a list of what you have handed over. If Sparrow's team found location in place of content, then at least be sure of the location. Write down which document holds which content, who edits it, and how it is recovered if it disappears. Then the memory you handed over does not get lost.

What to hand over and what to keep will depend on the work, but there is a place to start. Things you only need to see again accurately, such as schedules, account details, reference links and the decisions made in a past meeting, are good to hand over. Why a decision went the way it did, which standard picked one of two options, and where confirmed fact ends and guesswork begins should stay with you. A record keeps the outcome, and the reasons for a judgment usually stay in a person's head. This split is also not something the studies proved. It is a proposal that carries their conditions over to work.

The list matters more when several people work together. Transactive memory, as Wegner described it, runs well when people know who knows what. It seems reasonable to think the same applies to a team that has handed work to documents and tools. Which document is current, what each automation does, and who understands it should be spread through the team. Without that map, you have memory that exists and that nobody can find.

One more habit helps: now and then, deliberately take something back out. Check that the store still opens, that the content is still right, and that you can explain it. You do not need to memorize everything. Keep a handful of core items in a state where you could explain them. That connects to the earlier piece in this series, where what you cannot describe cannot be handed over either.

Putting memory outside the head is something people have always done and will keep doing. The trouble may lie less in handing things over than in forgetting that we did. Keep a list of what is out there, open it now and then, and read for yourself the material your judgment depends on. On those terms, “saved, so safe to forget” becomes a fairly trustworthy sentence.

Sources

A person seen from behind in a dark room late at night, looking at several quiet monitoring screens, with a cold cup of tea and a notepad on the desk.
Generated image. Not a record of any real site or project.View full image ↗

Essay / September 2026

What Is Left Is Watching

지켜보는 일이 남는다

When automation takes over the work of the hands, the work of watching stays with people. A look at a 1983 paper and a 1948 experiment, and at how checking might be designed as real work.

It is easy to assume that once a tool does the work, the person gets some rest. The hands are free, the story goes, so what remains is judgment and creativity. Automation research has been telling a different story for decades. The work does not disappear. It changes shape, from doing the task to watching it get done.

This essay is about that change. Generative tools can now produce a first draft in seconds, and the bottleneck has moved from writing speed to reading speed. Who makes something fastest matters less than who examines it properly. I will not pretend to tell this through personal anecdotes. Instead I want to lean on a few older papers and follow, step by step, why watching is hard.

What remains where the work used to be

In 1983 Lisanne Bainbridge published a short paper in the journal Automatica called “Ironies of Automation.” Its question is plain. When an industrial process is automated, do the problems of the human operator shrink, or do they take another form? Her answer leaned toward the second. Automation can expand the human problems it was meant to remove.

The logic goes like this. Designers see the person as a source of error and try to take them out of the loop. Whatever they cannot automate is left to the person: the exception, the moment the machine stops, the fault nobody has seen before. Most of the time the machine does everything, so the operator has nothing to practice on. Then they are called in for the hardest moments only. Practice has vanished while the required skill has gone up. That structure is what the paper calls ironic.

Then there is the weight of watching itself. Staring at a screen where nothing happens, while staying ready to catch the one odd signal, is more draining than it sounds. Bainbridge concluded that as automation advances, operators need more training, not less, because the interventions are rare and decisive.

The paper was written about process-control rooms, but it maps onto today's offices with only small edits. A tool drafts the report, a tool writes the summary, a tool proposes the first version of the code. A person skims the result and signs off. The writing hand rests while the reading eye gets busier. And the reading eye tires far more easily than the writing hand.

The number thirty minutes

Why watching is so hard becomes clearer if we go back to 1948. The British psychologist Norman Mackworth built a clock task to study the vigilance of radar operators. A clock hand ticks forward one step at a time and, very occasionally, jumps two. The participant only has to notice the double jump. It is a simple task, and the participants worked hard at it.

The result was clear. After roughly half an hour, missed jumps rose noticeably. The effect became known as the vigilance decrement, and it has been confirmed and argued over for decades since. A 2025 review of its first 75 years credits Mackworth's 1948 paper as the usual starting point for the experimental analysis of the decrement. Researchers still disagree about the mechanism. The broad pattern has held: the longer people watch, the more of the rare signals they miss.

One detail in that same review is especially interesting. Mackworth found that a salient interruption, such as a telephone call, or a half-hour break could reduce or even remove the decrement. So could feedback on performance. Urging participants to be more attentive had no effect. In other words, this is not something willpower fixes. A plea to concentrate is not a design. Breaks and feedback are materials a design can use.

Bainbridge noted the same experiments in her paper, in the sense that it is very hard for even a motivated person to keep effective visual attention on a source where little happens for much longer than half an hour. One caution is needed. Thirty minutes is a figure from particular tasks and conditions. It is not a law that applies to every kind of checking. What many studies share is the direction: watching something that is mostly normal for a long time dulls the watcher without any decision to be dull.

An early-morning desk by a window with a printed manuscript, a red pencil and a half-finished coffee, with one line on a page underlined.
Generated image. Not a record of any real site or project.View full image ↗

When trust stands in for attention

The better automation works, the more another problem appears: trust begins to replace checking. In a 2010 review in Human Factors (volume 52, issue 3), Raja Parasuraman and Dietrich Manzey gave this two names, complacency and automation bias. The first is failing to monitor automation closely enough. The second is following an automated recommendation without critical review. Earlier work had treated the two separately, and the review tried to explain both through attention.

Two points from the review are worth carrying over. First, such errors are not a flaw of careless or lazy people. The authors see them as the outcome of an interaction among personal characteristics, the situation and the properties of the automated system. Second, attention sits in the middle of it. So the response has to come from designing where attention goes, not from a slogan that says be careful. The authors present their integrated model as a framework for design options in automated and decision-support systems.

Take an everyday case. Suppose a tool gives ten correct answers in a row. The eleventh answer will be doubted far less, and not because the person is foolish. Ten correct answers create a reasonable expectation about the next one. The trouble is that the expectation quietly wears down the effort of checking. The more accurate the tool, the rarer its errors, and the rarer the error, the less likely the eye is to catch it. It is the same shape as the vigilance decrement.

Designing the work of checking

So what can be done? Translating the papers into ways of working produces a few ideas. To be clear, this list is my own reading of the material above, not a prescription from the studies.

First, limit the amount. Vigilance research suggests that the longer we look, the duller the eye gets. Short, separate passes are then better than one long one. For a long document, split it into sections and write down in advance what to look for in each. The goal of reading everything carefully sounds diligent, but it tends to become a promise that erodes as the hours pass.

Second, build in gaps on purpose. Given what Mackworth found about interruptions and breaks, a pause in the middle of a review is part of accuracy, not a loss of time. Feedback belongs here too. If you can find out afterwards what you missed, the checking eye can adjust. Where a missed error disappears without a trace, review slowly turns into a formality.

Third, keep some chances to do the task by hand. Bainbridge's irony is also a statement that skill needs practice. If automation takes every repetition, people lose their feel exactly when they must step in. It can be worth doing some tasks manually. You give up a little efficiency and keep the sense that lets you say something looks wrong when it does.

Fourth, make the items to check visible. Pull out numbers, dates, proper names, quotations and links, the things that hurt if wrong, and compare them one by one. Smooth sentences make the eye slide. Broken into items, the text lets you look at the facts without leaning on its fluency. This holds whether a machine or a person wrote it.

Fifth, write down outside the tool that it can be wrong. How accurate a tool is belongs to the tool. How much a person is answerable for belongs to the organization and the individual. If it is unclear what the person pressing approve is actually confirming, everyone can end up trusting each other while nobody checks.

— The faster the tools get, the more reading speed becomes the speed of the work. —

The dignity of watching

Watching is not glamorous. Done well, nothing happens. Done badly, something large happens later. Because the result is invisible, it is easy to rate it low. When work is divided, people and time flow toward the making side, and the checking side gets the leftover minutes.

But the deeper automation goes, the more the scale tips the other way. Making gets cheaper and checking gets relatively more expensive. When the cost of making falls, more gets made, and more must be checked. I think this is why a sentence Bainbridge wrote more than forty years ago fits better today than it did then.

So respecting the work of watching is more concrete than it sounds. Give checking its own time. Plan the amount and the pauses so the person checking does not wear out. Leave a way to trace what was missed. And build into the way of working the idea that approval is not just a click but a mark of responsibility.

Automation takes the work of the hands. What stays is the work of the eyes and of judgment. Not treating that remainder as minor may be the oldest skill a person working alongside machines can have. A clock from 1948 and a paper from 1983 say roughly that much. Watching is never easy, and that is why it is worth designing.

Sources