Fewer clicks did not settle the health-card experiment

Four health cards on a search page made it possible to finish more tasks without opening a result. In a historical laboratory experiment, that same interface also produced fewer correct diagnoses than a single-card layout. The apparent convenience and the scored decision moved in different directions.

Even a correct diagnosis left something important unresolved. Among post-search decisions that named the vignette's correct condition, 43% still chose the wrong urgency category. And choosing the better interface required another distinction: how well people responded when a matching card was supplied, and how often a search system would supply it. The authors' calculation changed the interface ordering when they changed that second probability.

This is an account of constructed consumer-health search tasks, not evidence of patient outcomes or a verdict on current answer engines. It uses the institutional edition of Jimmy, Zuccon, Demartini and Koopman's Health Cards to Assist Decision Making in Consumer Health Search, AMIA 2019. The paper reports 64 adults completing eight scenarios each: 512 repeated task responses, not 512 independent people. Its rotation description accounts for only 48 participants, an unresolved discrepancy. Its reported tests also treat within-person tasks with unpaired t-tests; they are not a participant-clustered reanalysis. These limits travel with the results below. Methods and analysis, pp.2–6.

Return to the two decision endpoints, the card-supply calculation, or the source and method notes. Download the rounded diagnosis inputs and weighting calculations.

The experiment asked for two answers

Each scenario described a person with symptoms. Participants had to name the condition, copying its mention from a card, a snippet or a linked page, and choose what to do next: self-treat, contact a health professional, or use an emergency service. The researchers scored those submissions against the modified vignette's keyed diagnosis and urgency. The copied mention recorded the final submitted source; it did not record every influence on the decision. Task protocol, p.3; vignette keys, Table 2, p.4.

The search pages were built for the study. Fixed queries retrieved archived Bing results from February 2, 2019, alongside Google-style health cards. Participants could follow links but could not rewrite the query. A “matching” card named the vignette's keyed condition; a nonmatching card named another condition. Nonmatching does not mean its medical text was fabricated.

There were four settings: one matching card, one nonmatching card, four cards including one matching card, or four nonmatching cards. There was no zero-card arm. The single card sat beside the results. The four cards occupied the area above the snippets, with only three snippets initially visible. Card count, placement and available space changed together. This was a comparison of interface families, rather than an isolated test of adding three cards. Interfaces and construction, pp.4–5; Figure 1, p.2.

At least two of the ten snippets mentioned the keyed diagnosis in every setting. A missing matching card therefore did not remove every source of matching information; participants could still find the condition through results. Result construction, p.5; appraisal discussion, p.9.

A completion label with no accuracy test

Across 256 task responses per interface family, the one-card layout produced 2.4844 opened pages per task, compared with 1.3438 with four cards. Completion without clicking a search-page link was 23.44% with one card and 47.27% with four. Keyed diagnosis accuracy went the other way: 37.89% with one card, 24.61% with four. Average duration barely changed: 157 seconds with one card and 158 with four, with no significant duration difference reported. Table 4, p.7; result interpretation, p.8.

The source calls the no-click completion measure good abandonment. Its operational test is that a scenario was completed without clicking a link. A correct diagnosis or urgency choice is not part of that test. The adjective cannot do the work of the answer key.

For four cards minus one, the differences are −1.1406 opened pages per task, +23.83 percentage points of no-click completion and −13.28 percentage points of diagnosis accuracy. These are three different units, not three stages of an observed participant journey. The authors report significant differences for these outcomes under their testing procedure, but no effect intervals or participant-clustered reanalysis are provided here. Parallel averages do not establish that stopping the clicks caused a wrong diagnosis. Nor do they establish that every no-click task failed, or that another click would have rescued it.

Getting the diagnosis right left another decision

The strongest separation is within the correctly diagnosed decisions. The institutional article's Discussion reports 43% with incorrect urgency. Of that wrong-urgency subset, 73% underestimated urgency. The first denominator is decisions with a correct diagnosis; the second is the incorrect-urgency decisions inside that group. Neither figure is a percentage of all participants. Discussion and original joint flow, Figure 4, p.9.

A condition label and a next-action category therefore cannot substitute for each other. The observation is a submitted urgency choice against a constructed key. It is not a record of someone delaying care or suffering harm.

The before-and-after figures sharpen the distinction. About 4% of diagnosis responses were correct before searching, then 38% with one card and 25% with four. Exact urgency matches were 49% before, then 50% with one card and 44% with four. Diagnosis accuracy improved descriptively while exact urgency changed little or declined. All pre-task questions were answered before the search tasks; without a contemporaneous no-search group, these changes do not isolate search from rereading, practice or time. Discussion, p.9.

There is also a second urgency measure in Table 4: 0.543 for one card and 0.4453 for four. These are mean penalty scores, not 54.3% and 44.53% correct. The score starts at 1 and subtracts the number of category steps missed, with a penalty of 1 per step for underestimation and 0.5 for overestimation. An exact match scores 1; a one-step underestimate scores 0; a two-step underestimate scores −1; a one-step overestimate scores 0.5. The mean can distinguish the direction and distance of mistakes that the exact-match fraction simply counts as wrong. Score definition, p.8.

That distinction changes a citation. “Single cards had a higher urgency penalty score” and “50% of single-card urgency choices matched the key” are supported statements about different endpoints. Merging them into one accuracy percentage loses what was measured.

Four response rates, then a different supply problem

The pooled diagnosis result initially favors the single card. To understand the authors' later calculation, first separate responses according to whether a matching card was present. Table 4 gives these four diagnosis rates:

• One matching card (SC): 57.81% of task responses named the keyed diagnosis.
• One nonmatching card (SN): 17.97%.
• Four cards including a matching card (MC): 40.62%.
• Four nonmatching cards (MN): 8.59%.

These are conditional response rates for the study's settings. They are not a search system's probabilities of retrieving a matching card. In particular, the MC rate describes diagnosis responses when the keyed card was included; it is not the probability that a user chose that card. People could also use snippets and linked pages. Table 4, p.7.

The single-card response rate is higher in both comparisons: matching supplied versus matching supplied, and matching absent versus matching absent. But those comparisons hold card availability fixed. A system that often supplies the wrong single card spends more of its time in the low-performing second setting. To compare systems, the two response rates must be weighted by how often each supply setting occurs.

First, reconstruct the balanced experiment. With matching and nonmatching settings weighted equally within each family, the rounded table inputs give:

• One card: half × 57.81% + half × 17.97% = 37.89%.
• Four cards: half × 40.62% + half × 8.59% = 24.605%, which rounds to the reported 24.61%.

The weights answer “how often was a matching card supplied?” The percentages multiplied by them answer “how often was the diagnosis right under that setting?” Their weighted sum answers a diagnosis question over the specified mixture. It still says nothing about joint diagnosis-and-urgency success.

Now change the supply odds, as the authors do. Their eight-query ranking example placed the keyed concept first for just one scenario: 1/8. It was among the first four for four scenarios: 4/8. A top-one selector therefore supplies a matching single card in 12.5% of these queries; a top-four selector includes one in 50%. Inclusion is not selection or success. Ranking inputs, Table 3, p.5; weighting, Equation 1, p.8.

Holding the four measured response rates fixed, that example becomes:

• One card: one eighth × 57.81% + seven eighths × 17.97% = 22.95%.
• Four cards: four eighths × 40.62% + four eighths × 8.59% = 24.605%, rounded to 24.61%.

The four-card result stays the same because its matching-card weight stays at one half. The single-card result falls because its matching weight drops from one half to one eighth. The ordering changes even though none of the four response rates changes. Under these inputs, the unrounded calculated gap favors four cards by 1.655 percentage points.

This is the authors' selector illustration using published rounded cells, not a second experiment testing those selector policies. It does not show a statistically demonstrated field advantage. The ranking example itself was constructed: five candidate concepts were constrained to include the key, replacing the bottom concept if necessary, and only concepts with Google cards qualified. The top-four matching scenarios were 2, 5, 7 and 8; only scenario 5 matched at the top rank. Table 3 and candidate construction, p.5.

There is a substantive assumption inside the multiplication. The 57.81% response rate pools scenarios in the single-matching setting; the selector's top-one match comes from just one scenario. Applying that pooled response rate to the selector assumes it remains appropriate for the topics, people, card positions, distractors and layout that the selector would produce. The analysis here has no per-topic participant results to test it, so that assumption is not validated. A more difficult matching scenario could change the response rate as well as the supply weight. Simply inserting a new engine's coverage number would not test this transport.

The useful comparison therefore has two parts: the chance of supplying a matching card, and the response conditional on what was supplied. The arithmetic joins them only when their populations and conditions fit. Neither part alone settles end-to-end decision quality.

Relevant cards could help with a different task

Separately scoped positive and null evidence prevents a blanket conclusion that cards necessarily worsen decisions. The same research team also ran a SIGIR 2019 study of 48 people completing eight tasks each. It compared card presence with absence, deliberately supplying relevant cards when present. Some tasks named a condition and asked about treatment or living with it; others required inference from observations. Fixed queries were expanded with card-title words, with archived Bing results from October 5, 2018. This did not test the AMIA mixture of matching and nonmatching cards. Health Cards for Consumer Health Search, Methods and Tables 1–2, pp.3–5.

The assigned card-present tasks had fewer followed links: 3.156 versus 3.615 without cards (reported p=.043). Participants also rated the work easier, 3.776 versus 3.589 on a scale where higher means easier (p=.043), and were more satisfied, 3.693 versus 3.453 (p=.018). These are favorable measured effort and experience findings. Discussion, p.9.

Correctness was scored on a 0–3 evidence rubric, from no correct response to a complete correct response. Overall means were 2.38 with no cards, 2.56 when cards were displayed but not selected, and 2.54 when card information was selected; those differences were nonsignificant. Durations of 240, 242 and 231 seconds, respectively, were also nonsignificant. Choosing card information was voluntary, so these three groups are not three randomized arms. Table 5 and Figures 4–5, p.7; results, pp.6–7.

Figure 9 compares the “Health cards” group, which selected card information, with the “Search results only” group, which did not select card information. It does not separately plot no-card tasks and displayed-but-not-selected tasks within its “Search results only” comparator. These are source-selection comparisons, not randomized effects of card use.

For named-condition tasks, the source reports significantly higher correctness, shorter duration, fewer followed links, greater perceived ease on the workload scale and greater satisfaction for card selectors. For inference tasks, the same card-information-selected versus search-results-only contrast was nonsignificant for all five outcomes; correctness was directionally lower and duration longer among card selectors. Figure 9 and accompanying results, pp.8–9; use its actual axes, because its caption swaps several panel descriptions.

These findings do not erase the AMIA diagnosis and urgency results: the tasks, supplied information and scoring scales differ. Shared investigators, recruitment context and ethics identifier do not establish whether the participants were disjoint. The two studies cannot be pooled as independent estimates of one common outcome. SIGIR's differently scoped no-card arm supplies neither AMIA's missing zero-card counterfactual nor its before-and-after no-search control.

The health-card experiment measured less browsing, submitted diagnoses and urgency choices separately. Its pooled single-card diagnosis advantage and its differently weighted four-card calculation can both be read faithfully, because they describe different mixtures. The missing click cannot settle the comparison; neither can a matching card or a correct condition label finish the decision on its own.