Analysis Error

How an analysis error changes a conclusion

An analysis error is a mistake in transforming data and assumptions into an estimate, model result, or interpretation. It can occur even when the underlying data were collected correctly and can change the magnitude, direction, uncertainty, or meaning of the reported result. This page explains where analysis errors arise, how they differ from uncertainty and bias, how they affect statistical and economic analyses, and how reproducible checks can prevent or correct them.

Error, uncertainty, and bias are not interchangeable

An analysis error is a preventable departure from the intended or appropriate analytical procedure. Statistical uncertainty is the expected imprecision that remains even when the analysis is performed correctly, while bias is systematic deviation that can arise from design, measurement, selection, confounding, or analysis. A biased estimator is not necessarily a programming mistake, and a wide confidence interval is not evidence of error.

Problem	Meaning	Example
Analysis error	The analysis was specified, implemented, or interpreted incorrectly	A treatment indicator is coded in the wrong direction.
Data error	The recorded or supplied value is incorrect	A decimal point is misplaced in a cost field.
Sampling variation	Different samples produce different estimates by chance	A confidence interval is wide in a small study.
Bias	The expected estimate differs systematically from the target	Confounding remains in an observational comparison.
Model uncertainty	Several plausible analytical structures exist	Alternative survival curves give different extrapolations.
Fraud or misconduct	A deliberate act misrepresents evidence	Results are knowingly fabricated or selectively altered.

An analysis can contain more than one problem. The investigation should identify what happened, which results are affected, and whether the issue was accidental, methodological, or deliberate without assuming intent from the numerical discrepancy alone.

The analytical target must be defined first

An analysis cannot be judged correct without knowing the question it was intended to answer. The estimand specifies the population-level effect or quantity of interest, including treatment conditions, outcome, population, and handling of events after treatment begins. A technically correct model can still answer the wrong question.

The analysis plan should define:

• The target population and unit of analysis.
• Intervention, comparator, exposure, or policy contrast.
• Outcome definition and assessment period.
• Summary measure, such as mean difference, risk ratio, hazard ratio, or incremental net benefit.
• Treatment of intercurrent events, competing risks, and missing outcomes.
• Primary, secondary, subgroup, and sensitivity analyses.

Errors can enter anywhere in the analytical pipeline

The pipeline includes data extraction, cleaning, transformation, linkage, derivation, modelling, validation, presentation, and interpretation. An early mistake can propagate into every later table and figure. Checkpoints should therefore test both intermediate data and final outputs.

Common locations include:

• Selecting the wrong records, dates, variables, or population.
• Applying inclusion and exclusion criteria incorrectly.
• Joining tables with duplicate or non-unique keys.
• Recoding categories or treatment groups incorrectly.
• Using the wrong units, scale, sign, currency, or price year.
• Implementing the wrong equation or software option.
• Reporting a different model, dataset, or scenario from the approved analysis.
• Misreading the direction or clinical meaning of the estimate.

Population errors change who is analysed

An incorrect cohort can arise from a wrong date window, eligibility rule, diagnosis code, washout period, or index event. The resulting analysis may be internally consistent but apply to a different population from the one described. Flow counts and reproducible inclusion logic are essential.

Checks should reconcile:

• Source population and each exclusion step.
• Unique people, episodes, observations, and repeated records.
• Baseline period and follow-up eligibility.
• Treatment assignment and crossover.
• Attrition, censoring, and outcome availability.
• Final counts in models, tables, figures, and abstracts.

Join and duplication errors can silently distort results

Many analyses combine patient, visit, treatment, cost, and outcome tables. A one-to-many join performed as though it were one-to-one can duplicate records and inflate totals or precision. Missing matches can also remove patients or outcomes selectively.

Before and after each join, analysts should test key uniqueness, row counts, match rates, duplicated identifiers, and aggregate totals. A join should fail loudly when its expected cardinality is violated rather than silently expand the dataset.

Unit and scale errors can be large

Health data may combine milligrams and micrograms, days and years, proportions and percentages, or costs in different currencies and price years. A correct formula applied to inconsistent units produces an incorrect result. Units should be attached to variable definitions, equations, validation rules, and outputs.

For example, converting an annual rate (r) to a cycle probability over (\Delta t) years under a constant hazard is:

$$ p=1-\exp(-r\Delta t) $$

Using (\Delta t=1) for a monthly cycle instead of (1/12) can greatly overstate the event probability. The assumption of a constant hazard must also be justified; correct conversion does not validate the underlying model.

Derived-variable errors alter meaning

Composite endpoints, baseline values, change scores, severity categories, adherence measures, and time-to-event variables require explicit derivation rules. A small choice about dates, thresholds, or denominator can change who is classified as responding or exposed. The programmed derivation should be compared with worked examples and source specifications.

Typical problems include:

• Reversing a scale without updating the interpretation.
• Using post-baseline information to define baseline groups.
• Calculating change as baseline minus follow-up when the plan specifies the reverse.
• Applying an outcome threshold before rather than after standardisation.
• Treating missing values as zero.
• Counting overlapping events as independent events.

Model specification errors answer the wrong relationship

Specification errors include omitting necessary variables or interactions, using an inappropriate functional form, ignoring clustering or repeated measures, mishandling time, or selecting a distribution inconsistent with the outcome. A model can converge and produce plausible-looking coefficients despite being inappropriate. Diagnostics and substantive reasoning are both required.

A simple linear model is:

$$ Y_i=\beta_0+\beta_1X_i+\boldsymbol{\gamma}^{\top}\mathbf{Z}_i+\varepsilon_i $$

Errors can arise if an important confounder is omitted from (\mathbf{Z}_i), a nonlinear relationship is forced into (\beta_1X_i), or correlated observations are treated as independent. Adding variables mechanically is not a substitute for a causal and statistical rationale.

Coding an exposure or reference group incorrectly reverses results

Regression coefficients and contrasts depend on category coding and reference levels. A treatment effect can reverse sign when the reference group is changed, even though the underlying fitted values remain the same. The error occurs when the analyst reports the coefficient as if the original reference still applied.

Outputs should print category labels, coding, reference group, event level, and contrast definition. A small synthetic dataset with a known expected direction can expose reversed coding before the full model is run.

Missing-data errors can create false precision

Deleting incomplete records without examining why data are missing can change the analysed population and bias estimates. Single imputation can understate uncertainty, while multiple imputation can be wrong when the imputation model omits important variables or conflicts with the analysis model. Missingness assumptions should be connected to the estimand and tested.

The report should show:

• Amount and pattern of missingness by variable and group.
• Reasons and timing where known.
• Variables used in the imputation model.
• Number of imputations and convergence checks.
• Pooling method and compatibility with the substantive model.
• Sensitivity analyses for departures from the main missingness assumption.

Time-to-event analyses have distinctive failure modes

Survival analysis requires correct time origin, event definition, censoring date, delayed entry, competing risks, and risk-set construction. Treating censoring as an event or assigning immortal time to the exposed group can cause serious bias. Proportional-hazards assumptions should be tested when a constant hazard ratio is reported.

The Kaplan-Meier estimator is:

$$ \widehat{S}(t)=\prod_{t_j\leq t}\left(1-\frac{d_j}{n_j}\right) $$

where (d_j) is the number of events and (n_j) is the number at risk at event time (t_j). Incorrect risk sets, event coding, or time ordering will corrupt the estimate even when the software function is used correctly.

Immortal-time bias can favour treatment incorrectly

Immortal time is a period during which a person must remain event-free to be classified as exposed. If that period is assigned to the treated group, the analysis gives treatment guaranteed survival time that it did not cause. Time-varying exposure, landmark analysis, target-trial emulation, or another justified design can address the problem.

The eligibility date, treatment assignment date, time zero, and start of follow-up should align. A diagram of these dates often reveals the error more clearly than model output.

Competing risks require the right probability

When one event prevents another, standard survival methods can overstate the cumulative probability of the event of interest if competing events are censored as though they could still occur. Cause-specific hazards and cumulative incidence answer different questions. The method should match the decision and estimand.

For event type (k), cumulative incidence can be represented as:

$$ F_k(t)=\int_0^t S(u^-)\lambda_k(u),du $$

where (S(u^-)) is the probability of remaining free of any event immediately before (u), and (\lambda_k(u)) is the cause-specific hazard. Reporting one method as though it estimates the other is an interpretation error.

Multiplicity increases false-positive findings

Testing many outcomes, subgroups, time points, or model variants increases the chance of at least one apparently significant result. Selective reporting after inspecting results compounds the problem. A pre-specified endpoint hierarchy and appropriate multiplicity strategy should distinguish confirmatory from exploratory findings.

If (m) independent null hypotheses are each tested at level (\alpha), the probability of at least one false positive is:

$$ P(\text{at least one false positive})=1-(1-\alpha)^m $$

With 20 independent tests at 5%, this probability is approximately 64%. Independence is a simplifying assumption, but the calculation shows why nominal p-values cannot be interpreted without the testing context.

P-values and confidence intervals can be misinterpreted

A p-value is not the probability that the null hypothesis is true, and a 95% confidence interval is not a 95% probability that the fixed true value lies in the realised interval under a frequentist interpretation. Statistical significance does not establish clinical importance, causality, or absence of error. Estimates, uncertainty, assumptions, and outcome meaning should be reported together.

A non-significant result is not proof of no effect. Equivalence or non-inferiority requires pre-specified margins, appropriate design, and corresponding analysis.

Causal analysis errors arise from an incorrect causal structure

Adjustment can reduce confounding, create bias, or block part of the effect depending on the variable's causal role. Controlling for a mediator changes the estimand, conditioning on a collider can create an association, and adjusting for an instrument can amplify bias. Variable selection should follow a causal model rather than statistical significance alone.

Directed acyclic graphs, target-trial specifications, negative controls, and sensitivity analyses can clarify assumptions. No software procedure can identify an unmeasured causal structure without substantive knowledge.

Regression to the mean can be mistaken for improvement

Patients or organisations selected because of an extreme measurement tend to have less extreme values on repeat measurement even without intervention. Before-and-after analysis without a suitable comparator can attribute this natural movement to treatment. The problem is especially important when eligibility requires a high symptom score or poor performance.

Randomised comparison, repeated baseline measurements, interrupted time series, or appropriate modelling can reduce misinterpretation. Merely applying a statistical test to the pre-post difference does not solve the design problem.

Clustered and repeated data need dependent-error methods

Patients treated by the same clinician, site, school, or region can have correlated outcomes. Repeated observations within a patient are also correlated. Treating them as independent generally understates standard errors and can overstate precision.

Methods can include cluster-robust standard errors, mixed-effects models, generalised estimating equations, or designs that aggregate at the cluster level. The number of clusters and small-sample corrections matter; a large number of patients does not compensate for very few independent clusters.

Economic models have additional analytical failure points

Economic evaluation combines clinical effects, costs, utilities, time, and uncertainty, creating opportunities for unit, logic, and aggregation errors. A model can balance internally yet still implement the wrong comparator, perspective, or time horizon. Verification should test equations and patient flow as well as headline cost-effectiveness results.

Common economic-model errors include:

• Discounting at the wrong rate, timing, or unit of time.
• Applying a one-off cost in every cycle.
• Counting an adverse event in both state utility and a separate decrement.
• Using a relative effect on the wrong baseline risk.
• Combining nominal and real prices or different price years.
• Applying mortality twice or allowing dead patients to accrue costs.
• Calculating an average ratio instead of an incremental comparison.
• Failing to remove dominated and extendedly dominated strategies.

Incremental analysis can be implemented incorrectly

Cost-effectiveness should compare ordered non-dominated strategies incrementally. Comparing every option only with a common baseline can hide extended dominance and yield misleading ratios. The calculation should also distinguish ICER from incremental net monetary benefit.

For strategy (a) versus (b):

$$ ICER_{a,b}=\frac{C_a-C_b}{E_a-E_b} $$

$$ INMB_{a,b}=\lambda(E_a-E_b)-(C_a-C_b) $$

An ICER is not meaningful when its sign and quadrant are ignored. Positive INMB favours strategy (a) at threshold (\lambda), conditional on correct inputs and implementation.

Probabilistic analysis can be coded incorrectly

Probabilistic sensitivity analysis should sample uncertain parameters jointly from appropriate distributions and recalculate the model for each draw. Errors include sampling probabilities outside 0 to 1, ignoring parameter correlations, mixing standard errors with standard deviations, and resampling deterministic identities independently. A smooth cost-effectiveness acceptability curve does not prove the simulation is correct.

Checks should compare sampled means and quantiles with intended inputs, preserve logical constraints, test correlations, control Monte Carlo error, and reproduce results using a fixed random seed.

Spreadsheet errors deserve the same scrutiny as code errors

Spreadsheets can hide overwritten formulas, inconsistent ranges, copied references, hard-coded values, circular calculations, and stale external links. Visual plausibility is not verification. Formula maps, input-output separation, protected cells, automated checks, and independent recalculation improve assurance.

Specific checks include:

• Scanning for constants inside formula regions.
• Comparing formulas across repeated rows and columns.
• Testing named ranges and absolute versus relative references.
• Recalculating in the intended software engine.
• Reviewing hidden sheets, rows, columns, names, and links.
• Tracing key outputs back to source inputs.

Software defaults can silently change the analysis

Statistical packages differ in reference categories, missing-value handling, variance estimators, optimisation, continuity corrections, and definitions of quantiles. Version updates can also change behaviour. Code should set material options explicitly rather than relying on defaults.

The execution record should preserve software, package versions, operating environment, random seeds, locale, and numerical settings. Warnings and convergence messages should be retained and resolved rather than suppressed.

Output and transcription errors occur after correct analysis

Correct model results can be copied incorrectly into a table, rounded inconsistently, labelled with the wrong unit, or described in the wrong direction. Manually retyping results increases risk. Tables and figures should be generated directly from validated analysis objects where possible.

Automated checks should reconcile numbers across the main text, abstract, supplement, model report, and decision materials. Denominators, confidence intervals, and footnotes should travel with the estimate.

Interpretation can be erroneous even when computation is correct

An association can be described as causal, a subgroup finding can be treated as confirmed, a relative effect can be described as an absolute benefit, or a modelled extrapolation can be presented as observed. These are analysis errors because interpretation is part of the analytical chain. Review should compare every conclusion with the design, estimand, estimate, uncertainty, and limitations.

The direction of scales and outcomes should be explicit. A positive coefficient can mean improvement or deterioration depending on coding.

Independent reproduction is a powerful test

Reproduction reruns the analysis from the same data and code, while independent replication or parallel programming recreates key results using separate code or methods. Exact reproduction detects environment and workflow problems; independent calculation is more likely to find shared coding assumptions and transcription errors. Both should focus first on decision-critical outputs.

A minimum reproducibility package includes:

• Frozen input data or a governed extraction specification.
• Executable code in the correct order.
• Software environment and dependencies.
• Configuration, seeds, and scenario settings.
• Automated expected outputs and tolerances.
• A log linking each result to its program and data version.

Validation checks should be designed before results

Checks are strongest when their expected result is known independently of the main analysis. Unit tests, invariants, boundary cases, simulated data, negative controls, and hand calculations can expose errors that ordinary output review misses. Tests written after a surprising result can unconsciously accommodate the existing implementation.

Examples include:

• Probabilities sum to one and remain between zero and one.
• Cohort totals reconcile across states and time.
• Costs and outcomes become zero when their drivers are set to zero.
• Identical treatment arms produce zero incremental results.
• A known synthetic treatment effect is recovered within simulation error.
• Changing a parameter affects outputs in the expected direction.

Worked error example

Suppose an analysis reports costs in dollars but one hospital supplies its values in cents. The uncorrected mean cost for that hospital is 100 times too large and can dominate a pooled estimate. The error may not be obvious if the records are few or the hospital treats complex patients.

If the recorded value is (C_{cents}), the correct conversion is:

$$ C_{dollars}=\frac{C_{cents}}{100} $$

Range checks by site, unit metadata, comparison with invoices, and inspection of extreme values could detect the problem. Deleting the apparent outliers without identifying the unit mismatch would create a second analytical error.

Correcting an error requires impact assessment

Finding an error should trigger a controlled process rather than an undocumented patch. The team must determine its origin, affected analyses, materiality, root cause, and whether published or submitted conclusions change. Correction should preserve an audit trail and prevent recurrence.

1. Contain the issue. Stop release or use of affected outputs and preserve the current files.
2. Reproduce the error. Identify the exact data, code, assumptions, and versions involved.
3. Assess scope and materiality. Determine which estimates, tables, models, and decisions are affected.
4. Correct and independently verify. Implement the change with tests and separate review.
5. Communicate transparently. Notify relevant users, update outputs, and issue a correction or retraction when required.
6. Prevent recurrence. Add controls, documentation, training, or workflow changes that address the root cause.

Common mistakes

Analysis errors are sometimes treated as obvious coding failures, while subtler specification and interpretation problems escape review. Conversely, disagreement between reasonable models can be mislabelled as an error. The following distinctions support a fair and effective investigation.

• A surprising result is not proof of an error, but it should trigger verification.
• Successful software execution does not prove correct specification.
• A statistically significant result does not validate the data pipeline or causal interpretation.
• Repeating the same code does not independently verify it.
• Agreement with expectations can conceal an error that happens to offset another error.
• Removing outliers without identifying their source can hide data or unit problems.
• Changing the analysis after seeing results is not automatically wrong, but it must be labelled and justified.
• A methodological judgment between plausible alternatives is uncertainty unless one violates the stated design or evidence.
• Correcting the headline number without tracing downstream outputs leaves the analysis unsafe.

Reporting and preventing analysis errors

Transparent reporting makes both prevention and correction easier. An analysis should preserve the planned methods, exact execution, deviations, quality checks, and links from data to conclusions. When an error is found, the record should explain its effect without minimising or overstating it.

• Register or date-stamp the protocol and statistical or economic analysis plan.
• Use version control for data specifications, code, models, and outputs.
• Separate raw data, derived data, analysis, and presentation layers.
• Automate validation, reconciliation, and regression tests.
• Require independent review for decision-critical code and results.
• Record software, packages, seeds, settings, warnings, and execution logs.
• Report deviations, sensitivity analyses, and exploratory work clearly.
• Publish corrections with affected results, revised conclusions, and prevention steps.

The decision standard

An analysis is trustworthy when its question, data transformation, model, computation, uncertainty, and interpretation form a traceable and verified chain. Error control does not require pretending that one model is uniquely correct; it requires distinguishing legitimate uncertainty from preventable mistakes. When an error is found, credibility depends on transparent correction, independent verification, and safeguards that stop the same failure from reaching another decision.