This vignette accompanies development version 0.99.2. The benchmark
covers selected configurations, and the validation study uses 100
datasets per metric with 99 resampling draws. These small simulations
provide diagnostics; they do not establish general statistical
calibration. The local study observed 24% peak false positives, 10% peak
detection, 74.9% transfer interval coverage, and 43% drift interval
coverage in the tested scenarios. These results limit inferential use of
these methods. Report sources are scripts/benchmark.R and
scripts/validate.R.
The eight Category C functions in crossStatR answer a specific kind of question: where does measurement uncertainty come from, and how does it propagate through the analysis pipeline? A single-point LOD is not enough. A drift-corrected mean is not enough. The full answer requires attributing variance to stages (sampling, instrument noise, calibration, drift, model selection), quantifying how robust the final result is to defensible analysis decisions, and producing concentration-dependent reliability curves rather than one number.
This vignette walks through the five Category C methods considered
most useful in practice: drift_uncertainty(),
calibration_transfer_risk(),
detection_reliability(),
experiment_fragility(), and
uncertainty_origin(). The example uses a synthetic
electrochemical sensor case throughout. These are diagnostic tools that
quantify how much the analyst should trust a reported number, not
confirmatory tests.
The example simulates a sensor that produces a linear response to a
known analyte concentration, with exponential drift in the baseline. The
example uses 30 calibration points spread over concentrations 0.1 to 10
µM and 60 “unknown” measurements taken over a 10-hour window with drift.
The true calibration is y = 2 + 3 * x, and the true drift
is 0.5 * (1 - exp(-0.3 * t)) with t in
hours.
set.seed(606)
# Calibration data
x_cal <- seq(0.1, 10, length.out = 30)
y_cal <- 2 + 3 * x_cal + rnorm(length(x_cal), sd = 0.25)
source_cal <- lm(y_cal ~ x_cal)
# Unknown measurements with drift
n_unk <- 60
t_unk <- seq(0, 10, length.out = n_unk)
drift_true <- 0.5 * (1 - exp(-0.3 * t_unk))
# True concentrations of unknowns (uniform on [0.5, 5])
x_true_unk <- runif(n_unk, 0.5, 5)
y_unk <- 2 + 3 * x_true_unk + drift_true + rnorm(n_unk, sd = 0.25)
cal_df <- data.frame(x = x_cal, y = y_cal)
unk_df <- data.frame(x = x_true_unk, y = y_unk, t = t_unk)drift_uncertainty() decomposes the total measurement
variance into a component attributable to drift and a residual noise
component. Drift is modelled as a smooth function of time, fit jointly
with calibration parameters when concentrations are supplied. The drift
variance is the variance of the fitted drift function evaluated at the
observed times; the noise variance is the residual variance of the joint
fit. The fraction of variance attributed to drift is
f_drift = sigma^2_drift / (sigma^2_drift + sigma^2_epsilon).
dru <- drift_uncertainty(y = y_unk, time = t_unk,
conc = x_true_unk,
drift_model = "exponential",
n_boot = 200, seed = 1)
print(dru)
#> <crossStatR::drift_uncertainty> Drift uncertainty (exponential drift model)
#> Drift variance: 0 (0.0% of total)
#> Noise variance: 0.07006
#> Drift fraction: 0.000 (95% CI [0.003, 0.005])
#> Lag-1 ACF of residual: -0.184 (critical: 0.258)The function reports the drift variance fraction with a bootstrap percentile CI. Read it as: of the total variance in the unknown measurements, roughly the reported fraction is attributable to drift under the assumed exponential model. The bootstrap CI assumes the drift model is correctly specified.
When is this appropriate? When the analyst has a smooth drift process
and known concentrations. When is it not? When drift and concentration
are confounded. The function warns if cor(time, conc)
exceeds 0.5 in absolute value; in that case the drift estimate is
biased.
Suppose the analyst built the sensor calibration on instrument A and
now wants to use it on instrument B without full recalibration. The
example measures a few shared transfer samples and apply a simple offset
correction. calibration_transfer_risk() quantifies the
prediction error, an out-of- distribution flag for unknown samples, and
an uncertainty interval that combines source calibration uncertainty,
transfer-correction uncertainty, and target measurement noise via a
parametric bootstrap.
# Simulate a target instrument with offset + small slope change
y_target <- 2.4 + 2.85 * x_cal + rnorm(length(x_cal), sd = 0.25)
target_df <- data.frame(x = x_cal, y = y_target)
ctr <- calibration_transfer_risk(source_cal = source_cal,
source_data = cal_df,
target_data = target_df,
transfer_samples = 6L,
transfer_model = "slope_offset",
n_boot = 200, seed = 1)
print(ctr)
#> <crossStatR::calibration_transfer_risk> Calibration transfer (slope_offset correction, 6 transfer samples)
#> TPE = 0.08127 (acceptable: < 0.1751): PASS
#> OOD rate = 3% (acceptable: < 5%): PASS
#> Overall transfer acceptable: YESThe transfer prediction error (TPE) indicates how far off concentration estimates on the target instrument will be, on average. The out-of-distribution rate flags samples that do not behave like the transfer set (matrix effects, sensor drift, different analyte). A transfer is “acceptable” if TPE is below a user-supplied tolerance (default: 2x the source calibration standard error) and the OOD rate on the evaluation set is below 5%. The documentation does not recommend reporting a transferred calibration without this check.
When is this appropriate? When source and target instruments share the same chemistry and analyte. When is it not? When the source calibration itself is biased, or when the transfer samples are not representative of the unknowns.
The single-point LOD reduces the entire detection performance of a
sensor to one concentration. detection_reliability()
produces the full curve: probability of detection (PD), false-positive
rate (FPR), and false-negative rate (FNR), each as a function of
concentration, with bootstrap confidence bands. Variance from matrix
effects, batch effects, or drift can be incorporated via the
extra_sigma2 argument, which adds an extra variance
contribution to the calibration noise variance.
dr <- detection_reliability(calibration = source_cal,
conc_grid = seq(0, 5, length.out = 30),
n_boot = 200, seed = 1)
print(dr)
#> <crossStatR::detection_reliability> Detection reliability curve
#> Decision threshold L_C = 2.445, sigma_eff = 0.2644
#> LOD (PD=0.95): 0.3156
#> Theoretical FPR: 0.0250 (95% CI [0.0009, 0.1092])
#>
#> Reliability curve (first 6 + last 6 rows):
#> Low concentration:
#> concentration mean_response pd pd_ci_low pd_ci_high fnr
#> 0.0000000 1.926915 0.0250000 0.000869807 0.1092241 9.750000e-01
#> 0.1724138 2.447530 0.5036805 0.217012501 0.7524123 4.963195e-01
#> 0.3448276 2.968145 0.9760591 0.900278179 0.9982547 2.394092e-02
#> 0.5172414 3.488760 0.9999605 0.997944956 1.0000000 3.946838e-05
#> 0.6896552 4.009375 1.0000000 0.999996981 1.0000000 1.641375e-09
#> 0.8620690 4.529990 1.0000000 1.000000000 1.0000000 1.554312e-15
#> High concentration:
#> concentration mean_response pd pd_ci_low pd_ci_high fnr
#> 4.137931 14.42167 1 1 1 0
#> 4.310345 14.94229 1 1 1 0
#> 4.482759 15.46290 1 1 1 0
#> 4.655172 15.98352 1 1 1 0
#> 4.827586 16.50413 1 1 1 0
#> 5.000000 17.02475 1 1 1 0
# Plot PD/FNR curves
curves <- dr$curve
ggplot(curves, aes(x = concentration)) +
geom_line(aes(y = pd, colour = "PD")) +
geom_line(aes(y = fnr, colour = "FNR")) +
geom_hline(yintercept = 0.95, linetype = "dashed", colour = "grey50") +
labs(x = "Concentration", y = "Probability",
colour = NULL) +
theme_bw()The reliability curve shows where the sensor actually works. If the PD curve reaches 0.95 only at a concentration above the regulatory threshold, the sensor is not adequate for the intended use, regardless of what the single-point LOD number says.
When is this appropriate? For analyses of detection performance across a range of concentrations. When is it not? When the calibration is not linear in the relevant range, or when the Gaussian noise assumption is badly wrong.
experiment_fragility() quantifies how sensitive a
reported result is to defensible analysis decisions: which replicates to
include, how to handle outliers, which baseline correction to use, which
model to fit, and how much measurement uncertainty to inject before the
result loses significance. It is a sensitivity analysis, not a
confirmatory test.
# A simple analysis function: returns the slope of the calibration.
# experiment_fragility passes a list with a 'data' element.
analysis_fn <- function(opts) {
d <- opts$data
fit <- lm(y ~ x, data = d)
unname(coef(fit)[2])
}
fg <- experiment_fragility(
analysis_fn = analysis_fn,
data = cal_df,
replicates = NULL,
outlier_methods = c("none", "iqr", "mad"),
measurement_uncertainties = c(0, 0.05, 0.1, 0.2, 0.5),
decision_threshold = 3.0,
n_boot = 100, seed = 1
)
print(fg)
#> <crossStatR::experiment_fragility> Sensitivity summary
#> Original result: 3.02 (threshold: 3, decision: above)
#>
#> Component fragility (fraction flipping decision):
#> replicate n/a
#> outlier 0.00
#> baseline 0.00
#> model n/a
#>
#> Overall fragility score: 0.000
#>
#> Most influential decisions:
#> decision swing
#> outlier_method 0
#> baseline_method 0
#> leave_one_replicate NA
#> model_alternative NAThe fragility score is a weighted sum of decision-flip fractions
across perturbation types. The “noise fragility threshold” is the
smallest injected-noise SD at which the decision (here, “slope = 3.0”)
flips. The reported most_influential_decision indicates
which perturbation type produces the largest swing in the result.
When is this appropriate? As an exploratory check before committing to a result. When is it not? As a confirmatory test. The fragility score has no known reference distribution; do not treat it as a p-value. The binary fragility index is less informative than the continuous range across perturbations.
uncertainty_origin() allocates the variance of an
additive total using supplied stage variances and covariances. Each
covariance is shared equally between the two stages involved.
Independent stages reduce to the usual additive decomposition. This
subset-sum variance game differs from the conditional-expectation
Shapley effects studied by Owen and Prieur (2017) and Song, Nelson and
Staum (2016). The function does not estimate the stage variances or
implement nonlinear propagation. Its total_ci is a quantile
interval for simulated additive errors, rather than a confidence
interval for a variance component.
# The example uses the following variance components from prior analyses:
stages <- c(
sampling = 0.01,
instrument_noise = 0.04,
calibration = 0.02,
drift = as.numeric(dru$sigma2_drift),
replicate = 0.005
)
# A few correlations between stages (illustrative)
correlations <- list(
sampling__instrument_noise = 0.1,
calibration__drift = 0.2
)
uo <- uncertainty_origin(stages = stages,
correlations = correlations,
n_mc = 1000, seed = 1)
print(uo)
#> <crossStatR::uncertainty_origin> Uncertainty-origin attribution framework
#> Total variance: 0.075 (additive), 0.079 (correlated)
#> Total variance (MC, n=1000): 0.08917 (MCSE ~ 0.006971)
#>
#> Attribution:
#> stage variance pct_additive pct_shapley
#> sampling 0.010 0.13333333 0.15189873
#> instrument_noise 0.040 0.53333333 0.53164557
#> calibration 0.020 0.26666667 0.25316456
#> drift 0.000 0.00000000 0.00000000
#> replicate 0.005 0.06666667 0.06329114The output reports each stage’s contribution to total variance, with
the cross-term variance fairly allocated. For independent stages the
Shapley attribution reduces to the obvious
sigma^2_k / sum(sigma^2).
When is this appropriate? When the analyst has already estimated the stage variances with the other Category B and C functions and want a final attribution table. When is it not? When the stage variances are themselves noisy estimates, or when the correlations are unknown.
The Category C functions are diagnostic tools. Their results depend on model specification, independence assumptions, and the estimated input variances. Residual autocorrelation requires inspection. Small simulation studies cannot establish confidence-interval coverage across experimental settings.
drift_uncertainty() separates drift from residual
variance when calibration information permits that separation.
calibration_transfer_risk() assesses a proposed transfer;
detection_reliability() describes concentration-dependent
detection. experiment_fragility() checks sensitivity to
analysis decisions, and uncertainty_origin() allocates
variance using the supplied covariance structure.
Currie, L. A. (1999). Detection and quantification limits: origins and historical overview. Analytica Chimica Acta, 391, 127-134. Source.
Davison, A. C. and Hinkley, D. V. (1997). Bootstrap Methods and their Application. Cambridge University Press. Source.
Owen, A. B. and Prieur, C. (2017). On Shapley Value for Measuring Importance of Dependent Inputs. SIAM/ASA Journal on Uncertainty Quantification, 5, 986-1002. Source.
Song, E., Nelson, B. L. and Staum, J. (2016). Shapley effects for global sensitivity analysis: Theory and computation. SIAM/ASA Journal on Uncertainty Quantification, 4, 1060-1083. Source.
Steegen, S., Tuerlinckx, F., Gelman, A. and Vanpaemel, W. (2016). Increasing Transparency Through a Multiverse Analysis. Perspectives on Psychological Science, 11, 702-712. Source.
Walsh, M., Srinathan, S. K., McAuley, D. F. et al. (2014). The statistical significance of randomized controlled trial results is frequently fragile: a case for a Fragility Index. Journal of Clinical Epidemiology, 67, 622-628. Source.