This vignette accompanies development version 0.99.2. The benchmark
covers selected configurations, and the validation study uses 100
datasets per metric with 99 resampling draws. These small simulations
provide diagnostics; they do not establish general statistical
calibration. The local study observed 24% peak false positives, 10% peak
detection, 74.9% transfer interval coverage, and 43% drift interval
coverage in the tested scenarios. These results limit inferential use of
these methods. Report sources are scripts/benchmark.R and
scripts/validate.R.
Most biological measurements are taken on a small number of
biological units (animals, cell cultures, donor samples), each measured
several times. The several-per-unit measurements are technical
replicates. They share instrument noise and sample-handling artefacts
with the other measurements on the same unit, so they are correlated.
Treating them as independent inflates the apparent sample size and
shrinks the standard error by roughly a factor of \sqrt{k},
where k is the number of technical replicates per
biological unit.
This is the standard pseudo-replication failure described by Vaux,
Fidler and Cumming (2012). It is the single most common statistical
mistake in biological experimental data reported in experimental
practice. crossStatR makes the correct workflow the default: pass a
bio_rep identifier (and optionally a tech_rep
identifier) to sci_describe(), and the function averages
the technical replicates within each biological unit before computing
summary statistics or fitting tests.
The example uses a small dose-response example that resembles an enzyme-kinetics assay. The example simulates three biological replicates per dose condition, with two technical replicates per biological unit. The two technical replicates share a biological-unit-level random effect so they are positively correlated.
set.seed(101)
doses <- c(0, 1, 5, 25) # µM, nominal
n_bio <- 6 # biological replicates per dose
n_tech <- 2 # technical replicates per bio unit
mu <- c(2.0, 2.6, 3.4, 4.0) # true means per dose
sigma_bio <- 0.25 # between-bio SD
sigma_tech <- 0.07 # within-bio, between-tech SD
rows <- list()
for (d in seq_along(doses)) {
for (b in seq_len(n_bio)) {
bio_effect <- rnorm(1, mean = mu[d], sd = sigma_bio)
for (t in seq_len(n_tech)) {
rows[[length(rows) + 1L]] <- data.frame(
dose = doses[d], dose_f = factor(doses[d]),
bio = paste0("d", d, "_b", b),
tech = t,
y = bio_effect + rnorm(1, sd = sigma_tech)
)
}
}
}
df <- do.call(rbind, rows)
head(df, 4)
#> dose dose_f bio tech y
#> 1 0 0 d1_b1 1 1.957163
#> 2 0 0 d1_b1 2 1.871245
#> 3 0 0 d1_b2 1 2.075344
#> 4 0 0 d1_b2 2 2.135768The correct call averages technical replicates within each biological unit before computing per-dose means and standard errors:
desc <- sci_describe(df$y, group = df$dose_f,
bio_rep = df$bio, tech_rep = df$tech)
print(desc)
#> <crossStatR::describe> Descriptive statistics (bio-replicate level)
#> Total observations: 48 Biological replicates: 24
#> group n mean sd var se median mad cv
#> 0 6 2.059781 0.1605761 0.02578467 0.06555490 2.020173 0.1498705 0.07795784
#> 1 6 2.559742 0.2936013 0.08620174 0.11986224 2.660236 0.2234282 0.11469959
#> 5 6 3.295095 0.2852819 0.08138578 0.11646586 3.359209 0.1932193 0.08657775
#> 25 6 4.051093 0.1803273 0.03251795 0.07361833 4.073642 0.2172382 0.04451326
#> iqr mean_ci_low mean_ci_high median_ci_low median_ci_high n_missing
#> 0.2368491 1.891267 2.228295 1.914204 2.297316 0
#> 0.3543908 2.251626 2.867857 2.098015 2.868780 0
#> 0.2237330 2.995710 3.594480 2.765397 3.555541 0
#> 0.2283861 3.861851 4.240335 3.831244 4.308557 0The reported n per dose is 6, the number of biological
replicates. The standard error is the between-bio-unit standard error.
Compare this with the naive call that ignores the replicate
structure:
naive <- sci_describe(df$y, group = df$dose_f)
print(naive)
#> <crossStatR::describe> Descriptive statistics
#> Total observations: 48 Biological replicates: n/a
#> group n mean sd var se median mad
#> 0 12 2.059781 0.1583916 0.02508789 0.04572371 2.028190 0.1887523
#> 1 12 2.559742 0.2849494 0.08119618 0.08225782 2.639848 0.2834606
#> 5 12 3.295095 0.2772160 0.07684872 0.08002537 3.348409 0.2096497
#> 25 12 4.051093 0.1749491 0.03060720 0.05050347 4.072342 0.2273265
#> cv iqr mean_ci_low mean_ci_high median_ci_low median_ci_high
#> 0.07689730 0.2249627 1.959144 2.160418 1.888546 2.254262
#> 0.11131961 0.4223750 2.378693 2.740790 2.165013 2.843275
#> 0.08412989 0.2838532 3.118961 3.471230 2.788003 3.545889
#> 0.04318567 0.2577371 3.939935 4.162250 3.836267 4.252502
#> n_missing
#> 0
#> 0
#> 0
#> 0The naive call reports n = 12 per dose. The standard
error is roughly half of the correct value, because it confuses
within-bio noise with between-bio variation. Any p-value computed
downstream from the naive summary will be far too small.
The documentation does not recommend reporting the naive summary in a
paper, even as a supplementary table. The temptation to compare the two
summaries and pick the one that supports the hypothesis is exactly the
kind of analysis-decision sensitivity that
experiment_fragility() is designed to expose (see the
Advanced Uncertainty Attribution vignette).
A honest plot shows each biological replicate as a point (after averaging its technical replicates) and overlays the group mean. Plotting all 24 raw points as if they were independent is the visual analogue of pseudo-replication.
bio_means <- aggregate(y ~ dose_f + bio, data = df, FUN = mean)
group_summary <- desc$summary
group_summary$dose_f <- factor(group_summary$group)
ggplot(bio_means, aes(x = dose_f, y = y)) +
geom_jitter(width = 0.08, alpha = 0.6, size = 2) +
geom_pointrange(data = group_summary,
aes(y = mean, ymin = mean_ci_low, ymax = mean_ci_high),
colour = "black", size = 0.6, fatten = 2.5,
shape = 18) +
labs(x = "Dose (µM)", y = "Response (a.u.)",
title = "Dose-response: 6 biological replicates per dose") +
theme_bw()
#> Warning: The `fatten` argument of `geom_pointrange()` is deprecated as of ggplot2 4.0.0.
#> ℹ Please use the `size` aesthetic instead.
#> This warning is displayed once per session.
#> Call `lifecycle::last_lifecycle_warnings()` to see where this warning was
#> generated.Dose-response with biological replicates shown as points (technical replicates pre-averaged). Black diamonds are group means; error bars are 95% CIs on the mean.
To compare two dose levels the example calls
sci_compare(). The default test is Welch’s t, which the
example prefers over Student’s t in essentially all experimental
settings because the equal-variance assumption rarely holds. If the
distributions are visibly non-Gaussian (small n, heavy
tails), use the permutation test instead; it does not require a Gaussian
null and it controls Type I error under exchangeability.
sub <- df[df$dose %in% c(1, 5), ]
bio_means2 <- aggregate(y ~ dose_f + bio, data = sub, FUN = mean)
x <- bio_means2$y[bio_means2$dose_f == "1"]
y <- bio_means2$y[bio_means2$dose_f == "5"]
sci_compare(x, y, test = "welch")
#> <crossStatR::compare> Comparison: welch
#> statistic = -4.4000, p = 0.00133775
#> effect (mean(x) - mean(y)) = -0.7354
#> Effect sizes:
#> name estimate se ci_low ci_high
#> Cohen_d -2.540331 0.8180058 -4.143593 -0.9370693
#> Hedges_g -2.344921 0.7905300 NA NAThe example deliberately computes the comparison on the per-bio-unit means rather than on the raw 12-row subset. The two-level averaging (technical replicates within bio units, then compare bio means) is what justifies the Welch test’s independence assumption.
A one-way ANOVA across all four doses asks whether the means differ
anywhere. The example uses the default aov path with Tukey
HSD post-hoc and report the effect-size estimates.
bio_means_all <- aggregate(y ~ dose_f + bio, data = df, FUN = mean)
fit <- sci_anova(y ~ dose_f, data = bio_means_all,
posthoc = "tukey")
print(fit)
#> <crossStatR::anova> ANOVA
#> Df Sum Sq Mean Sq F value Pr(>F)
#> dose_f 3 13.616536 4.53884542 80.37262 2.480147e-11
#> Residuals 20 1.129451 0.05647254 NA NA
#>
#> Effect sizes:
#> effect estimate interpretation
#> eta_squared 0.9234062 large
#> partial_eta_squared 0.9234062 large
#> omega_squared 0.9084381 large
#>
#> Levene's test p = 0.7600The effect_sizes table reports eta-squared, partial
eta-squared, and omega-squared. The example prefers omega-squared for
one-way designs because it is less biased upwards in small samples
(Kelley, 1935; Carroll and Nordholm, 1975). The Levene’s test p-value is
reported so the analyst can decide whether to refit with
welch = TRUE; if the variances look equal across groups the
classic F-test is fine, and if they do not, refit:
fit_w <- sci_anova(y ~ dose_f, data = bio_means_all, welch = TRUE)
print(fit_w)
#> <crossStatR::anova> Welch's ANOVA
#> Df Sum Sq Mean Sq F value Pr(>F)
#> num df 3 NA NA 126.8305 1.045475e-08
#>
#> Effect sizes:
#> effect estimate interpretation
#> eta_squared NA <NA>
#> partial_eta_squared NA <NA>
#> omega_squared NA <NA>
#>
#> Levene's test p = 0.7600A p-value alone is uninformative about the magnitude of the difference. The example recommends reporting a standardised effect size alongside the test. For the two-group comparison above:
sci_effect_size(x, y, type = "cohens_d")
#> <crossStatR::effect_size> Cohen's d (independent)
#> estimate = -2.5403, se = 0.8180
#> CI: [-4.1436, -0.9371]
#> interpretation: large
sci_effect_size(x, y, type = "hedges_g")
#> <crossStatR::effect_size> Hedges' g (independent)
#> estimate = -2.3449, se = 0.7905
#> CI: [-3.8943, -0.7955]
#> interpretation: largeThe Hedges’ g applies a small-sample bias correction to
Cohen’s d. For n = 6 per group, the correction
is roughly 6%, which is large enough to matter. The example prefers
g for small samples and d for sample sizes
above roughly 20 per group.
If the biological units are themselves clustered (mice within a cage,
cells within a passage, donor samples within a batch), the simple
average-then-test workflow above is no longer adequate because it
ignores the cluster-level correlation. Use
replicate_decompose() instead. It fits a linear mixed model
with lme4::lmer and reports variance components for each
level of replication.
If the analyst wants to attribute variance across several
experimental stages (sampling, instrument noise, calibration, drift),
see the Advanced Uncertainty Attribution vignette. The Category
C functions uncertainty_origin() and
drift_uncertainty() are designed for that question.
Three things. First, average technical replicates within biological
units before any inferential statistics; sci_describe()
does this for the analyst when the analyst supplies bio_rep
and tech_rep. Second, default to Welch’s t unless the
analyst have a stated reason to use Student’s, and use the permutation
test when the Gaussian assumption is doubtful. Third, report effect
sizes with confidence intervals, not just p-values, and prefer Hedges’
g for small samples.
Carroll, R. M. and Nordholm, L. A. (1975). Sampling characteristics of Kelley’s epsilon-squared and Hays’ omega-squared. Educational and Psychological Measurement, 35, 541-554. Source.
Kelley, T. L. (1935). An Unbiased Correlation Ratio Measure. Proceedings of the National Academy of Sciences, 21, 554-559. Source.
Vaux, D. L., Fidler, F. and Cumming, G. (2012). Replicates and repeats—what is the difference and is it significant? The EMBO Reports, 13, 291-296. Source.