Biological Experiment Statistics

library(crossStatR)
library(ggplot2)

Development validation

This vignette accompanies development version 0.99.2. The benchmark covers selected configurations, and the validation study uses 100 datasets per metric with 99 resampling draws. These small simulations provide diagnostics; they do not establish general statistical calibration. The local study observed 24% peak false positives, 10% peak detection, 74.9% transfer interval coverage, and 43% drift interval coverage in the tested scenarios. These results limit inferential use of these methods. Report sources are scripts/benchmark.R and scripts/validate.R.

The pseudo-replication problem

Most biological measurements are taken on a small number of biological units (animals, cell cultures, donor samples), each measured several times. The several-per-unit measurements are technical replicates. They share instrument noise and sample-handling artefacts with the other measurements on the same unit, so they are correlated. Treating them as independent inflates the apparent sample size and shrinks the standard error by roughly a factor of \sqrt{k}, where k is the number of technical replicates per biological unit.

This is the standard pseudo-replication failure described by Vaux, Fidler and Cumming (2012). It is the single most common statistical mistake in biological experimental data reported in experimental practice. crossStatR makes the correct workflow the default: pass a bio_rep identifier (and optionally a tech_rep identifier) to sci_describe(), and the function averages the technical replicates within each biological unit before computing summary statistics or fitting tests.

The example uses a small dose-response example that resembles an enzyme-kinetics assay. The example simulates three biological replicates per dose condition, with two technical replicates per biological unit. The two technical replicates share a biological-unit-level random effect so they are positively correlated.

Simulating a dose-response experiment

set.seed(101)
doses <- c(0, 1, 5, 25)              # µM, nominal
n_bio <- 6                            # biological replicates per dose
n_tech <- 2                           # technical replicates per bio unit
mu <- c(2.0, 2.6, 3.4, 4.0)           # true means per dose
sigma_bio <- 0.25                     # between-bio SD
sigma_tech <- 0.07                    # within-bio, between-tech SD

rows <- list()
for (d in seq_along(doses)) {
  for (b in seq_len(n_bio)) {
    bio_effect <- rnorm(1, mean = mu[d], sd = sigma_bio)
    for (t in seq_len(n_tech)) {
      rows[[length(rows) + 1L]] <- data.frame(
        dose = doses[d], dose_f = factor(doses[d]),
        bio = paste0("d", d, "_b", b),
        tech = t,
        y = bio_effect + rnorm(1, sd = sigma_tech)
      )
    }
  }
}
df <- do.call(rbind, rows)
head(df, 4)
#>   dose dose_f   bio tech        y
#> 1    0      0 d1_b1    1 1.957163
#> 2    0      0 d1_b1    2 1.871245
#> 3    0      0 d1_b2    1 2.075344
#> 4    0      0 d1_b2    2 2.135768

Descriptive statistics at the right level

The correct call averages technical replicates within each biological unit before computing per-dose means and standard errors:

desc <- sci_describe(df$y, group = df$dose_f,
                    bio_rep = df$bio, tech_rep = df$tech)
print(desc)
#> <crossStatR::describe> Descriptive statistics (bio-replicate level)
#> Total observations: 48  Biological replicates: 24
#>  group n     mean        sd        var         se   median       mad         cv
#>      0 6 2.059781 0.1605761 0.02578467 0.06555490 2.020173 0.1498705 0.07795784
#>      1 6 2.559742 0.2936013 0.08620174 0.11986224 2.660236 0.2234282 0.11469959
#>      5 6 3.295095 0.2852819 0.08138578 0.11646586 3.359209 0.1932193 0.08657775
#>     25 6 4.051093 0.1803273 0.03251795 0.07361833 4.073642 0.2172382 0.04451326
#>        iqr mean_ci_low mean_ci_high median_ci_low median_ci_high n_missing
#>  0.2368491    1.891267     2.228295      1.914204       2.297316         0
#>  0.3543908    2.251626     2.867857      2.098015       2.868780         0
#>  0.2237330    2.995710     3.594480      2.765397       3.555541         0
#>  0.2283861    3.861851     4.240335      3.831244       4.308557         0

The reported n per dose is 6, the number of biological replicates. The standard error is the between-bio-unit standard error. Compare this with the naive call that ignores the replicate structure:

naive <- sci_describe(df$y, group = df$dose_f)
print(naive)
#> <crossStatR::describe> Descriptive statistics
#> Total observations: 48  Biological replicates: n/a
#>  group  n     mean        sd        var         se   median       mad
#>      0 12 2.059781 0.1583916 0.02508789 0.04572371 2.028190 0.1887523
#>      1 12 2.559742 0.2849494 0.08119618 0.08225782 2.639848 0.2834606
#>      5 12 3.295095 0.2772160 0.07684872 0.08002537 3.348409 0.2096497
#>     25 12 4.051093 0.1749491 0.03060720 0.05050347 4.072342 0.2273265
#>          cv       iqr mean_ci_low mean_ci_high median_ci_low median_ci_high
#>  0.07689730 0.2249627    1.959144     2.160418      1.888546       2.254262
#>  0.11131961 0.4223750    2.378693     2.740790      2.165013       2.843275
#>  0.08412989 0.2838532    3.118961     3.471230      2.788003       3.545889
#>  0.04318567 0.2577371    3.939935     4.162250      3.836267       4.252502
#>  n_missing
#>          0
#>          0
#>          0
#>          0

The naive call reports n = 12 per dose. The standard error is roughly half of the correct value, because it confuses within-bio noise with between-bio variation. Any p-value computed downstream from the naive summary will be far too small.

The documentation does not recommend reporting the naive summary in a paper, even as a supplementary table. The temptation to compare the two summaries and pick the one that supports the hypothesis is exactly the kind of analysis-decision sensitivity that experiment_fragility() is designed to expose (see the Advanced Uncertainty Attribution vignette).

Plotting the data honestly

A honest plot shows each biological replicate as a point (after averaging its technical replicates) and overlays the group mean. Plotting all 24 raw points as if they were independent is the visual analogue of pseudo-replication.

bio_means <- aggregate(y ~ dose_f + bio, data = df, FUN = mean)
group_summary <- desc$summary
group_summary$dose_f <- factor(group_summary$group)

ggplot(bio_means, aes(x = dose_f, y = y)) +
  geom_jitter(width = 0.08, alpha = 0.6, size = 2) +
  geom_pointrange(data = group_summary,
                  aes(y = mean, ymin = mean_ci_low, ymax = mean_ci_high),
                  colour = "black", size = 0.6, fatten = 2.5,
                  shape = 18) +
  labs(x = "Dose (µM)", y = "Response (a.u.)",
       title = "Dose-response: 6 biological replicates per dose") +
  theme_bw()
#> Warning: The `fatten` argument of `geom_pointrange()` is deprecated as of ggplot2 4.0.0.
#> ℹ Please use the `size` aesthetic instead.
#> This warning is displayed once per session.
#> Call `lifecycle::last_lifecycle_warnings()` to see where this warning was
#> generated.
Dose-response with biological replicates shown as points (technical replicates pre-averaged). Black diamonds are group means; error bars are 95% CIs on the mean.

Dose-response with biological replicates shown as points (technical replicates pre-averaged). Black diamonds are group means; error bars are 95% CIs on the mean.

Two-group comparison

To compare two dose levels the example calls sci_compare(). The default test is Welch’s t, which the example prefers over Student’s t in essentially all experimental settings because the equal-variance assumption rarely holds. If the distributions are visibly non-Gaussian (small n, heavy tails), use the permutation test instead; it does not require a Gaussian null and it controls Type I error under exchangeability.

sub <- df[df$dose %in% c(1, 5), ]
bio_means2 <- aggregate(y ~ dose_f + bio, data = sub, FUN = mean)
x <- bio_means2$y[bio_means2$dose_f == "1"]
y <- bio_means2$y[bio_means2$dose_f == "5"]
sci_compare(x, y, test = "welch")
#> <crossStatR::compare> Comparison: welch
#> statistic = -4.4000,  p = 0.00133775
#> effect (mean(x) - mean(y)) = -0.7354
#> Effect sizes:
#>      name  estimate        se    ci_low    ci_high
#>   Cohen_d -2.540331 0.8180058 -4.143593 -0.9370693
#>  Hedges_g -2.344921 0.7905300        NA         NA

The example deliberately computes the comparison on the per-bio-unit means rather than on the raw 12-row subset. The two-level averaging (technical replicates within bio units, then compare bio means) is what justifies the Welch test’s independence assumption.

ANOVA across all doses

A one-way ANOVA across all four doses asks whether the means differ anywhere. The example uses the default aov path with Tukey HSD post-hoc and report the effect-size estimates.

bio_means_all <- aggregate(y ~ dose_f + bio, data = df, FUN = mean)
fit <- sci_anova(y ~ dose_f, data = bio_means_all,
                posthoc = "tukey")
print(fit)
#> <crossStatR::anova> ANOVA
#>             Df    Sum Sq    Mean Sq  F value       Pr(>F)
#> dose_f       3 13.616536 4.53884542 80.37262 2.480147e-11
#> Residuals   20  1.129451 0.05647254       NA           NA
#> 
#> Effect sizes:
#>               effect  estimate interpretation
#>          eta_squared 0.9234062          large
#>  partial_eta_squared 0.9234062          large
#>        omega_squared 0.9084381          large
#> 
#> Levene's test p = 0.7600

The effect_sizes table reports eta-squared, partial eta-squared, and omega-squared. The example prefers omega-squared for one-way designs because it is less biased upwards in small samples (Kelley, 1935; Carroll and Nordholm, 1975). The Levene’s test p-value is reported so the analyst can decide whether to refit with welch = TRUE; if the variances look equal across groups the classic F-test is fine, and if they do not, refit:

fit_w <- sci_anova(y ~ dose_f, data = bio_means_all, welch = TRUE)
print(fit_w)
#> <crossStatR::anova> Welch's ANOVA
#>        Df Sum Sq Mean Sq  F value       Pr(>F)
#> num df  3     NA      NA 126.8305 1.045475e-08
#> 
#> Effect sizes:
#>               effect estimate interpretation
#>          eta_squared       NA           <NA>
#>  partial_eta_squared       NA           <NA>
#>        omega_squared       NA           <NA>
#> 
#> Levene's test p = 0.7600

Effect size with confidence interval

A p-value alone is uninformative about the magnitude of the difference. The example recommends reporting a standardised effect size alongside the test. For the two-group comparison above:

sci_effect_size(x, y, type = "cohens_d")
#> <crossStatR::effect_size> Cohen's d (independent)
#> estimate = -2.5403,  se = 0.8180
#> CI: [-4.1436, -0.9371]
#> interpretation: large
sci_effect_size(x, y, type = "hedges_g")
#> <crossStatR::effect_size> Hedges' g (independent)
#> estimate = -2.3449,  se = 0.7905
#> CI: [-3.8943, -0.7955]
#> interpretation: large

The Hedges’ g applies a small-sample bias correction to Cohen’s d. For n = 6 per group, the correction is roughly 6%, which is large enough to matter. The example prefers g for small samples and d for sample sizes above roughly 20 per group.

When the design is more complex

If the biological units are themselves clustered (mice within a cage, cells within a passage, donor samples within a batch), the simple average-then-test workflow above is no longer adequate because it ignores the cluster-level correlation. Use replicate_decompose() instead. It fits a linear mixed model with lme4::lmer and reports variance components for each level of replication.

If the analyst wants to attribute variance across several experimental stages (sampling, instrument noise, calibration, drift), see the Advanced Uncertainty Attribution vignette. The Category C functions uncertainty_origin() and drift_uncertainty() are designed for that question.

What to take away

Three things. First, average technical replicates within biological units before any inferential statistics; sci_describe() does this for the analyst when the analyst supplies bio_rep and tech_rep. Second, default to Welch’s t unless the analyst have a stated reason to use Student’s, and use the permutation test when the Gaussian assumption is doubtful. Third, report effect sizes with confidence intervals, not just p-values, and prefer Hedges’ g for small samples.

References

  • Carroll, R. M. and Nordholm, L. A. (1975). Sampling characteristics of Kelley’s epsilon-squared and Hays’ omega-squared. Educational and Psychological Measurement, 35, 541-554. Source.

  • Kelley, T. L. (1935). An Unbiased Correlation Ratio Measure. Proceedings of the National Academy of Sciences, 21, 554-559. Source.

  • Vaux, D. L., Fidler, F. and Cumming, G. (2012). Replicates and repeats—what is the difference and is it significant? The EMBO Reports, 13, 291-296. Source.