Validation
Every statistic, checked against reference software
Scintria's tests, models and sample sizes are computed by its own code. This page runs that code on data sets it was not tuned on and compares each number with the value established software gives for the same data. If a number stops matching, the automated tests fail and the change cannot ship.
- numbers compared
- 277numbers compared
- methods
- 33methods
- match their reference
- 277 / 277match their reference
Largest relative difference: 1.1 × 10⁻⁶. P-values are required to agree to 6 significant figures, estimates and test statistics to 8 or more, sample sizes exactly. Computed by the code that serves this page.
Comparing groups
| Quantity | Scintria | Reference | Relative difference | match their reference |
|---|---|---|---|---|
| Welch t-test (unequal variances)SciPy 1.18.1 ttest_ind(equal_var=False) | ||||
| t | -2.11759408 | -2.11759408 | 2.1 × 10⁻¹⁴ | Matches |
| degrees of freedom | 150.3279686 | 150.3279686 | identical | Matches |
| p | 0.03585497436 | 0.03585497552 | 3.2 × 10⁻⁸ | Matches |
| 95% CI, lower | -9.170560632 | -9.170560706 | 8.1 × 10⁻⁹ | Matches |
| 95% CI, upper | -0.3175029629 | -0.3175028883 | 2.3 × 10⁻⁷ | Matches |
| t | -2.11759408 | -2.11759408 | 2.6 × 10⁻¹⁴ | Matches |
| degrees of freedom | 150.3279686 | 150.3279686 | 7.6 × 10⁻¹⁶ | Matches |
| p | 0.03585497436 | 0.03585497552 | 3.2 × 10⁻⁸ | Matches |
| 95% CI, lower | -9.170560632 | -9.170560706 | 8.1 × 10⁻⁹ | Matches |
| 95% CI, upper | -0.3175029629 | -0.3175028883 | 2.3 × 10⁻⁷ | Matches |
| Mann–Whitney U (ties, continuity correction)SciPy 1.18.1 mannwhitneyu(method="asymptotic") | ||||
| U | 3732 | 3732 | identical | Matches |
| p | 0.397642665 | 0.397642665 | identical | Matches |
| U | 3732 | 3732 | identical | Matches |
| p | 0.397642665 | 0.397642665 | 5.6 × 10⁻¹⁶ | Matches |
| One-way ANOVASciPy 1.18.1 f_oneway | ||||
| F | 1.828340135 | 1.828340135 | 2.4 × 10⁻¹⁴ | Matches |
| p | 0.1637015625 | 0.1637015621 | 2.5 × 10⁻⁹ | Matches |
| F | 1.828340135 | 1.828340135 | 2.9 × 10⁻¹⁴ | Matches |
| p | 0.1637015625 | 0.1637015621 | 2.5 × 10⁻⁹ | Matches |
| Kruskal–Wallis (tie correction)SciPy 1.18.1 kruskal | ||||
| H | 5.717245106 | 5.717245106 | identical | Matches |
| p | 0.0573476993 | 0.0573476993 | 1.2 × 10⁻¹⁶ | Matches |
| H | 5.717245106 | 5.717245106 | 7.8 × 10⁻¹⁶ | Matches |
| p | 0.0573476993 | 0.0573476993 | 8.5 × 10⁻¹⁶ | Matches |
Tables and epidemiology
| Quantity | Scintria | Reference | Relative difference | match their reference |
|---|---|---|---|---|
| Chi-square test, 3 × 2 tableSciPy 1.18.1 chi2_contingency(correction=False) | ||||
| χ² | 19.5349349 | 19.5349349 | identical | Matches |
| p | 5.7285246e-5 | 5.7285246e-5 | 6.6 × 10⁻¹³ | Matches |
| χ² | 19.5349349 | 19.5349349 | 2.0 × 10⁻¹⁵ | Matches |
| p | 5.7285246e-5 | 5.7285246e-5 | 6.6 × 10⁻¹³ | Matches |
| Fisher's exact test, 2 × 2SciPy 1.18.1 fisher_exact | ||||
| p (two-sided) | 0.5216296032 | 0.5216296032 | 4.8 × 10⁻¹¹ | Matches |
| p (two-sided) | 0.5216296032 | 0.5216296032 | 4.8 × 10⁻¹¹ | Matches |
| Odds ratio, Woolf intervalstatsmodels 0.15.0 Table2x2 | ||||
| odds ratio | 0.7889060092 | 0.7889060092 | identical | Matches |
| 95% CI, lower | 0.4186048106 | 0.4186048106 | 4.0 × 10⁻¹⁶ | Matches |
| 95% CI, upper | 1.486778641 | 1.486778641 | 6.0 × 10⁻¹⁶ | Matches |
| Risk ratio, Katz log intervalstatsmodels 0.15.0 Table2x2 | ||||
| risk ratio | 0.8499452355 | 0.8499452355 | identical | Matches |
| 95% CI, lower | 0.549282796 | 0.549282796 | identical | Matches |
| 95% CI, upper | 1.315182104 | 1.315182104 | identical | Matches |
| Mantel–Haenszel odds ratio, 3 stratastatsmodels 0.15.0 StratifiedTable | ||||
| pooled odds ratio | 0.7729426222 | 0.7729426222 | identical | Matches |
| 95% CI, lower (Robins–Breslow–Greenland) | 0.4096413795 | 0.4096413795 | identical | Matches |
| 95% CI, upper | 1.458447137 | 1.458447137 | identical | Matches |
| pooled odds ratio | 0.7729426222 | 0.7729426222 | identical | Matches |
| 95% CI, lower (Robins–Breslow–Greenland) | 0.4096413795 | 0.4096413795 | 1.4 × 10⁻¹⁶ | Matches |
| 95% CI, upper | 1.458447137 | 1.458447137 | 1.5 × 10⁻¹⁵ | Matches |
| Mantel–Haenszel risk ratio, 3 stratastatsmodels 0.15.0 StratifiedTable | ||||
| pooled risk ratio | 0.8354152917 | 0.8354152917 | identical | Matches |
| Breslow–Day test of homogeneitystatsmodels 0.15.0 test_equal_odds(adjust=False) | ||||
| χ² | 3.711843772 | 3.711843772 | identical | Matches |
| p | 0.1563087773 | 0.1563087773 | 1.1 × 10⁻¹³ | Matches |
Correlation
| Quantity | Scintria | Reference | Relative difference | match their reference |
|---|---|---|---|---|
| Pearson correlationSciPy 1.18.1 pearsonr | ||||
| r | 0.4378568414 | 0.4378568414 | 1.3 × 10⁻¹⁶ | Matches |
| p | 7.9107143e-10 | 7.9107147e-10 | 4.4 × 10⁻⁸ | Matches |
| 95% CI, lower (Fisher z) | 0.3115456539 | 0.3115456539 | 1.8 × 10⁻¹⁶ | Matches |
| 95% CI, upper | 0.5489632384 | 0.5489632384 | 4.0 × 10⁻¹⁶ | Matches |
| r | 0.4378568414 | 0.4378568414 | 3.8 × 10⁻¹⁶ | Matches |
| p | 7.9107143e-10 | 7.9107147e-10 | 4.4 × 10⁻⁸ | Matches |
| 95% CI, lower (Fisher z) | 0.3115456539 | 0.3115456539 | 1.4 × 10⁻¹⁵ | Matches |
| 95% CI, upper | 0.5489632384 | 0.5489632384 | 8.1 × 10⁻¹⁶ | Matches |
| Spearman correlation (ties)SciPy 1.18.1 spearmanr | ||||
| ρ | 0.0738792705 | 0.0738792705 | identical | Matches |
| p | 0.3243114251 | 0.3243114058 | 6.0 × 10⁻⁸ | Matches |
| ρ | 0.0738792705 | 0.0738792705 | 1.9 × 10⁻¹⁶ | Matches |
| p | 0.3243114251 | 0.3243114058 | 6.0 × 10⁻⁸ | Matches |
Regression
| Quantity | Scintria | Reference | Relative difference | match their reference |
|---|---|---|---|---|
| Linear regressionstatsmodels 0.15.0 OLS | ||||
| Intercept coefficient | 98.03986107 | 98.03986107 | 2.9 × 10⁻¹⁶ | Matches |
| Intercept standard error | 4.136415359 | 4.136415359 | 4.7 × 10⁻¹⁵ | Matches |
| age coefficient | 0.5649740285 | 0.5649740285 | 2.5 × 10⁻¹³ | Matches |
| age standard error | 0.08882399065 | 0.08882399065 | 6.5 × 10⁻¹⁴ | Matches |
| arm[T.B] coefficient | 4.865284054 | 4.865284054 | 8.0 × 10⁻¹⁵ | Matches |
| arm[T.B] standard error | 2.331973569 | 2.331973569 | 9.5 × 10⁻¹⁶ | Matches |
| arm[T.C] coefficient | 1.113812805 | 1.113812805 | 2.8 × 10⁻¹⁵ | Matches |
| arm[T.C] standard error | 2.485395416 | 2.485395416 | 1.3 × 10⁻¹⁵ | Matches |
| sex[T.M] coefficient | 3.269669462 | 3.269669462 | 1.2 × 10⁻¹⁴ | Matches |
| sex[T.M] standard error | 1.984971038 | 1.984971038 | 4.5 × 10⁻¹⁶ | Matches |
| R² | 0.2247990045 | 0.2247990045 | 1.5 × 10⁻¹⁵ | Matches |
| adjusted R² | 0.2070801246 | 0.2070801246 | 1.1 × 10⁻¹⁵ | Matches |
| F | 12.68697603 | 12.68697603 | 1.7 × 10⁻¹⁵ | Matches |
| log-likelihood | -717.139907 | -717.139907 | identical | Matches |
| Intercept coefficient | 98.03986107 | 98.03986107 | 4.3 × 10⁻¹⁶ | Matches |
| Intercept standard error | 4.136415359 | 4.136415359 | 1.3 × 10⁻¹⁵ | Matches |
| age coefficient | 0.5649740285 | 0.5649740285 | 2.4 × 10⁻¹⁵ | Matches |
| age standard error | 0.08882399065 | 0.08882399065 | 1.6 × 10⁻¹⁶ | Matches |
| arm[T.B] coefficient | 4.865284054 | 4.865284054 | 2.2 × 10⁻¹⁵ | Matches |
| arm[T.B] standard error | 2.331973569 | 2.331973569 | 3.8 × 10⁻¹⁶ | Matches |
| arm[T.C] coefficient | 1.113812805 | 1.113812805 | 5.4 × 10⁻¹⁵ | Matches |
| arm[T.C] standard error | 2.485395416 | 2.485395416 | 2.1 × 10⁻¹⁵ | Matches |
| sex[T.M] coefficient | 3.269669462 | 3.269669462 | 2.4 × 10⁻¹⁵ | Matches |
| sex[T.M] standard error | 1.984971038 | 1.984971038 | 6.7 × 10⁻¹⁶ | Matches |
| R² | 0.2247990045 | 0.2247990045 | 1.1 × 10⁻¹⁵ | Matches |
| adjusted R² | 0.2070801246 | 0.2070801246 | 1.2 × 10⁻¹⁵ | Matches |
| F | 12.68697603 | 12.68697603 | 4.9 × 10⁻¹⁵ | Matches |
| log-likelihood | -717.139907 | -717.139907 | 3.2 × 10⁻¹⁶ | Matches |
| Logistic regressionstatsmodels 0.15.0 Logit | ||||
| Intercept coefficient | -3.047973524 | -3.047973524 | 4.4 × 10⁻¹¹ | Matches |
| Intercept standard error | 0.7807510382 | 0.7807510382 | 1.8 × 10⁻¹¹ | Matches |
| age coefficient | 0.02912971917 | 0.02912971917 | 1.4 × 10⁻¹¹ | Matches |
| age standard error | 0.01537647758 | 0.01537647758 | 8.9 × 10⁻¹² | Matches |
| arm[T.B] coefficient | 1.612293191 | 1.612293191 | 7.4 × 10⁻¹¹ | Matches |
| arm[T.B] standard error | 0.4526204548 | 0.4526204549 | 2.8 × 10⁻¹¹ | Matches |
| arm[T.C] coefficient | 2.019331016 | 2.019331016 | 6.0 × 10⁻¹¹ | Matches |
| arm[T.C] standard error | 0.4735150035 | 0.4735150035 | 2.6 × 10⁻¹¹ | Matches |
| sex[T.M] coefficient | -0.3398266523 | -0.3398266523 | 1.4 × 10⁻¹¹ | Matches |
| sex[T.M] standard error | 0.3483097942 | 0.3483097942 | 1.0 × 10⁻¹¹ | Matches |
| log-likelihood | -99.68891422 | -99.68891422 | 5.7 × 10⁻¹⁶ | Matches |
| null log-likelihood | -112.3796352 | -112.3796352 | 1.1 × 10⁻¹¹ | Matches |
| Intercept coefficient | -3.047973524 | -3.047973524 | 4.4 × 10⁻¹¹ | Matches |
| Intercept standard error | 0.7807510382 | 0.7807510382 | 4.3 × 10⁻¹⁶ | Matches |
| age coefficient | 0.02912971917 | 0.02912971917 | 1.4 × 10⁻¹¹ | Matches |
| age standard error | 0.01537647758 | 0.01537647758 | 4.5 × 10⁻¹⁶ | Matches |
| arm[T.B] coefficient | 1.612293191 | 1.612293191 | 7.4 × 10⁻¹¹ | Matches |
| arm[T.B] standard error | 0.4526204548 | 0.4526204548 | 4.9 × 10⁻¹⁶ | Matches |
| arm[T.C] coefficient | 2.019331016 | 2.019331016 | 6.0 × 10⁻¹¹ | Matches |
| arm[T.C] standard error | 0.4735150035 | 0.4735150035 | 2.3 × 10⁻¹⁶ | Matches |
| sex[T.M] coefficient | -0.3398266523 | -0.3398266523 | 1.4 × 10⁻¹¹ | Matches |
| sex[T.M] standard error | 0.3483097942 | 0.3483097942 | 4.8 × 10⁻¹⁶ | Matches |
| log-likelihood | -99.68891422 | -99.68891422 | 1.4 × 10⁻¹⁶ | Matches |
| null log-likelihood | -112.3796352 | -112.3796352 | identical | Matches |
| Poisson regression, person-time offsetstatsmodels 0.15.0 GLM Poisson | ||||
| Intercept coefficient | -2.62823187 | -2.62823187 | 3.8 × 10⁻¹³ | Matches |
| Intercept standard error | 0.2475753226 | 0.2475753226 | 4.9 × 10⁻¹¹ | Matches |
| arm[T.B] coefficient | 0.1198957687 | 0.1198957687 | 4.7 × 10⁻¹³ | Matches |
| arm[T.B] standard error | 0.1302486843 | 0.1302486843 | 2.3 × 10⁻¹¹ | Matches |
| arm[T.C] coefficient | -0.2747658976 | -0.2747658976 | 7.5 × 10⁻¹² | Matches |
| arm[T.C] standard error | 0.1582061621 | 0.1582061621 | 6.8 × 10⁻¹¹ | Matches |
| age coefficient | 0.01058090724 | 0.01058090724 | 2.0 × 10⁻¹² | Matches |
| age standard error | 0.005153441503 | 0.005153441503 | 3.9 × 10⁻¹¹ | Matches |
| log-likelihood | -273.782855 | -273.782855 | identical | Matches |
| Intercept coefficient | -2.62823187 | -2.62823187 | 3.8 × 10⁻¹³ | Matches |
| Intercept standard error | 0.2475753226 | 0.2475753226 | 2.7 × 10⁻¹⁵ | Matches |
| arm[T.B] coefficient | 0.1198957687 | 0.1198957687 | 5.0 × 10⁻¹³ | Matches |
| arm[T.B] standard error | 0.1302486843 | 0.1302486843 | 1.3 × 10⁻¹⁵ | Matches |
| arm[T.C] coefficient | -0.2747658976 | -0.2747658976 | 7.5 × 10⁻¹² | Matches |
| arm[T.C] standard error | 0.1582061621 | 0.1582061621 | 8.8 × 10⁻¹⁶ | Matches |
| age coefficient | 0.01058090724 | 0.01058090724 | 1.9 × 10⁻¹² | Matches |
| age standard error | 0.005153441503 | 0.005153441503 | 3.4 × 10⁻¹⁶ | Matches |
| log-likelihood | -273.782855 | -273.782855 | 1.5 × 10⁻¹⁵ | Matches |
Sample size
| Quantity | Scintria | Reference | Relative difference | match their reference |
|---|---|---|---|---|
| Two means (normal approximation)statsmodels 0.15.0 NormalIndPower (rounded up) | ||||
| n in group 1: Δ = 5, σ = 12, ratio 1:1 | 91 | 91 | identical | Matches |
| n in group 1: Δ = 5, σ = 12, ratio 1:2 | 68 | 68 | identical | Matches |
| n in group 1: Δ = 0.4, σ = 1, ratio 1:1 | 99 | 99 | identical | Matches |
| Two proportions (pooled null variance)statsmodels 0.15.0 samplesize_proportions_2indep_onetail (rounded up) | ||||
| n in group 1: 0.3 vs 0.45, ratio 1:1 | 163 | 163 | identical | Matches |
| n in group 1: 0.3 vs 0.45, ratio 1:2 | 123 | 123 | identical | Matches |
| n in group 1: 0.1 vs 0.05, ratio 1:1 | 435 | 435 | identical | Matches |
Survival
| Quantity | Scintria | Reference | Relative difference | match their reference |
|---|---|---|---|---|
| Kaplan–Meier, 3 arms, heavy tiesstatsmodels 0.15 SurvfuncRight | ||||
| S(12) in arm A | 0.4877788631 | 0.4877788631 | 1.1 × 10⁻¹⁶ | Matches |
| S(24) in arm A | 0.3704904471 | 0.3704904471 | identical | Matches |
| median survival, arm A | 12 | 12 | identical | Matches |
| S(12) in arm B | 0.401090595 | 0.401090595 | 1.4 × 10⁻¹⁶ | Matches |
| S(24) in arm B | 0.1456059975 | 0.1456059975 | 1.9 × 10⁻¹⁶ | Matches |
| median survival, arm B | 9 | 9 | identical | Matches |
| S(12) in arm C | 0.3422459893 | 0.3422459893 | 3.2 × 10⁻¹⁶ | Matches |
| S(24) in arm C | 0.1604278075 | 0.1604278075 | 3.5 × 10⁻¹⁶ | Matches |
| median survival, arm C | 7 | 7 | identical | Matches |
| S(12) in arm A | 0.4877788631 | 0.4877788631 | 9.1 × 10⁻¹⁶ | Matches |
| S(24) in arm A | 0.3704904471 | 0.3704904471 | 6.0 × 10⁻¹⁶ | Matches |
| median survival, arm A | 12 | 12 | identical | Matches |
| S(12) in arm B | 0.401090595 | 0.401090595 | identical | Matches |
| S(24) in arm B | 0.1456059975 | 0.1456059975 | 1.9 × 10⁻¹⁵ | Matches |
| median survival, arm B | 9 | 9 | identical | Matches |
| S(12) in arm C | 0.3422459893 | 0.3422459893 | 6.5 × 10⁻¹⁶ | Matches |
| S(24) in arm C | 0.1604278075 | 0.1604278075 | 1.7 × 10⁻¹⁶ | Matches |
| median survival, arm C | 7 | 7 | identical | Matches |
| Log-rank teststatsmodels 0.15 survdiff | ||||
| χ² | 10.12613118 | 10.12613118 | 1.2 × 10⁻¹⁵ | Matches |
| p | 0.006326136411 | 0.006326136411 | 1.8 × 10⁻¹⁴ | Matches |
| χ² | 10.12613118 | 10.12613118 | 2.6 × 10⁻¹⁵ | Matches |
| p | 0.006326136411 | 0.006326136411 | 7.8 × 10⁻¹⁵ | Matches |
| Cox proportional hazards (Efron ties)statsmodels 0.15 PHReg | ||||
| arm_B coefficient | 0.4845457522 | 0.4845457522 | 6.9 × 10⁻¹⁶ | Matches |
| arm_B standard error | 0.2252779953 | 0.2252779953 | 6.2 × 10⁻¹⁶ | Matches |
| arm_C coefficient | 0.7258522773 | 0.7258522773 | 7.6 × 10⁻¹⁶ | Matches |
| arm_C standard error | 0.2517177148 | 0.2517177148 | 2.2 × 10⁻¹⁶ | Matches |
| sex_M coefficient | 0.3366014484 | 0.3366014484 | 1.2 × 10⁻¹⁵ | Matches |
| sex_M standard error | 0.1938004888 | 0.1938004888 | 1.4 × 10⁻¹⁶ | Matches |
| age coefficient | 0.01529787984 | 0.01529787984 | 1.7 × 10⁻¹⁵ | Matches |
| age standard error | 0.006814500083 | 0.006814500083 | 7.6 × 10⁻¹⁶ | Matches |
| log partial likelihood | -464.2144038 | -464.2144038 | 3.7 × 10⁻¹⁶ | Matches |
| arm_B coefficient | 0.4845457522 | 0.4845457522 | 4.4 × 10⁻¹⁵ | Matches |
| arm_B standard error | 0.2252779953 | 0.2252779953 | 1.5 × 10⁻¹⁵ | Matches |
| arm_C coefficient | 0.7258522773 | 0.7258522773 | 2.8 × 10⁻¹⁵ | Matches |
| arm_C standard error | 0.2517177148 | 0.2517177148 | 8.8 × 10⁻¹⁶ | Matches |
| sex_M coefficient | 0.3366014484 | 0.3366014484 | 5.1 × 10⁻¹⁵ | Matches |
| sex_M standard error | 0.1938004888 | 0.1938004888 | 1.7 × 10⁻¹⁵ | Matches |
| age coefficient | 0.01529787984 | 0.01529787984 | 1.1 × 10⁻¹⁵ | Matches |
| age standard error | 0.006814500083 | 0.006814500083 | 3.8 × 10⁻¹⁶ | Matches |
| log partial likelihood | -464.2144038 | -464.2144038 | 1.5 × 10⁻¹⁵ | Matches |
Post hoc comparisons
| Quantity | Scintria | Reference | Relative difference | match their reference |
|---|---|---|---|---|
| Tukey–Kramer, unbalanced groups with unequal variancesstatsmodels 0.15 pairwise_tukeyhsd | ||||
| adjusted p, G1 vs G2 | 0.2331580734 | 0.2331580734 | 4.2 × 10⁻¹¹ | Matches |
| adjusted p, G1 vs G3 | 0.002562749194 | 0.002562749184 | 4.1 × 10⁻⁹ | Matches |
| adjusted p, G1 vs G4 | 0.8894321743 | 0.8894321743 | 3.0 × 10⁻¹² | Matches |
| adjusted p, G2 vs G3 | 0.1627520701 | 0.1627520701 | 8.1 × 10⁻¹¹ | Matches |
| adjusted p, G2 vs G4 | 0.5865478138 | 0.5865478138 | 7.9 × 10⁻¹² | Matches |
| adjusted p, G3 vs G4 | 0.01024361054 | 0.01024361053 | 1.1 × 10⁻⁹ | Matches |
| adjusted p, G1 vs G2 | 0.2331580734 | 0.2331580734 | 3.3 × 10⁻¹¹ | Matches |
| adjusted p, G1 vs G3 | 0.002562749194 | 0.002562749184 | 3.9 × 10⁻⁹ | Matches |
| adjusted p, G1 vs G4 | 0.8894321743 | 0.8894321743 | 1.3 × 10⁻¹² | Matches |
| adjusted p, G2 vs G3 | 0.1627520701 | 0.1627520701 | 5.2 × 10⁻¹¹ | Matches |
| adjusted p, G2 vs G4 | 0.5865478138 | 0.5865478138 | 7.1 × 10⁻¹² | Matches |
| adjusted p, G3 vs G4 | 0.01024361054 | 0.01024361053 | 9.7 × 10⁻¹⁰ | Matches |
| Games–Howell, unbalanced groups with unequal variancespingouin 0.7 pairwise_gameshowell | ||||
| adjusted p, G1 vs G2 | 0.1079105314 | 0.1079105314 | 2.6 × 10⁻¹¹ | Matches |
| adjusted p, G1 vs G3 | 2.0947839e-5 | 2.0947838e-5 | 1.4 × 10⁻⁸ | Matches |
| adjusted p, G1 vs G4 | 0.383335934 | 0.383335934 | 2.1 × 10⁻¹² | Matches |
| adjusted p, G2 vs G3 | 0.3021047381 | 0.3021047381 | 4.6 × 10⁻¹² | Matches |
| adjusted p, G2 vs G4 | 0.4931079162 | 0.4931079162 | 7.7 × 10⁻¹³ | Matches |
| adjusted p, G3 vs G4 | 0.0009990085368 | 0.0009990085347 | 2.1 × 10⁻⁹ | Matches |
| Dunn (Holm-adjusted), unbalanced groups with unequal variancesscikit-posthocs posthoc_dunn | ||||
| adjusted p, G1 vs G2 | 0.1396739609 | 0.1396739609 | 4.8 × 10⁻¹⁵ | Matches |
| adjusted p, G1 vs G3 | 0.002507400029 | 0.002507400029 | 5.1 × 10⁻¹⁴ | Matches |
| adjusted p, G1 vs G4 | 0.4253979602 | 0.4253979602 | identical | Matches |
| adjusted p, G2 vs G3 | 0.3737591137 | 0.3737591137 | 1.2 × 10⁻¹⁵ | Matches |
| adjusted p, G2 vs G4 | 0.4253979602 | 0.4253979602 | identical | Matches |
| adjusted p, G3 vs G4 | 0.02939213766 | 0.02939213766 | 5.8 × 10⁻¹⁵ | Matches |
Paired and repeated measures
| Quantity | Scintria | Reference | Relative difference | match their reference |
|---|---|---|---|---|
| Paired t-testSciPy ttest_rel | ||||
| t | -5.249309053 | -5.249309053 | 1.7 × 10⁻¹⁶ | Matches |
| p | 2.8868021e-5 | 2.8868021e-5 | 3.3 × 10⁻¹¹ | Matches |
| t | -5.249309053 | -5.249309053 | 6.8 × 10⁻¹⁶ | Matches |
| p | 2.8868021e-5 | 2.8868021e-5 | 3.3 × 10⁻¹¹ | Matches |
| Wilcoxon signed-rank (ties, zero differences)SciPy wilcoxon(correction=True) | ||||
| p | 0.0003000912401 | 0.0003000912401 | 6.2 × 10⁻¹⁴ | Matches |
| p | 0.0003000912401 | 0.0003000912401 | 5.9 × 10⁻¹⁴ | Matches |
| Friedman testSciPy friedmanchisquare | ||||
| χ² | 9.333333333 | 9.333333333 | identical | Matches |
| p | 0.02517213998 | 0.02517213998 | 5.7 × 10⁻¹⁵ | Matches |
| χ² | 9.333333333 | 9.333333333 | 1.7 × 10⁻¹⁵ | Matches |
| p | 0.02517213998 | 0.02517213998 | 1.4 × 10⁻¹⁴ | Matches |
| Repeated-measures ANOVApingouin 0.7 rm_anova | ||||
| F | 3.045619778 | 3.045619778 | 1.2 × 10⁻¹⁵ | Matches |
| Greenhouse–Geisser ε | 0.8994030163 | 0.8994030163 | 3.7 × 10⁻¹⁶ | Matches |
| p (Greenhouse–Geisser) | 0.04291093068 | 0.04291093002 | 1.5 × 10⁻⁸ | Matches |
| F | 3.045619778 | 3.045619778 | 4.4 × 10⁻¹⁵ | Matches |
| McNemar teststatsmodels 0.15 mcnemar(exact=True) | ||||
| exact p | 0.1686375439 | 0.1686375439 | 9.9 × 10⁻¹⁶ | Matches |
| χ² p (continuity correction) | 0.1698105051 | 0.1698105051 | 2.3 × 10⁻¹³ | Matches |
| exact p | 0.1686375439 | 0.1686375439 | 2.6 × 10⁻¹⁵ | Matches |
| χ² p (continuity correction) | 0.1698105051 | 0.1698105051 | 2.1 × 10⁻¹³ | Matches |
Mixed models
| Quantity | Scintria | Reference | Relative difference | match their reference |
|---|---|---|---|---|
| Linear mixed model (REML): random intercept, 35 unbalanced clustersstatsmodels 0.15 MixedLM; standard errors (X′V⁻¹X)⁻¹ as lme4 and Stata, in NumPy | ||||
| Intercept coefficient | 44.07067475 | 44.07067472 | 6.4 × 10⁻¹⁰ | Matches |
| Intercept standard error | 2.500820279 | 2.500819227 | 4.2 × 10⁻⁷ | Matches |
| time coefficient | 1.092544686 | 1.092544666 | 1.8 × 10⁻⁸ | Matches |
| time standard error | 0.1108661528 | 0.1108661612 | 7.6 × 10⁻⁸ | Matches |
| arm[T.B] coefficient | 3.265883263 | 3.265883291 | 8.8 × 10⁻⁹ | Matches |
| arm[T.B] standard error | 1.392797052 | 1.392796456 | 4.3 × 10⁻⁷ | Matches |
| age coefficient | 0.1395958453 | 0.1395958458 | 3.6 × 10⁻⁹ | Matches |
| age standard error | 0.05068772334 | 0.05068770177 | 4.3 × 10⁻⁷ | Matches |
| REML log-likelihood | -516.1068913 | -516.1068913 | 2.5 × 10⁻¹⁴ | Matches |
| random-intercept variance | 14.52836717 | 14.52835304 | 9.7 × 10⁻⁷ | Matches |
| residual variance | 8.090505596 | 8.090507007 | 1.7 × 10⁻⁷ | Matches |
| Intercept coefficient | 44.07067475 | 44.07067475 | 1.1 × 10⁻¹⁰ | Matches |
| Intercept standard error | 2.500820279 | 2.500820459 | 7.2 × 10⁻⁸ | Matches |
| time coefficient | 1.092544686 | 1.092544689 | 3.1 × 10⁻⁹ | Matches |
| time standard error | 0.1108661528 | 0.1108661513 | 1.3 × 10⁻⁸ | Matches |
| arm[T.B] coefficient | 3.265883263 | 3.265883258 | 1.5 × 10⁻⁹ | Matches |
| arm[T.B] standard error | 1.392797052 | 1.392797155 | 7.4 × 10⁻⁸ | Matches |
| age coefficient | 0.1395958453 | 0.1395958453 | 6.1 × 10⁻¹⁰ | Matches |
| age standard error | 0.05068772334 | 0.05068772704 | 7.3 × 10⁻⁸ | Matches |
| REML log-likelihood | -516.1068913 | -516.1068913 | 6.6 × 10⁻¹⁵ | Matches |
| random-intercept variance | 14.52836717 | 14.5283696 | 1.7 × 10⁻⁷ | Matches |
| residual variance | 8.090505596 | 8.090505354 | 3.0 × 10⁻⁸ | Matches |
| Linear mixed model (REML): random intercept and slope, 35 unbalanced clustersstatsmodels 0.15 MixedLM; standard errors (X′V⁻¹X)⁻¹ as lme4 and Stata, in NumPy | ||||
| Intercept coefficient | 44.44193218 | 44.44193159 | 1.3 × 10⁻⁸ | Matches |
| Intercept standard error | 2.232990923 | 2.232989962 | 4.3 × 10⁻⁷ | Matches |
| time coefficient | 1.002008135 | 1.00200826 | 1.3 × 10⁻⁷ | Matches |
| time standard error | 0.2006136234 | 0.2006134161 | 1.0 × 10⁻⁶ | Matches |
| arm[T.B] coefficient | 3.273578921 | 3.273578832 | 2.7 × 10⁻⁸ | Matches |
| arm[T.B] standard error | 1.246992279 | 1.246991736 | 4.4 × 10⁻⁷ | Matches |
| age coefficient | 0.1335385938 | 0.1335386053 | 8.6 × 10⁻⁸ | Matches |
| age standard error | 0.045274621 | 0.04527460139 | 4.3 × 10⁻⁷ | Matches |
| REML log-likelihood | -497.8908023 | -497.8908023 | 1.7 × 10⁻¹³ | Matches |
| random-intercept variance | 11.68038689 | 11.680374 | 1.1 × 10⁻⁶ | Matches |
| residual variance | 5.205644307 | 5.205647949 | 7.0 × 10⁻⁷ | Matches |
| Intercept coefficient | 44.44193218 | 44.44193193 | 5.4 × 10⁻⁹ | Matches |
| Intercept standard error | 2.232990923 | 2.232990615 | 1.4 × 10⁻⁷ | Matches |
| time coefficient | 1.002008135 | 1.002008022 | 1.1 × 10⁻⁷ | Matches |
| time standard error | 0.2006136234 | 0.2006137306 | 5.3 × 10⁻⁷ | Matches |
| arm[T.B] coefficient | 3.273578921 | 3.273578928 | 1.9 × 10⁻⁹ | Matches |
| arm[T.B] standard error | 1.246992279 | 1.246992093 | 1.5 × 10⁻⁷ | Matches |
| age coefficient | 0.1335385938 | 0.1335386017 | 6.0 × 10⁻⁸ | Matches |
| age standard error | 0.045274621 | 0.04527461428 | 1.5 × 10⁻⁷ | Matches |
| REML log-likelihood | -497.8908023 | -497.8908023 | 1.5 × 10⁻¹⁴ | Matches |
| random-intercept variance | 11.68038689 | 11.6803867 | 1.6 × 10⁻⁸ | Matches |
| residual variance | 5.205644307 | 5.205643401 | 1.7 × 10⁻⁷ | Matches |
Survey designs
| Quantity | Scintria | Reference | Relative difference | match their reference |
|---|---|---|---|---|
| Weighted linear model, 4 strata of 2–5 clustersLinearisation formulas as R survey / Stata svy, in NumPy | ||||
| (Intercept) coefficient | 20.39638617 | 20.39638617 | 1.7 × 10⁻¹⁴ | Matches |
| (Intercept) standard error | 2.42202154 | 2.42202154 | 5.3 × 10⁻¹⁵ | Matches |
| x coefficient | 0.4870808794 | 0.4870808794 | 1.7 × 10⁻¹⁴ | Matches |
| x standard error | 0.04891098998 | 0.04891098998 | 2.1 × 10⁻¹⁵ | Matches |
| g_b coefficient | 3.471712775 | 3.471712775 | 1.3 × 10⁻¹⁶ | Matches |
| g_b standard error | 0.9380096036 | 0.9380096036 | 1.1 × 10⁻¹⁴ | Matches |
| g_c coefficient | -2.507923016 | -2.507923016 | 1.4 × 10⁻¹⁵ | Matches |
| g_c standard error | 0.6904310683 | 0.6904310683 | 2.0 × 10⁻¹⁴ | Matches |
| (Intercept) coefficient | 20.39638617 | 20.39638617 | 3.8 × 10⁻¹⁵ | Matches |
| (Intercept) standard error | 2.42202154 | 2.42202154 | 2.0 × 10⁻¹⁴ | Matches |
| x coefficient | 0.4870808794 | 0.4870808794 | 3.5 × 10⁻¹⁵ | Matches |
| x standard error | 0.04891098998 | 0.04891098998 | 2.0 × 10⁻¹⁴ | Matches |
| g_b coefficient | 3.471712775 | 3.471712775 | 2.6 × 10⁻¹⁶ | Matches |
| g_b standard error | 0.9380096036 | 0.9380096036 | 8.3 × 10⁻¹⁶ | Matches |
| g_c coefficient | -2.507923016 | -2.507923016 | 1.9 × 10⁻¹⁵ | Matches |
| g_c standard error | 0.6904310683 | 0.6904310683 | 3.5 × 10⁻¹⁵ | Matches |
| Weighted meanLinearisation formulas as R survey / Stata svy, in NumPy | ||||
| mean | 39.60418044 | 39.60418044 | 5.4 × 10⁻¹⁶ | Matches |
| standard error | 0.8314803437 | 0.8314803437 | 6.7 × 10⁻¹⁶ | Matches |
| mean | 39.60418044 | 39.60418044 | 1.3 × 10⁻¹⁵ | Matches |
| standard error | 0.8314803437 | 0.8314803437 | 1.1 × 10⁻¹⁵ | Matches |
Re-run it yourself
The data sets and reference values are published. Load them in R, Stata, SPSS or Python and compare. The script that produced the core reference values is included.
- core.csvCore data set: tests, tables, correlation, regression (180 rows)
- ref-core.jsonReference values for the core data set (SciPy, statsmodels)
- core_reference.pyPython script that made the core data and its reference values
- surv.csvSurvival data: 3 arms, heavy ties
- posthoc.csvGroups for post hoc comparisons
- mixed.csvClustered data for mixed models
- survey.csvSurvey data: weights, strata and clusters
- ref.jsonReference values for the survival, post hoc, paired, mixed-model and survey checks
- ref-r.jsonReference values from R (base R, survival, survey, lme4)
- r_reference.RR script that produced the R reference values
What this shows, and what it does not
- R 4.3 was run here: base R, survival, survey and lme4 give the same numbers as Scintria for every check R can make directly (124 of them), shown as separate rows. Stata is commercial software and was not run; its mixed-model standard errors and survey linearisation use the same formulas as lme4 and R's survey package, which are checked here.
- The data are simulated from fixed seeds and include awkward features on purpose: ties, zero differences, unbalanced groups with unequal variances, sparse strata and random slopes.
- Agreement with reference software shows that the arithmetic is right. It does not show that a method suits your study; Scintria states each test's assumptions beside the result.