Back to the Future: Human-friendly Scheffé Contrasts, or, the Art of Complex Multiple Comparisons… or… Everything Old Is New Again

Gordon P Brooks & Nina Adjanin

Program for “Barcikowski Human-Friendly” Contrasts

Citation

Brooks, G. P., & Adjanin, N. (2023, April). Back to the Future: Human-friendly Scheffé Contrasts, or, the Art of Multiple Comparisons. Paper presented at the 2023 annual meeting of the American Educational Research Association, Chicago, IL.

Background

• The idea for “Human-Friendly Contrasts” in One-way ANOVA was originally conceived by Robert S. Barcikowski (2000, unpublished) based on Scheffe (1956) and Hollingsworth (1978). Barcikowski’s original FORTRAN code was amended and adapted by Brooks & Adjanin.
• “Human-Friendly Contrasts” are based primarily on Helmert-type contrasts, providing “reasonable” comparisons among multiple groups
• For example, in an experiment with one control group and two treatment groups, a “reasonable” comparison might be “SOMETHING” versus “NOTHING”. Such a comparison would require contrasts of the form:
  • Contrast Coefficients: Coef1 = 1.0, Coef2 = -0.50, Coef3 = -0.50
• Or maybe two trait/demographic groups (combined in a reasonable way, perhaps Associate and Full Tenured Professors) versus two other groups (combined reasonably, such as Assistant Professors and Instructors), a comparison might be “TENURED” versus “NOT TENURED”. Such a comparison would require contrasts of the form:
  • Contrast Coefficients: Coef1 = 0.5, Coef2 = 0.50, Coef3 = -0.50, Coef4 = -0.50
• Maximum number of groups allowed by the program = 8

qqBarth R package created by the authors

• https://people.ohio.edu/brooksg/qqBarth.tar.gz
• Maximum Scheffe Contrasts and Human-Friendly Contrasts
• Version: 1.4
• Description: Functions Identify the maximum Scheffe contrast and analyzes the contrasts for an applied researcher dataset. The code also identifies the slightly different normalized maximum posttest contrast described by Hollingsworth (1978). Critically, however, the coefficient weights from complex contrasts such as these are likely to be uninterpretable. Therefore, we have also implemented a method to identify maximum human-friendly contrasts that may be meaningful to and interpretable by a human researcher. We have named these human-friendly contrasts Barcikowski contrasts in memory of Robert Barcikowski, the scholar who introduced the idea to the authors.

Data (import or create a data frame called myData)

• Put your data into a data frame called myData before running any more code
• Your grouping variable must be column 1 and your dependent variable must be column 2
• Replace myAlpha to some alpha value to set your desired alpha level to use for output of Human-Friendly Contrasts (you will always get at least 4 printed out)
• This code will show your first 6 cases, just to check

##   x  y
## 1 1 49
## 2 1 54
## 3 1 40
## 4 1 60
## 5 1 43
## 6 1 65

OUTPUT: Barcikowski’s “Human-Friendly” Contrasts

Total Sample Descriptive Statistics

   n mean   sd  var median min max  skew kurtosis   se
Y 40 49.3 9.02 81.4     49  34  70 0.314   -0.613 1.43

By-Group Descriptive Statistics

       n mean   sd  var median min max    skew kurtosis   se
Grp_1 10 54.9 9.76 95.2   56.5  40  70 -0.0936   -1.027 3.09
Grp_2 10 45.9 7.28 53.0   45.0  36  56  0.1203   -1.643 2.30
Grp_3 10 51.7 8.29 68.7   49.5  39  65  0.2800   -0.606 2.62
Grp_4 10 44.7 7.65 58.5   44.0  34  57  0.1426   -1.240 2.42

Test of Normality (of residuals)

    Shapiro-Wilk normality test

data:  Residuals
W = 0.96881, p-value = 0.3297

Tests of Homoscedasticity

Levene (mean)

Levene's Test for Homogeneity of Variance (center = mean)
      Df F value Pr(>F)
group  3  0.4331 0.7306
      36               

Levene (median)

Levene's Test for Homogeneity of Variance (center = median)
      Df F value Pr(>F)
group  3   0.395 0.7573
      36               

Breusch-Pagan

    studentized Breusch-Pagan test

data:  lm(y ~ x, data = myDF)
BP = 2.3095, df = 3, p-value = 0.5107

Omnibus (Overall) ANOVA

Fisher F

            Df Sum Sq Mean Sq F value Pr(>F)  
x            3  698.4  232.80   3.382 0.0285 *
Residuals   36 2478.0   68.83                 
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Robust Welch F

    One-way analysis of means (not assuming equal variances)

data:  y and x
F = 2.9801, num df = 3.000, denom df = 19.906, p-value = 0.05603

Robust Brown-Forsythe F

BROWN-FORSYTHE F 

 F = 3.382082 , num df = 3 , den df = 34.10243 , p-value =  0.02918407 

Barcikowski’s Most Explanatory Human-Friendly Contrasts

• The most explanatory contrasts significant at chosen alpha (at least 4 are shown with the MAXIMUM Human Contrast listed first)

• The p values reported in the section below have already been adjusted using Scheffe adjustments, so compare the p value to your desired alpha level without further adjustment

• A later section provides the Brown-Forsythe adjustment for unequal variances (for the Maximum contrast only by default, but can be changed to all those significant at chosen alpha)

25 total Human-Friendly Contrasts were tested using alpha =  0.15 

   CONTRAST   SSQ Coef1  Coef2  Coef3  Coef4   diff  lwr.ci upr.ci  pval
24       24 0.930   0.5 -0.500  0.500 -0.500  8.000   0.307 15.693 0.039
10       10 0.893   1.0 -0.500  0.000 -0.500  9.600   0.178 19.022 0.044
5         5 0.756   1.0  0.000  0.000 -1.000 10.200  -0.680 21.080 0.073
16       16 0.716  -0.5  0.000 -0.500  1.000 -8.600 -18.022  0.822 0.085
1         1 0.607   1.0 -0.333 -0.333 -0.333  7.467  -1.417 16.350 0.128
8         8 0.588   1.0 -1.000  0.000  0.000  9.000  -1.880 19.880 0.137

Maximum Contrasts (Barcikowski, Scheffe, Hollingsworth)

• These are the Maximum contrasts for all methods for comparison purposes

  Grp Mean  N Hollingsworth ScheffeMAX ScaledMAX HumanMAX
1   1 54.9 10        0.6701     2.1190     0.700      0.5
2   2 45.9 10       -0.4068    -1.2866    -0.425     -0.5
3   3 51.7 10        0.2872     0.9082     0.300      0.5
4   4 44.7 10       -0.5504    -1.7406    -0.575     -0.5

Tests of Scheffe-only contrasts without Brown-Forsythe adjustment

• These analyses performed with DescTools ScheffeTest function with the maximum contrasts as input

• The p values reported in sections below have already been adjusted using Scheffe adjustments, so compare the p value to your desired alpha level without further adjustment

• The sections following this one provide the Brown-Forsythe adjustments for unequal variances for these Maximum Contrasts

              Coef1  Coef2 Coef3  Coef4  Family  Diff lwr.ci upr.ci   pval
Hollingsworth  0.67 -0.407 0.287 -0.550 1,3-2,4  8.36  0.664   16.1 0.0285
ScheffeMAX     2.12 -1.287 0.908 -1.741 1,3-2,4 26.43  2.099   50.8 0.0285
ScaledMAX      0.70 -0.425 0.300 -0.575 1,3-2,4  8.73  0.693   16.8 0.0285
HumanMAX       0.50 -0.500 0.500 -0.500 1,3-2,4  8.00  0.307   15.7 0.0387

Tests of Scheffe contrasts with Brown-Forsythe adjustment

BARCIKOWSKI HUMAN-FRIENDLY Maximum Contrast(s)

• The sections below use calculations of statistics and critical values for statistical significance as opposed to the p values provided by the DescTools output above

• One critically important part of the results sections below is the robust Brown-Forsythe Adjustment applied to Scheffe for unequal variances, something not available many places

CONTRAST 1 HAS COEFFICIENTS 0.5 -0.5 0.5 -0.5 
SUM OF SQUARES =  640 (out of ANOVA Between Sum of Squares = 698.4 ) 
PROPORTION OF BETWEEN SUM OF SQUARES ACCOUNTED FOR =  0.916 
ESTIMATE OF THIS CONTRAST DIFFERENCE =  8 

SCHEFFE (EQUAL VARIANCES, BALANCED OR UNBALANCED GROUP SIZES) 

CONTRAST IS STATISTICALLY SIGNIFICANT WHEN SCHEFFE_F STATISTIC IS LARGER THAN CONTRAST_FCRIT 
 (OR EQUIVALENTLY WHEN THE VALUE OF ESTIMATED CONTRAST IS LARGER THAN THE CONTRAST_CV) 
  ALPHA F_Critical Contrast_Fcrit SCHEFFE_F F_Sig Estimate Contrast_CV
1  0.10   2.242605       6.727816  9.297821  TRUE        8    6.805130
2  0.05   2.866266       8.598797  9.297821  TRUE        8    7.693399
3  0.01   4.377096      13.131287  9.297821 FALSE        8    9.507209

BROWN-FORSYTHE SCHEFFE (UNEQUAL VARIANCES, BALANCED OR UNBALANCED GROUP SIZES)) 

CONTRAST IS STATISTICALLY SIGNIFICANT WHEN BROWN-FORSYTHE F (BF_F_Stat) IS LARGER THAN CONTRAST_BFCRIT 
  ALPHA BF_Critical Contrast_BFcrit BF_F_Stat F_Sig Estimate
1  0.10    2.381600        7.144800  9.297821  TRUE        8
2  0.05    3.100975        9.302926  9.297821 FALSE        8
3  0.01    4.944608       14.833823  9.297821 FALSE        8

HOLLINGSWORTH Maximum Contrast

• Hollingsworth, H. (1978). The coefficients of the normalized maximum contrast as statistics for posttest ANOVA data interpretations. Journal of Experimental Education, 46(4), 4-6.

• Hollingsworth, H. (1980/1981). Maximized posttest contrasts: A clarification. Journal of Experimental Education, 49(2), 92-93.

HOLLINGSWORTH CONTRAST HAS COEFFICIENTS 0.67 -0.407 0.287 -0.55 
SUM OF SQUARES =  698.4 (out of ANOVA Between Sum of Squares = 698.4 ) 
PROPORTION OF BETWEEN SUM OF SQUARES ACCOUNTED FOR =  1 
ESTIMATE OF THIS CONTRAST DIFFERENCE =  8.357 

SCHEFFE (EQUAL VARIANCES, BALANCED OR UNBALANCED GROUP SIZES) 

CONTRAST IS STATISTICALLY SIGNIFICANT WHEN SCHEFFE_F STATISTIC IS LARGER THAN CONTRAST_FCRIT 
 (OR EQUIVALENTLY WHEN THE VALUE OF ESTIMATED CONTRAST IS LARGER THAN THE CONTRAST_CV) 
  Alpha F_Critical Contrast_Fcrit SCHEFFE_F F_Sig Estimate Contrast_CV
1  0.10   2.242605       6.727816  10.14625  TRUE 8.357033    6.805130
2  0.05   2.866266       8.598797  10.14625  TRUE 8.357033    7.693399
3  0.01   4.377096      13.131287  10.14625 FALSE 8.357033    9.507209

BROWN-FORSYTHE SCHEFFE (UNEQUAL VARIANCES, BALANCED OR UNBALANCED GROUP SIZES)) 

CONTRAST IS STATISTICALLY SIGNIFICANT WHEN BROWN-FORSYTHE F (BF_F_Stat) IS LARGER THAN CONTRAST_BFCRIT 
  Alpha BF_Critical Contrast_BFcrit BF_F_Stat BF_Sig Estimate
1  0.10    2.345618        7.144800  9.324683   TRUE 8.357033
2  0.05    3.039693        9.302926  9.324683   TRUE 8.357033
3  0.01    4.793481       14.833823  9.324683  FALSE 8.357033

SCHEFFE (SCALED) Maximum Contrast

SCHEFFE Maximum Contrast & Maximum Contrast SCALED so positive & negative coefficient sets sum to 0

Original Scheffe contrast coefficients were: 2.119 -1.287 0.908 -1.741 
Scaled Scheffe contrasts were original Coefficients divided by: 3.027177 
SCALED SCHEFFE CONTRAST HAS COEFFICIENTS 0.7 -0.425 0.3 -0.575 
SUM OF SQUARES =  698.4 (out of ANOVA Between Sum of Squares = 698.4 ) 
PROPORTION OF BETWEEN SUM OF SQUARES ACCOUNTED FOR =  1 
ESTIMATE OF THIS CONTRAST DIFFERENCE =  8.73 

SCHEFFE (EQUAL VARIANCES, BALANCED OR UNBALANCED GROUP SIZES) 

CONTRAST IS STATISTICALLY SIGNIFICANT WHEN SCHEFFE_F STATISTIC IS LARGER THAN CONTRAST_FCRIT 
 (OR EQUIVALENTLY WHEN THE VALUE OF ESTIMATED CONTRAST IS LARGER THAN THE CONTRAST_CV) 
  Alpha F_Critical Contrast_Fcrit SCHEFFE_F F_Sig Estimate Contrast_CV
1  0.10   2.242605       6.727816  10.14625  TRUE     8.73    7.108837
2  0.05   2.866266       8.598797  10.14625  TRUE     8.73    8.036748
3  0.01   4.377096      13.131287  10.14625 FALSE     8.73    9.931507

BROWN-FORSYTHE SCHEFFE (UNEQUAL VARIANCES, BALANCED OR UNBALANCED GROUP SIZES)) 

CONTRAST IS STATISTICALLY SIGNIFICANT WHEN BROWN-FORSYTHE F (BF_F_Stat) IS LARGER THAN CONTRAST_BFCRIT 
  ALPHA BF_Critical Contrast_BFcrit BF_F_Stat BF_Sig Estimate
1  0.10    2.381600        7.144800  9.324683   TRUE     8.73
2  0.05    3.100975        9.302926  9.324683   TRUE     8.73
3  0.01    4.944608       14.833823  9.324683  FALSE     8.73

END