T-tests and Analysis of Variance (ANOVA)
Updated:
Overview¶
This notebook will analyze a fabricated dataset representing race and age of voters.
Tests used will include one-way ANOVA test, paired t-test, and Tukey's Range Test, starting with a baseline comparison before altering the data and moving on to the post hoc analyses.
Imports¶
%matplotlib inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import scipy.stats as ss
from statsmodels.stats.multicomp import pairwise_tukeyhsd
Data¶
The data used in this notebook is randomly generated, following the procedures outlined in this blog post with minor alterations for segmentation purposes.
# Set a random state variable and assign it to scipy's random state
ss.poisson.random_state = rand = np.random.RandomState(404)
# Generate random data selected from a weighted distribution
races = ["asian", "black", "hispanic", "other", "white"]
voter_race = rand.choice(a=races, p=[0.05, 0.15, 0.25, 0.05, 0.5], size=1000)
voter_age = ss.poisson.rvs(loc=18, mu=30, size=1000)
white_ages = ss.poisson.rvs(loc=18, mu=32, size=1000) # increase the mean age of white voters
age_shift = np.where(voter_race=="white", white_ages, voter_age) # Swap in altered white ages
Exploratory Data Analysis¶
The data used in this notebook attempts to mock a racial distribution of a voter population sample. As it is artificially generated, no preprocessing is required.
# Group age data by race
voters = pd.DataFrame({"race":voter_race, "age":voter_age})
groups = voters.groupby("race").groups
# Perform the ANOVA
ss.f_oneway(*[voter_age[v] for v in groups.values()])
F_onewayResult(statistic=0.26737204976027656, pvalue=0.8990043212082999)
With a F-statistic of 0.2673 and a p-value of 0.899, the one-way ANOVA test provides a strong evidence that there is not a significant difference between means of each group. Since the data was all pulled from the same mock distribution, this is exactly what we would expect to see.
Modify Distribution and Reanalyze¶
plt.hist(voter_age, alpha=0.5, label='all ages: $\mu=30$')
plt.hist(age_shift, alpha=0.5, label='white age: $\mu=32$')
plt.legend()
plt.show()
Now, we will be performing the same test on the modified dataset. Visually, we can already see how increasing the mean age of white voters effects the distribution.
# Group age data by race
voters_shift = pd.DataFrame({"race":voter_race, "age":age_shift})
groups_shift = voters_shift.groupby("race").groups
# Perform the ANOVA
ss.f_oneway(*[age_shift[v] for v in groups_shift.values()])
F_onewayResult(statistic=13.550717021659178, pvalue=9.24549061364278e-11)
After the change, the F-statistic is 13.55 and the p-value is 9.245e-11
A p-value of this size is many orders of magnitude smaller than p=0.05, a commonly used significance level. This serves as a very strong indication that there are indeed differences between the groups.
An F-statistic in the simplest sense is just the ratio of two variances. Building on this, an F-test is used for assessing whether the statistic follows an F-distribution under the null hypothesis.
A one-way ANOVA test uses F-tests as means to determine whether variance is due to differences within groups of data or differences between the groups. Put another way, a one-way ANOVA test assess whether variance between two or more (typically three or more) sample means is statistically significant.
# Get all unique racial pairings
race_pairs = [(races[r1], races[r2]) for r1 in range(4) for r2 in range(r1+1,5)]
# Conduct t-test on each pair for the altered data
for race1, race2 in race_pairs:
print(race1, race2,'\t:', ss.ttest_ind(age_shift[groups_shift[race1]], age_shift[groups_shift[race2]]))
asian black : Ttest_indResult(statistic=0.5081230371721365, pvalue=0.6119242403088412) asian hispanic : Ttest_indResult(statistic=0.6204493257066322, pvalue=0.5354223107249675) asian other : Ttest_indResult(statistic=0.8804714769956019, pvalue=0.38034939855706185) asian white : Ttest_indResult(statistic=-2.6589616806422485, pvalue=0.008072090696940136) black hispanic : Ttest_indResult(statistic=0.12543582439538467, pvalue=0.9002425060815309) black other : Ttest_indResult(statistic=0.6237279268453277, pvalue=0.5334896890146905) black white : Ttest_indResult(statistic=-4.852584065988863, pvalue=1.5413623060362876e-06) hispanic other : Ttest_indResult(statistic=0.5585844610829792, pvalue=0.5768434528987283) hispanic white : Ttest_indResult(statistic=-6.0120762205487575, pvalue=2.89291138112763e-09) other white : Ttest_indResult(statistic=-4.130139676281511, pvalue=4.195825609847373e-05)
# print the most likely candidates based on p-value and significance level
for race1, race2 in race_pairs:
siglvl = 0.05
result = ss.ttest_ind(age_shift[groups_shift[race1]], age_shift[groups_shift[race2]])
if result.pvalue < siglvl:
print(race1, race2,'\t:',result)
asian white : Ttest_indResult(statistic=-2.6589616806422485, pvalue=0.008072090696940136) black white : Ttest_indResult(statistic=-4.852584065988863, pvalue=1.5413623060362876e-06) hispanic white : Ttest_indResult(statistic=-6.0120762205487575, pvalue=2.89291138112763e-09) other white : Ttest_indResult(statistic=-4.130139676281511, pvalue=4.195825609847373e-05)
With the one-way ANOVA test indicating a significant between groups did exist, conducting a pairwise t-test on each group pairing allows us to hone in on which group(s) stands out.
The p-values should be taken with a grain of salt, however, since as you increase the number of comparisons made, you as well introduce more opportunities for random chance to play a large factor. One of the simplest ways to address this is issue is with a Bonferroni correction.
A Bonferroni correction addresses the Multiple comparisons problem by simply dividing the starting significance level (e.g. p=0.05) by the number of comparisons made (in this case 10).
# endog = data, alpha = Significance level
tukey2 = pairwise_tukeyhsd(endog=age_shift, groups=voter_race, alpha=0.05)
tukey2.plot_simultaneous() # Plot group confidence intervals
plt.vlines(x=49.55, ymin=-0.5, ymax=4.5, color="red")
tukey2.summary()
| group1 | group2 | meandiff | lower | upper | reject |
|---|---|---|---|---|---|
| asian | black | -0.4483 | -2.8332 | 1.9367 | False |
| asian | hispanic | -0.5205 | -2.757 | 1.7159 | False |
| asian | other | -0.9576 | -3.7303 | 1.8152 | False |
| asian | white | 2.0986 | -0.0352 | 4.2323 | False |
| black | hispanic | -0.0723 | -1.6735 | 1.529 | False |
| black | other | -0.5093 | -2.8007 | 1.7821 | False |
| black | white | 2.5468 | 1.0925 | 4.0012 | True |
| hispanic | other | -0.437 | -2.5734 | 1.6994 | False |
| hispanic | white | 2.6191 | 1.4238 | 3.8144 | True |
| other | white | 3.0561 | 1.0275 | 5.0847 | True |
Tukey's test is more exacting in the way that it approaches the Multiple comparisons problem. Rather than conducting pairwise t-tests and adjusting the significance level to compensate, Tukey's test, in effect, combines t-tests with an adjustment made to correct for the family-wise error rate.
Tukey's test is not the most precise for all cases but works quite well when confidence intervals are needed or sample sizes are unequal.
Conclusions¶
This note demonstrated the use of:
- one-way ANOVA test
- pairwise t-tests
- Tukey's range test
on a generated dataset representing fictitious voter demographics.
Future works could involve the use of a real vote demographic dataset, as opposed to a mocked up one, exploring other methods of dealing with the multiple comparison problem (e.g. Scheffé's method, Newman–Keuls method), and potentially using other variants of ANOVA tests.
References¶
Text:¶
- http://hamelg.blogspot.com/2015/11/python-for-data-analysis-part-16_23.html
- http://statisticsbyjim.com/anova/f-tests-anova/
- https://www.statisticshowto.datasciencecentral.com/probability-and-statistics/f-statistic-value-test/#FandP
- http://blog.minitab.com/blog/adventures-in-statistics-2/understanding-analysis-of-variance-anova-and-the-f-test
- https://en.wikipedia.org/wiki/One-way_analysis_of_variance
- https://en.wikipedia.org/wiki/F-test
Code:¶
Tags: ANOVA, python, statistics, ttests, voting
Categories: python, statistics