Documentation ¶
Contents¶
Partial confounder test
Full confounder test
Module ‘mlconfound.stats’
Module ‘mlconfound.plot’
Module ‘mlconfound.simulate’
Documentation indices¶
Partial confounder test¶
- mlconfound.stats.partial_confound_test(y, yhat, c, num_perms=1000, cat_y=False, cat_yhat=False, cat_c=False, mcmc_steps=50, cond_dist_method='gam', return_null_dist=False, random_state=None, progress=True, n_jobs=-1)[source]¶
The Partial Confounder Test, to test partial confounder bias in machine learning models, based on the target variable, the confounder and the model predictions. Low p-value indicates that the model predictions are partially driven by the confoudner.
- Parameters:
- yarray_like
Target variable.
- yhatarray_like
Predictions.
- carray_like
Confounder variable.
- num_permsint
Number of conditional permutations.
- cat_ybool
Flag for categorical target variable (classification).
- cat_yhatbool
Flag for categorical predictions (classification). Must be false, if class probabilities are used.
- cat_cbool
Flag for categorical confounder variable (e.g center, sex or batch).
- mcmc_stepsint
Number of steps for the Markov-chain Monte Carlo sampling.
- cond_dist_methodstr
Method to estimate the c|y conditional distribution. Can be “linear” or “gam”. Recommended: “gam”.
- return_null_distbool
Whether permuted null distribution should be returned (e.g. for plotting purposes).
- random_stateint or None
Random state for the conditional permutation.
- progressbool
Whether to print out progress bar.
- n_jobsint
Number of parallel jobs (-1: use all available processors).
- Returns:
- ResultsPartiallyConfounded
Named tuple with the fields:
“r2_y_c”: coefficient-of-determination between y and c,
“r2_y_yhat”: coefficient-of-determination between y and yhat,
“r2_yhat_c”: coefficient-of-determination between yhat and c,
“expected_r2_yhat_c”: expected (0.05, 0.5, 0.95) quantiles of the coefficient-of-determination between yhat
and c (given y),
“p”: p-value,
“p_ci”: binomial 95% confidence interval for the p-value,
- “null_distribution”: numpy.ndarray containing the permuted null distribution or None depending on teh value of
return_null_dist.
See also
Notes
Performs the ‘partial confounder test’, a statistical test described in [1]_, based on the conditional permutation test for independence [2]_, using a linear or a general additive model (for numerical y, based on the parameter cond_dist_method) and a multinomial logit model (for categorical c, undependent on cond_dist_method) to estimate the c|y conditional distribution. This allows handling non-normal and non-linear dependencies between the target variable, the model output and the confounder variable. The null hypothesis of the test is that the model predictions are independent of the confounder, given the target variable, i.e. there is no-confounder bias. A low p-value therefore indicates significant confounder bias.
References
- [1] Spisak, Tamas “A conditional permutation-based approach to test confounder effect
and center-bias in machine learning models”, in prep.
[2] Berrett, Thomas B., et al. “The conditional permutation test for independence while controlling for confounders.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 82.1 (2020): 175-197.
Examples
See notebooks/quickstart.ipynb for more detailed examples.
>>> partial_confound_test(y=[1,2,3,4,5,6], yhat=[1.5,2.3,2.9,4.2,5,5.7], c=[3,5,4,6,1,2], random_state=42, num_perms=100).p 1.0
Full confounder test¶
- mlconfound.stats.full_confound_test(y, yhat, c, num_perms=1000, cat_y=False, cat_yhat=False, cat_c=False, mcmc_steps=50, cond_dist_method='gam', return_null_dist=False, random_state=None, progress=True, n_jobs=-1)[source]¶
The Full Confounder Test, to test for full confounder bias in machine learning models, based on the target variable, the confounder and the model predictions. Low p-value indicates that the model predictions are not fully driven by the confoudner.
- Parameters:
- yarray_like
Target variable.
- yhatarray_like
Predictions.
- carray_like
Confounder variable.
- num_permsint
Number of conditional permutations.
- cat_ybool
Flag for categorical target variable (classification).
- cat_yhatbool
Flag for categorical predictions (classification). Must be false, if class probabilities are used.
- cat_cbool
Flag for categorical confounder variable (e.g center, sex or batch).
- mcmc_stepsint
Number of steps for the Markov-chain Monte Carlo sampling.
- cond_dist_methodstr
Method to estimate the y|c conditional distribution. Can be “linear” or “gam”. Recommended: “gam”.
- return_null_distbool
Whether permuted null distribution should be returned (e.g. for plotting purposes).
- random_stateint or None
Random state for the conditional permutation.
- progressbool
Whether to print out progress bar.
- n_jobsint
Number of parallel jobs (-1: use all available processors).
- Returns:
- ResultsFullyConfounded
Named tuple with the fields:
“r2_y_c”: coefficient-of-determination between y and c,
“r2_yhat_c”: coefficient-of-determination between yhat and c,
“r2_y_yhat”: coefficient-of-determination between y and yhat,
“expected_r2_y_yhat”: expected (0.05, 0.5, 0.95) quantiles of the coefficient-of-determination between y and yhat (given c),
“p”: p-value,
“p_ci”: binomial 95% confidence interval for the p-value,
- “null_distribution”: numpy.ndarray containing the permuted null distribution or None depending on teh value of
return_null_dist.
See also
Notes
Performs the ‘full confounder test’, a statistical test described in [1]_, based on the conditional permutation test for independence [2]_, using a linear or a general additive model (for numerical y, based on the parameter cond_dist_method) and a multinomial logit model (for categorical y, undependent on cond_dist_method) to estimate the y|c conditional distribution. This allows handling non-normal and non-linear dependencies between the target variable, the model output and the confounder variable. The null hypothesis of the test is that the model predictions are independent of the target variable, given the confounder variable, i.e. the model is entirely driven by the confounder. A low p-value therefore indicates that the model is not fully driven by the confounder.
References
- [1] Spisak, Tamas “A conditional permutation-based approach to test confounder effect
and center-bias in machine learning models”, in prep.
[2] Berrett, Thomas B., et al. “The conditional permutation test for independence while controlling for confounders.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 82.1 (2020): 175-197.
Examples
See notebooks/quickstart.ipynb for more detailed examples.
>>> full_confound_test(y=[1,2,3,4,5,6], yhat=[1.5,2.3,2.9,4.2,5,5.7], c=[3,5,4,6,1,2], random_state=42, num_perms=100).p 1.0
Module ‘mlconfound.stats’¶
- class mlconfound.stats.CptResults(r2_x_z, r2_y_z, r2_x_y, expected_r2_x_y, p, p_ci, null_distribution)¶
- Attributes:
expected_r2_x_y
Alias for field number 3
null_distribution
Alias for field number 6
p
Alias for field number 4
p_ci
Alias for field number 5
r2_x_y
Alias for field number 2
r2_x_z
Alias for field number 0
r2_y_z
Alias for field number 1
Methods
count
(value, /)Return number of occurrences of value.
index
(value[, start, stop])Return first index of value.
- property expected_r2_x_y¶
Alias for field number 3
- property null_distribution¶
Alias for field number 6
- property p¶
Alias for field number 4
- property p_ci¶
Alias for field number 5
- property r2_x_y¶
Alias for field number 2
- property r2_x_z¶
Alias for field number 0
- property r2_y_z¶
Alias for field number 1
- class mlconfound.stats.ResultsFullyConfounded(r2_y_c, r2_yhat_c, r2_y_yhat, expected_r2_y_yhat, p, p_ci, null_distribution)¶
- Attributes:
expected_r2_y_yhat
Alias for field number 3
null_distribution
Alias for field number 6
p
Alias for field number 4
p_ci
Alias for field number 5
r2_y_c
Alias for field number 0
r2_y_yhat
Alias for field number 2
r2_yhat_c
Alias for field number 1
Methods
count
(value, /)Return number of occurrences of value.
index
(value[, start, stop])Return first index of value.
- property expected_r2_y_yhat¶
Alias for field number 3
- property null_distribution¶
Alias for field number 6
- property p¶
Alias for field number 4
- property p_ci¶
Alias for field number 5
- property r2_y_c¶
Alias for field number 0
- property r2_y_yhat¶
Alias for field number 2
- property r2_yhat_c¶
Alias for field number 1
- class mlconfound.stats.ResultsPartiallyConfounded(r2_y_c, r2_y_yhat, r2_yhat_c, expected_r2_yhat_c, p, p_ci, null_distribution)¶
- Attributes:
expected_r2_yhat_c
Alias for field number 3
null_distribution
Alias for field number 6
p
Alias for field number 4
p_ci
Alias for field number 5
r2_y_c
Alias for field number 0
r2_y_yhat
Alias for field number 1
r2_yhat_c
Alias for field number 2
Methods
count
(value, /)Return number of occurrences of value.
index
(value[, start, stop])Return first index of value.
- property expected_r2_yhat_c¶
Alias for field number 3
- property null_distribution¶
Alias for field number 6
- property p¶
Alias for field number 4
- property p_ci¶
Alias for field number 5
- property r2_y_c¶
Alias for field number 0
- property r2_y_yhat¶
Alias for field number 1
- property r2_yhat_c¶
Alias for field number 2
- mlconfound.stats.full_confound_test(y, yhat, c, num_perms=1000, cat_y=False, cat_yhat=False, cat_c=False, mcmc_steps=50, cond_dist_method='gam', return_null_dist=False, random_state=None, progress=True, n_jobs=-1)[source]¶
The Full Confounder Test, to test for full confounder bias in machine learning models, based on the target variable, the confounder and the model predictions. Low p-value indicates that the model predictions are not fully driven by the confoudner.
- Parameters:
- yarray_like
Target variable.
- yhatarray_like
Predictions.
- carray_like
Confounder variable.
- num_permsint
Number of conditional permutations.
- cat_ybool
Flag for categorical target variable (classification).
- cat_yhatbool
Flag for categorical predictions (classification). Must be false, if class probabilities are used.
- cat_cbool
Flag for categorical confounder variable (e.g center, sex or batch).
- mcmc_stepsint
Number of steps for the Markov-chain Monte Carlo sampling.
- cond_dist_methodstr
Method to estimate the y|c conditional distribution. Can be “linear” or “gam”. Recommended: “gam”.
- return_null_distbool
Whether permuted null distribution should be returned (e.g. for plotting purposes).
- random_stateint or None
Random state for the conditional permutation.
- progressbool
Whether to print out progress bar.
- n_jobsint
Number of parallel jobs (-1: use all available processors).
- Returns:
- ResultsFullyConfounded
Named tuple with the fields:
“r2_y_c”: coefficient-of-determination between y and c,
“r2_yhat_c”: coefficient-of-determination between yhat and c,
“r2_y_yhat”: coefficient-of-determination between y and yhat,
“expected_r2_y_yhat”: expected (0.05, 0.5, 0.95) quantiles of the coefficient-of-determination between y and yhat (given c),
“p”: p-value,
“p_ci”: binomial 95% confidence interval for the p-value,
- “null_distribution”: numpy.ndarray containing the permuted null distribution or None depending on teh value of
return_null_dist.
See also
Notes
Performs the ‘full confounder test’, a statistical test described in [1]_, based on the conditional permutation test for independence [2]_, using a linear or a general additive model (for numerical y, based on the parameter cond_dist_method) and a multinomial logit model (for categorical y, undependent on cond_dist_method) to estimate the y|c conditional distribution. This allows handling non-normal and non-linear dependencies between the target variable, the model output and the confounder variable. The null hypothesis of the test is that the model predictions are independent of the target variable, given the confounder variable, i.e. the model is entirely driven by the confounder. A low p-value therefore indicates that the model is not fully driven by the confounder.
References
- [1] Spisak, Tamas “A conditional permutation-based approach to test confounder effect
and center-bias in machine learning models”, in prep.
[2] Berrett, Thomas B., et al. “The conditional permutation test for independence while controlling for confounders.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 82.1 (2020): 175-197.
Examples
See notebooks/quickstart.ipynb for more detailed examples.
>>> full_confound_test(y=[1,2,3,4,5,6], yhat=[1.5,2.3,2.9,4.2,5,5.7], c=[3,5,4,6,1,2], random_state=42, num_perms=100).p 1.0
- mlconfound.stats.generalization_test(y, yhat, c, num_perms=1000, cat_y=False, cat_yhat=False, cat_c=False, mcmc_steps=50, cond_dist_method='gam', return_null_dist=False, random_state=None, progress=True, n_jobs=-1)[source]¶
The Generalization Test is simply an alias for the Partial Confounder Test, to test if the machine learning model generalizes to a third, “positive validator” variable. Low p-value indicates that the model predictions explain more variance in the “positive validator” variable than expected from its correlation to the target, indicating that the model is driven by signal that is directly related to the positive validator.
- Parameters:
- yarray_like
Target variable.
- yhatarray_like
Predictions.
- carray_like
Positive validator variable.
- num_permsint
Number of conditional permutations.
- cat_ybool
Flag for categorical target variable (classification).
- cat_yhatbool
Flag for categorical predictions (classification). Must be false, if class probabilities are used.
- cat_cbool
Flag for categorical confounder variable (e.g center, sex or batch).
- mcmc_stepsint
Number of steps for the Markov-chain Monte Carlo sampling.
- cond_dist_methodstr
Method to estimate the c|y conditional distribution. Can be “linear” or “gam”. Recommended: “gam”.
- return_null_distbool
Whether permuted null distribution should be returned (e.g. for plotting purposes).
- random_stateint or None
Random state for the conditional permutation.
- progressbool
Whether to print out progress bar.
- n_jobsint
Number of parallel jobs (-1: use all available processors).
- Returns:
- ResultsPartiallyConfounded
Named tuple with the fields:
“r2_y_c”: coefficient-of-determination between y and c,
“r2_yhat_c”: coefficient-of-determination between yhat and c,
“r2_y_yhat”: coefficient-of-determination between y and yhat,
“p”: p-value,
“p_ci”: binomial 95% confidence interval for the p-value,
- “null_distribution”: numpy.ndarray containing the permuted null distribution or None depending on teh value of
return_null_dist.
See also
Notes
Under the hood, the Generalization Test simply performs the ‘partial confounder test’, a statistical test described in [1]_, based on the conditional permutation test for independence [2]_, using a linear or a general additive model (for numerical y, based on the parameter cond_dist_method) and a multinomial logit model (for categorical c, undependent on cond_dist_method) to estimate the c|y conditional distribution. This allows handling non-normal and non-linear dependencies between the target variable, the model output and the confounder variable. The null hypothesis of the test is that the model predictions are independent of the positive validator variable, given the target variable, i.e. there is true generalization. A low p-value therefore indicates a significant generalization value.
References
- [1] Spisak, Tamas “A conditional permutation-based approach to test confounder effect
and center-bias in machine learning models”, in prep.
[2] Berrett, Thomas B., et al. “The conditional permutation test for independence while controlling for confounders.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 82.1 (2020): 175-197.
Examples
See notebooks/quickstart.ipynb for more detailed examples.
>>> generalization_test(y=[1,2,3,4,5,6], yhat=[1.5,2.3,2.9,4.2,5,5.7], c=[3,5,4,6,1,2], random_state=42, num_perms=100).p 1.0
- mlconfound.stats.partial_confound_test(y, yhat, c, num_perms=1000, cat_y=False, cat_yhat=False, cat_c=False, mcmc_steps=50, cond_dist_method='gam', return_null_dist=False, random_state=None, progress=True, n_jobs=-1)[source]¶
The Partial Confounder Test, to test partial confounder bias in machine learning models, based on the target variable, the confounder and the model predictions. Low p-value indicates that the model predictions are partially driven by the confoudner.
- Parameters:
- yarray_like
Target variable.
- yhatarray_like
Predictions.
- carray_like
Confounder variable.
- num_permsint
Number of conditional permutations.
- cat_ybool
Flag for categorical target variable (classification).
- cat_yhatbool
Flag for categorical predictions (classification). Must be false, if class probabilities are used.
- cat_cbool
Flag for categorical confounder variable (e.g center, sex or batch).
- mcmc_stepsint
Number of steps for the Markov-chain Monte Carlo sampling.
- cond_dist_methodstr
Method to estimate the c|y conditional distribution. Can be “linear” or “gam”. Recommended: “gam”.
- return_null_distbool
Whether permuted null distribution should be returned (e.g. for plotting purposes).
- random_stateint or None
Random state for the conditional permutation.
- progressbool
Whether to print out progress bar.
- n_jobsint
Number of parallel jobs (-1: use all available processors).
- Returns:
- ResultsPartiallyConfounded
Named tuple with the fields:
“r2_y_c”: coefficient-of-determination between y and c,
“r2_y_yhat”: coefficient-of-determination between y and yhat,
“r2_yhat_c”: coefficient-of-determination between yhat and c,
“expected_r2_yhat_c”: expected (0.05, 0.5, 0.95) quantiles of the coefficient-of-determination between yhat
and c (given y),
“p”: p-value,
“p_ci”: binomial 95% confidence interval for the p-value,
- “null_distribution”: numpy.ndarray containing the permuted null distribution or None depending on teh value of
return_null_dist.
See also
Notes
Performs the ‘partial confounder test’, a statistical test described in [1]_, based on the conditional permutation test for independence [2]_, using a linear or a general additive model (for numerical y, based on the parameter cond_dist_method) and a multinomial logit model (for categorical c, undependent on cond_dist_method) to estimate the c|y conditional distribution. This allows handling non-normal and non-linear dependencies between the target variable, the model output and the confounder variable. The null hypothesis of the test is that the model predictions are independent of the confounder, given the target variable, i.e. there is no-confounder bias. A low p-value therefore indicates significant confounder bias.
References
- [1] Spisak, Tamas “A conditional permutation-based approach to test confounder effect
and center-bias in machine learning models”, in prep.
[2] Berrett, Thomas B., et al. “The conditional permutation test for independence while controlling for confounders.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 82.1 (2020): 175-197.
Examples
See notebooks/quickstart.ipynb for more detailed examples.
>>> partial_confound_test(y=[1,2,3,4,5,6], yhat=[1.5,2.3,2.9,4.2,5,5.7], c=[3,5,4,6,1,2], random_state=42, num_perms=100).p 1.0
Module ‘mlconfound.plot’¶
- mlconfound.plot.plot_graph(confound_test_results, y_name='y', yhat_name='<ŷ>', c_name='c', outfile_base=None, precision=3)[source]¶
Plot confounder test results as a graph depicting the involved variables.
- Parameters:
- confound_test_resultsnamedtuple
The object returned by test_partially_confounded or test_fully_confounded.
- y_name: str
Name of the target variable.
- yhat_name: str
Name for the model predictions.
- c_name: str
Name of the confounder variable.
- outfile_base: str
Path for output files, without extension (None: figure not saved).
- precision: int
Precision for r squared values.
- Returns:
- dot
The graphviz object to plot.
See also
- mlconfound.plot.plot_null_dist(confound_test_results, **kwargs)[source]¶
Plot the histogram of the permutation-based null distribution of the confounder test.
- Parameters:
- confound_test_results: namedtuple
The object returned by test_partially_confounded or test_fully_confounded.
- kwargsdict
Additional named argumnets, passed to seaborn.histplot.
- Returns:
- matplotlib.axes.Axes
The matplotlib axes containing the plot.
See also
Module ‘mlconfound.simulate’¶
- mlconfound.simulate.identity(x)[source]¶
Identity transformation
- Parameters:
- x
- Returns:
See also
Notes
To be used as a transformation function in simulate_y_c_yhat.
- mlconfound.simulate.simulate_y_c_yhat(w_yc, w_yyhat, w_cyhat, n, delta=1, epsilon=0, nonlin_trf_fun=<function identity>, bin_y=False, bin_c=False, bin_yhat=False, scale=True, random_state=None)[source]¶
Simulate normally distributed target (y), confounder (c) and predictions (yhat).
- Parameters:
- w_yc: float
The weight of y in c.
- w_yyhat: float
The weight of y in yhat.
- w_cyhat: float
The weight of c in yhat. Set it to zero for H0.
- n: int
Number of observations.
- delta: float
The delta param of the sinh_archsin transformation on y’s contribution in yhat (only affects yhat).
- epsilon: float
The epsilon param of the sinh_archsin transformation on y’s contribution in yhat (only affects yhat).
- nonlin_trf_fun: callable
Callable to introduce non-linearity in the conditional distributions. (default: no non-linearity).
- scale: bool
Scale variables to zero mean and unit variance.
- random_state: int
Numpy random state.
- Returns:
- tuple
y: the simulated target variable
c: the simulated confounder variable
yhat: the simulated predictions
See also
Notes
\[y \sim \mathcal{N}(0, 1)\]\[c | y_i \sim \mathcal{N}(w_{y,c} f(y_i), 1)\]\[\hat{y} | y_i, c_i \sim sinh(\delta sinh^{-1}( \mathcal{N}(w_{y,c} f(y_i), 1))-\epsilon)\]Examples
>>> y, c, yhat = simulate_y_c_yhat(0.3, 0.2, 0.2, n=3, random_state=42) >>> print(y, c, yhat) [ 0.30471708 -1.03998411 0.7504512 ] [ 1.32193037 -1.09613765 -0.22579272] [ 1.03955979 -1.35013318 0.31057339]
- mlconfound.simulate.sinh_arcsinh(x, delta=1, epsilon=0)[source]¶
Sinh-arcsinh transformation
- Parameters:
- xarray_like
Normally distributed input data.
- deltafloat
Parameter to control kurtosis, delta=1 means no change.
- epsilonfloat
Parameter to control skewness, epsilon=0 means no change.
- Returns
- ——-
- array_like
Transformed data.
See also
Notes
The sinh-arcsinh transformation of Jones and Pewsey [1]_ can be used to transfrom Normal distribution to non-normal.
References
[1] Jones, M. C. and Pewsey A. (2009). Sinh-arcsinh distributions. Biometrika 96: 761–780
Examples
See validation/simulation.py for an application example.
>>> result = sinh_arcsinh([-1, -0.5, -0.1, 0.1, 0.5, 1], delta=2, epsilon=1) >>> print(result) [-7.8900947 -3.48801839 -1.50886059 -0.88854985 -0.03758519 0.83888754]
>>> result = sinh_arcsinh([-1, -0.5, -0.1, 0.1, 0.5, 1], delta=1, epsilon=0) >>> print(result) [-1. -0.5 -0.1 0.1 0.5 1. ]