Documentation ¶

Contents¶

Partial confounder test
Full confounder test
Module ‘mlconfound.stats’
Module ‘mlconfound.plot’
Module ‘mlconfound.simulate’

Documentation indices¶

Partial confounder test¶

mlconfound.stats.partial_confound_test(y, yhat, c, num_perms=1000, cat_y=False, cat_yhat=False, cat_c=False, mcmc_steps=50, cond_dist_method='gam', return_null_dist=False, random_state=None, progress=True, n_jobs=-1)[source]¶

The Partial Confounder Test, to test partial confounder bias in machine learning models, based on the target variable, the confounder and the model predictions. Low p-value indicates that the model predictions are partially driven by the confoudner.

Parameters:

yarray_like: Target variable.
yhatarray_like: Predictions.
carray_like: Confounder variable.
num_permsint: Number of conditional permutations.
cat_ybool: Flag for categorical target variable (classification).
cat_yhatbool: Flag for categorical predictions (classification). Must be false, if class probabilities are used.
cat_cbool: Flag for categorical confounder variable (e.g center, sex or batch).
mcmc_stepsint: Number of steps for the Markov-chain Monte Carlo sampling.
cond_dist_methodstr: Method to estimate the c|y conditional distribution. Can be “linear” or “gam”. Recommended: “gam”.
return_null_distbool: Whether permuted null distribution should be returned (e.g. for plotting purposes).
random_stateint or None: Random state for the conditional permutation.
progressbool: Whether to print out progress bar.
n_jobsint: Number of parallel jobs (-1: use all available processors).

Returns:

ResultsPartiallyConfounded

Named tuple with the fields:

“r2_y_c”: coefficient-of-determination between y and c,
“r2_y_yhat”: coefficient-of-determination between y and yhat,
“r2_yhat_c”: coefficient-of-determination between yhat and c,
“expected_r2_yhat_c”: expected (0.05, 0.5, 0.95) quantiles of the coefficient-of-determination between yhat

and c (given y),

“p”: p-value,
“p_ci”: binomial 95% confidence interval for the p-value,
“null_distribution”: numpy.ndarray containing the permuted null distribution or None depending on teh value of
return_null_dist.

See also

full_confound_test
generalization_test

Notes

Performs the ‘partial confounder test’, a statistical test described in [1]_, based on the conditional permutation test for independence [2]_, using a linear or a general additive model (for numerical y, based on the parameter cond_dist_method) and a multinomial logit model (for categorical c, undependent on cond_dist_method) to estimate the c|y conditional distribution. This allows handling non-normal and non-linear dependencies between the target variable, the model output and the confounder variable. The null hypothesis of the test is that the model predictions are independent of the confounder, given the target variable, i.e. there is no-confounder bias. A low p-value therefore indicates significant confounder bias.

References

[1] Spisak, Tamas “A conditional permutation-based approach to test confounder effect: and center-bias in machine learning models”, in prep.

[2] Berrett, Thomas B., et al. “The conditional permutation test for independence while controlling for confounders.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 82.1 (2020): 175-197.

Examples

See notebooks/quickstart.ipynb for more detailed examples.

>>> partial_confound_test(y=[1,2,3,4,5,6], yhat=[1.5,2.3,2.9,4.2,5,5.7], c=[3,5,4,6,1,2], random_state=42, num_perms=100).p
1.0

Full confounder test¶

mlconfound.stats.full_confound_test(y, yhat, c, num_perms=1000, cat_y=False, cat_yhat=False, cat_c=False, mcmc_steps=50, cond_dist_method='gam', return_null_dist=False, random_state=None, progress=True, n_jobs=-1)[source]¶

The Full Confounder Test, to test for full confounder bias in machine learning models, based on the target variable, the confounder and the model predictions. Low p-value indicates that the model predictions are not fully driven by the confoudner.

Parameters:

yarray_like: Target variable.
yhatarray_like: Predictions.
carray_like: Confounder variable.
num_permsint: Number of conditional permutations.
cat_ybool: Flag for categorical target variable (classification).
cat_yhatbool: Flag for categorical predictions (classification). Must be false, if class probabilities are used.
cat_cbool: Flag for categorical confounder variable (e.g center, sex or batch).
mcmc_stepsint: Number of steps for the Markov-chain Monte Carlo sampling.
cond_dist_methodstr: Method to estimate the y|c conditional distribution. Can be “linear” or “gam”. Recommended: “gam”.
return_null_distbool: Whether permuted null distribution should be returned (e.g. for plotting purposes).
random_stateint or None: Random state for the conditional permutation.
progressbool: Whether to print out progress bar.
n_jobsint: Number of parallel jobs (-1: use all available processors).

Returns:

ResultsFullyConfounded

Named tuple with the fields:

“r2_y_c”: coefficient-of-determination between y and c,
“r2_yhat_c”: coefficient-of-determination between yhat and c,
“r2_y_yhat”: coefficient-of-determination between y and yhat,
“expected_r2_y_yhat”: expected (0.05, 0.5, 0.95) quantiles of the coefficient-of-determination between y and yhat (given c),
“p”: p-value,
“p_ci”: binomial 95% confidence interval for the p-value,
“null_distribution”: numpy.ndarray containing the permuted null distribution or None depending on teh value of
return_null_dist.

See also

partial_confound_test
generalization_test

Notes

Performs the ‘full confounder test’, a statistical test described in [1]_, based on the conditional permutation test for independence [2]_, using a linear or a general additive model (for numerical y, based on the parameter cond_dist_method) and a multinomial logit model (for categorical y, undependent on cond_dist_method) to estimate the y|c conditional distribution. This allows handling non-normal and non-linear dependencies between the target variable, the model output and the confounder variable. The null hypothesis of the test is that the model predictions are independent of the target variable, given the confounder variable, i.e. the model is entirely driven by the confounder. A low p-value therefore indicates that the model is not fully driven by the confounder.

References

[1] Spisak, Tamas “A conditional permutation-based approach to test confounder effect: and center-bias in machine learning models”, in prep.

[2] Berrett, Thomas B., et al. “The conditional permutation test for independence while controlling for confounders.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 82.1 (2020): 175-197.

Examples

See notebooks/quickstart.ipynb for more detailed examples.

>>> full_confound_test(y=[1,2,3,4,5,6], yhat=[1.5,2.3,2.9,4.2,5,5.7], c=[3,5,4,6,1,2], random_state=42, num_perms=100).p
1.0

Module ‘mlconfound.stats’¶

class mlconfound.stats.CptResults(r2_x_z, r2_y_z, r2_x_y, expected_r2_x_y, p, p_ci, null_distribution)¶

Attributes:

expected_r2_x_y: Alias for field number 3
null_distribution: Alias for field number 6
p: Alias for field number 4
p_ci: Alias for field number 5
r2_x_y: Alias for field number 2
r2_x_z: Alias for field number 0
r2_y_z: Alias for field number 1

Methods

`count`(value, /)	Return number of occurrences of value.
`index`(value[, start, stop])	Return first index of value.

property expected_r2_x_y¶: Alias for field number 3

property null_distribution¶: Alias for field number 6

property p¶: Alias for field number 4

property p_ci¶: Alias for field number 5

property r2_x_y¶: Alias for field number 2

property r2_x_z¶: Alias for field number 0

property r2_y_z¶: Alias for field number 1

class mlconfound.stats.ResultsFullyConfounded(r2_y_c, r2_yhat_c, r2_y_yhat, expected_r2_y_yhat, p, p_ci, null_distribution)¶

Attributes:

expected_r2_y_yhat: Alias for field number 3
null_distribution: Alias for field number 6
p: Alias for field number 4
p_ci: Alias for field number 5
r2_y_c: Alias for field number 0
r2_y_yhat: Alias for field number 2
r2_yhat_c: Alias for field number 1

Methods

`count`(value, /)	Return number of occurrences of value.
`index`(value[, start, stop])	Return first index of value.

property expected_r2_y_yhat¶: Alias for field number 3

property null_distribution¶: Alias for field number 6

property p¶: Alias for field number 4

property p_ci¶: Alias for field number 5

property r2_y_c¶: Alias for field number 0

property r2_y_yhat¶: Alias for field number 2

property r2_yhat_c¶: Alias for field number 1

class mlconfound.stats.ResultsPartiallyConfounded(r2_y_c, r2_y_yhat, r2_yhat_c, expected_r2_yhat_c, p, p_ci, null_distribution)¶

Attributes:

expected_r2_yhat_c: Alias for field number 3
null_distribution: Alias for field number 6
p: Alias for field number 4
p_ci: Alias for field number 5
r2_y_c: Alias for field number 0
r2_y_yhat: Alias for field number 1
r2_yhat_c: Alias for field number 2

Methods

`count`(value, /)	Return number of occurrences of value.
`index`(value[, start, stop])	Return first index of value.

property expected_r2_yhat_c¶: Alias for field number 3

property null_distribution¶: Alias for field number 6

property p¶: Alias for field number 4

property p_ci¶: Alias for field number 5

property r2_y_c¶: Alias for field number 0

property r2_y_yhat¶: Alias for field number 1

property r2_yhat_c¶: Alias for field number 2

mlconfound.stats.full_confound_test(y, yhat, c, num_perms=1000, cat_y=False, cat_yhat=False, cat_c=False, mcmc_steps=50, cond_dist_method='gam', return_null_dist=False, random_state=None, progress=True, n_jobs=-1)[source]¶

The Full Confounder Test, to test for full confounder bias in machine learning models, based on the target variable, the confounder and the model predictions. Low p-value indicates that the model predictions are not fully driven by the confoudner.

Parameters:

yarray_like: Target variable.
yhatarray_like: Predictions.
carray_like: Confounder variable.
num_permsint: Number of conditional permutations.
cat_ybool: Flag for categorical target variable (classification).
cat_yhatbool: Flag for categorical predictions (classification). Must be false, if class probabilities are used.
cat_cbool: Flag for categorical confounder variable (e.g center, sex or batch).
mcmc_stepsint: Number of steps for the Markov-chain Monte Carlo sampling.
cond_dist_methodstr: Method to estimate the y|c conditional distribution. Can be “linear” or “gam”. Recommended: “gam”.
return_null_distbool: Whether permuted null distribution should be returned (e.g. for plotting purposes).
random_stateint or None: Random state for the conditional permutation.
progressbool: Whether to print out progress bar.
n_jobsint: Number of parallel jobs (-1: use all available processors).

Returns:

ResultsFullyConfounded

Named tuple with the fields:

“r2_y_c”: coefficient-of-determination between y and c,
“r2_yhat_c”: coefficient-of-determination between yhat and c,
“r2_y_yhat”: coefficient-of-determination between y and yhat,
“expected_r2_y_yhat”: expected (0.05, 0.5, 0.95) quantiles of the coefficient-of-determination between y and yhat (given c),
“p”: p-value,
“p_ci”: binomial 95% confidence interval for the p-value,
“null_distribution”: numpy.ndarray containing the permuted null distribution or None depending on teh value of
return_null_dist.

See also

partial_confound_test
generalization_test

Notes

Performs the ‘full confounder test’, a statistical test described in [1]_, based on the conditional permutation test for independence [2]_, using a linear or a general additive model (for numerical y, based on the parameter cond_dist_method) and a multinomial logit model (for categorical y, undependent on cond_dist_method) to estimate the y|c conditional distribution. This allows handling non-normal and non-linear dependencies between the target variable, the model output and the confounder variable. The null hypothesis of the test is that the model predictions are independent of the target variable, given the confounder variable, i.e. the model is entirely driven by the confounder. A low p-value therefore indicates that the model is not fully driven by the confounder.

References

[1] Spisak, Tamas “A conditional permutation-based approach to test confounder effect: and center-bias in machine learning models”, in prep.

[2] Berrett, Thomas B., et al. “The conditional permutation test for independence while controlling for confounders.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 82.1 (2020): 175-197.

Examples

See notebooks/quickstart.ipynb for more detailed examples.

>>> full_confound_test(y=[1,2,3,4,5,6], yhat=[1.5,2.3,2.9,4.2,5,5.7], c=[3,5,4,6,1,2], random_state=42, num_perms=100).p
1.0

mlconfound.stats.generalization_test(y, yhat, c, num_perms=1000, cat_y=False, cat_yhat=False, cat_c=False, mcmc_steps=50, cond_dist_method='gam', return_null_dist=False, random_state=None, progress=True, n_jobs=-1)[source]¶

The Generalization Test is simply an alias for the Partial Confounder Test, to test if the machine learning model generalizes to a third, “positive validator” variable. Low p-value indicates that the model predictions explain more variance in the “positive validator” variable than expected from its correlation to the target, indicating that the model is driven by signal that is directly related to the positive validator.

Parameters:

yarray_like: Target variable.
yhatarray_like: Predictions.
carray_like: Positive validator variable.
num_permsint: Number of conditional permutations.
cat_ybool: Flag for categorical target variable (classification).
cat_yhatbool: Flag for categorical predictions (classification). Must be false, if class probabilities are used.
cat_cbool: Flag for categorical confounder variable (e.g center, sex or batch).
mcmc_stepsint: Number of steps for the Markov-chain Monte Carlo sampling.
cond_dist_methodstr: Method to estimate the c|y conditional distribution. Can be “linear” or “gam”. Recommended: “gam”.
return_null_distbool: Whether permuted null distribution should be returned (e.g. for plotting purposes).
random_stateint or None: Random state for the conditional permutation.
progressbool: Whether to print out progress bar.
n_jobsint: Number of parallel jobs (-1: use all available processors).

Returns:

ResultsPartiallyConfounded

Named tuple with the fields:

“r2_y_c”: coefficient-of-determination between y and c,
“r2_yhat_c”: coefficient-of-determination between yhat and c,
“r2_y_yhat”: coefficient-of-determination between y and yhat,
“p”: p-value,
“p_ci”: binomial 95% confidence interval for the p-value,
“null_distribution”: numpy.ndarray containing the permuted null distribution or None depending on teh value of
return_null_dist.

See also

partial_confound_test

Notes

Under the hood, the Generalization Test simply performs the ‘partial confounder test’, a statistical test described in [1]_, based on the conditional permutation test for independence [2]_, using a linear or a general additive model (for numerical y, based on the parameter cond_dist_method) and a multinomial logit model (for categorical c, undependent on cond_dist_method) to estimate the c|y conditional distribution. This allows handling non-normal and non-linear dependencies between the target variable, the model output and the confounder variable. The null hypothesis of the test is that the model predictions are independent of the positive validator variable, given the target variable, i.e. there is true generalization. A low p-value therefore indicates a significant generalization value.

References

[1] Spisak, Tamas “A conditional permutation-based approach to test confounder effect: and center-bias in machine learning models”, in prep.

[2] Berrett, Thomas B., et al. “The conditional permutation test for independence while controlling for confounders.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 82.1 (2020): 175-197.

Examples

See notebooks/quickstart.ipynb for more detailed examples.

>>> generalization_test(y=[1,2,3,4,5,6], yhat=[1.5,2.3,2.9,4.2,5,5.7], c=[3,5,4,6,1,2], random_state=42, num_perms=100).p
1.0

mlconfound.stats.partial_confound_test(y, yhat, c, num_perms=1000, cat_y=False, cat_yhat=False, cat_c=False, mcmc_steps=50, cond_dist_method='gam', return_null_dist=False, random_state=None, progress=True, n_jobs=-1)[source]¶

The Partial Confounder Test, to test partial confounder bias in machine learning models, based on the target variable, the confounder and the model predictions. Low p-value indicates that the model predictions are partially driven by the confoudner.

Parameters:

yarray_like: Target variable.
yhatarray_like: Predictions.
carray_like: Confounder variable.
num_permsint: Number of conditional permutations.
cat_ybool: Flag for categorical target variable (classification).
cat_yhatbool: Flag for categorical predictions (classification). Must be false, if class probabilities are used.
cat_cbool: Flag for categorical confounder variable (e.g center, sex or batch).
mcmc_stepsint: Number of steps for the Markov-chain Monte Carlo sampling.
cond_dist_methodstr: Method to estimate the c|y conditional distribution. Can be “linear” or “gam”. Recommended: “gam”.
return_null_distbool: Whether permuted null distribution should be returned (e.g. for plotting purposes).
random_stateint or None: Random state for the conditional permutation.
progressbool: Whether to print out progress bar.
n_jobsint: Number of parallel jobs (-1: use all available processors).

Returns:

ResultsPartiallyConfounded

Named tuple with the fields:

“r2_y_c”: coefficient-of-determination between y and c,
“r2_y_yhat”: coefficient-of-determination between y and yhat,
“r2_yhat_c”: coefficient-of-determination between yhat and c,
“expected_r2_yhat_c”: expected (0.05, 0.5, 0.95) quantiles of the coefficient-of-determination between yhat

and c (given y),

“p”: p-value,
“p_ci”: binomial 95% confidence interval for the p-value,
“null_distribution”: numpy.ndarray containing the permuted null distribution or None depending on teh value of
return_null_dist.

See also

full_confound_test
generalization_test

Notes

Performs the ‘partial confounder test’, a statistical test described in [1]_, based on the conditional permutation test for independence [2]_, using a linear or a general additive model (for numerical y, based on the parameter cond_dist_method) and a multinomial logit model (for categorical c, undependent on cond_dist_method) to estimate the c|y conditional distribution. This allows handling non-normal and non-linear dependencies between the target variable, the model output and the confounder variable. The null hypothesis of the test is that the model predictions are independent of the confounder, given the target variable, i.e. there is no-confounder bias. A low p-value therefore indicates significant confounder bias.

References

[1] Spisak, Tamas “A conditional permutation-based approach to test confounder effect: and center-bias in machine learning models”, in prep.

[2] Berrett, Thomas B., et al. “The conditional permutation test for independence while controlling for confounders.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 82.1 (2020): 175-197.

Examples

See notebooks/quickstart.ipynb for more detailed examples.

>>> partial_confound_test(y=[1,2,3,4,5,6], yhat=[1.5,2.3,2.9,4.2,5,5.7], c=[3,5,4,6,1,2], random_state=42, num_perms=100).p
1.0

Module ‘mlconfound.plot’¶

mlconfound.plot.plot_graph(confound_test_results, y_name='y', yhat_name='<ŷ>', c_name='c', outfile_base=None, precision=3)[source]¶

Plot confounder test results as a graph depicting the involved variables.

Parameters:

confound_test_resultsnamedtuple: The object returned by test_partially_confounded or test_fully_confounded.
y_name: str: Name of the target variable.
yhat_name: str: Name for the model predictions.
c_name: str: Name of the confounder variable.
outfile_base: str: Path for output files, without extension (None: figure not saved).
precision: int: Precision for r squared values.

Returns:

dot: The graphviz object to plot.

See also

plot_null_dist

mlconfound.plot.plot_null_dist(confound_test_results, **kwargs)[source]¶

Plot the histogram of the permutation-based null distribution of the confounder test.

Parameters:

confound_test_results: namedtuple: The object returned by test_partially_confounded or test_fully_confounded.
kwargsdict: Additional named argumnets, passed to seaborn.histplot.

Returns:

matplotlib.axes.Axes: The matplotlib axes containing the plot.

See also

plot_graph

Module ‘mlconfound.simulate’¶

mlconfound.simulate.identity(x)[source]¶

Identity transformation

Parameters:

x

Returns:

See also

simulate_y_c_yhat

Notes

To be used as a transformation function in simulate_y_c_yhat.

mlconfound.simulate.simulate_y_c_yhat(w_yc, w_yyhat, w_cyhat, n, delta=1, epsilon=0, nonlin_trf_fun=<function identity>, bin_y=False, bin_c=False, bin_yhat=False, scale=True, random_state=None)[source]¶

Simulate normally distributed target (y), confounder (c) and predictions (yhat).

Parameters:

w_yc: float: The weight of y in c.
w_yyhat: float: The weight of y in yhat.
w_cyhat: float: The weight of c in yhat. Set it to zero for H0.
n: int: Number of observations.
delta: float: The delta param of the sinh_archsin transformation on y’s contribution in yhat (only affects yhat).
epsilon: float: The epsilon param of the sinh_archsin transformation on y’s contribution in yhat (only affects yhat).
nonlin_trf_fun: callable: Callable to introduce non-linearity in the conditional distributions. (default: no non-linearity).
scale: bool: Scale variables to zero mean and unit variance.
random_state: int: Numpy random state.

Returns:

tuple

y: the simulated target variable
c: the simulated confounder variable
yhat: the simulated predictions

See also

sinh_arcsinh

Notes

\[y \sim \mathcal{N}(0, 1)\]

\[c | y_i \sim \mathcal{N}(w_{y,c} f(y_i), 1)\]

\[\hat{y} | y_i, c_i \sim sinh(\delta sinh^{-1}( \mathcal{N}(w_{y,c} f(y_i), 1))-\epsilon)\]

Examples

>>> y, c, yhat = simulate_y_c_yhat(0.3, 0.2, 0.2, n=3, random_state=42)
>>> print(y, c, yhat)
[ 0.30471708 -1.03998411  0.7504512 ] [ 1.32193037 -1.09613765 -0.22579272] [ 1.03955979 -1.35013318  0.31057339]

mlconfound.simulate.sinh_arcsinh(x, delta=1, epsilon=0)[source]¶

Sinh-arcsinh transformation

Parameters:

xarray_like: Normally distributed input data.
deltafloat: Parameter to control kurtosis, delta=1 means no change.
epsilonfloat: Parameter to control skewness, epsilon=0 means no change.
Returns
——-
array_like: Transformed data.

See also

simulate_y_c_yhat

Notes

The sinh-arcsinh transformation of Jones and Pewsey [1]_ can be used to transfrom Normal distribution to non-normal.

References

[1] Jones, M. C. and Pewsey A. (2009). Sinh-arcsinh distributions. Biometrika 96: 761–780

Examples

See validation/simulation.py for an application example.

>>> result = sinh_arcsinh([-1, -0.5, -0.1, 0.1, 0.5, 1], delta=2, epsilon=1)
>>> print(result)
[-7.8900947  -3.48801839 -1.50886059 -0.88854985 -0.03758519  0.83888754]

>>> result = sinh_arcsinh([-1, -0.5, -0.1, 0.1, 0.5, 1], delta=1, epsilon=0)
>>> print(result)
[-1.  -0.5 -0.1  0.1  0.5  1. ]

Documentation ¶

Contents¶

Documentation indices¶

Partial confounder test¶

Full confounder test¶

Module ‘mlconfound.stats’¶

Module ‘mlconfound.plot’¶

Module ‘mlconfound.simulate’¶

Navigation:¶

mlconfound

Navigation

Related Topics