Documentation Documentation Status


Contents

  • Partial confounder test

  • Full confounder test

  • Module ‘mlconfound.stats’

  • Module ‘mlconfound.plot’

  • Module ‘mlconfound.simulate’

Documentation indices


Partial confounder test

mlconfound.stats.partial_confound_test(y, yhat, c, num_perms=1000, cat_y=False, cat_yhat=False, cat_c=False, mcmc_steps=50, cond_dist_method='gam', return_null_dist=False, random_state=None, progress=True, n_jobs=-1)[source]

The Partial Confounder Test, to test partial confounder bias in machine learning models, based on the target variable, the confounder and the model predictions. Low p-value indicates that the model predictions are partially driven by the confoudner.

Parameters
yarray_like

Target variable.

yhatarray_like

Predictions.

carray_like

Confounder variable.

num_permsint

Number of conditional permutations.

cat_ybool

Flag for categorical target variable (classification).

cat_yhatbool

Flag for categorical predictions (classification). Must be false, if class probabilities are used.

cat_cbool

Flag for categorical confounder variable (e.g center, sex or batch).

mcmc_stepsint

Number of steps for the Markov-chain Monte Carlo sampling.

cond_dist_methodstr

Method to estimate the c|y conditional distribution. Can be “linear” or “gam”. Recommended: “gam”.

return_null_distbool

Whether permuted null distribution should be returned (e.g. for plotting purposes).

random_stateint or None

Random state for the conditional permutation.

progressbool

Whether to print out progress bar.

n_jobsint

Number of parallel jobs (-1: use all available processors).

Returns
ResultsPartiallyConfounded

Named tuple with the fields:

  • “r2_y_c”: coefficient-of-determination between y and c,

  • “r2_yhat_c”: coefficient-of-determination between yhat and c,

  • “r2_y_yhat”: coefficient-of-determination between y and yhat,

  • “p”: p-value,

  • “p_ci”: binomial 95% confidence interval for the p-value,

  • “null_distribution”: numpy.ndarray containing the permuted null distribution or None depending on teh value of

    return_null_dist.

Notes

Performs the ‘partial confounder test’, a statistical test described in [1]_, based on the conditional permutation test for independence [2]_, using a linear or a general additive model (for numerical y, based on the parameter cond_dist_method) and a multinomial logit model (for categorical c, undependent on cond_dist_method) to estimate the c|y conditional distribution. This allows handling non-normal and non-linear dependencies between the target variable, the model output and the confounder variable. The null hypothesis of the test is that the model predictions are independent of the confounder, given the target variable, i.e. there is no-confounder bias. A low p-value therefore indicates significant confounder bias.

References

[1] Spisak, Tamas “A conditional permutation-based approach to test confounder effect

and center-bias in machine learning models”, in prep.

[2] Berrett, Thomas B., et al. “The conditional permutation test for independence while controlling for confounders.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 82.1 (2020): 175-197.

Examples

See notebooks/quickstart.ipynb for more detailed examples.

>>> partial_confound_test(y=[1,2,3,4,5,6], yhat=[1.5,2.3,2.9,4.2,5,5.7], c=[3,5,4,6,1,2], random_state=42, num_perms=100).p
1.0

Full confounder test

mlconfound.stats.full_confound_test(y, yhat, c, num_perms=1000, cat_y=False, cat_yhat=False, cat_c=False, mcmc_steps=50, cond_dist_method='gam', return_null_dist=False, random_state=None, progress=True, n_jobs=-1)[source]

The Full Confounder Test, to test for full confounder bias in machine learning models, based on the target variable, the confounder and the model predictions. Low p-value indicates that the model predictions are not fully driven by the confoudner.

Parameters
yarray_like

Target variable.

yhatarray_like

Predictions.

carray_like

Confounder variable.

num_permsint

Number of conditional permutations.

cat_ybool

Flag for categorical target variable (classification).

cat_yhatbool

Flag for categorical predictions (classification). Must be false, if class probabilities are used.

cat_cbool

Flag for categorical confounder variable (e.g center, sex or batch).

mcmc_stepsint

Number of steps for the Markov-chain Monte Carlo sampling.

cond_dist_methodstr

Method to estimate the y|c conditional distribution. Can be “linear” or “gam”. Recommended: “gam”.

return_null_distbool

Whether permuted null distribution should be returned (e.g. for plotting purposes).

random_stateint or None

Random state for the conditional permutation.

progressbool

Whether to print out progress bar.

n_jobsint

Number of parallel jobs (-1: use all available processors).

Returns
ResultsFullyConfounded

Named tuple with the fields:

  • “r2_y_c”: coefficient-of-determination between y and c,

  • “r2_y_yhat”: coefficient-of-determination between y and yhat,

  • “r2_yhat_c”: coefficient-of-determination between yhat and c,

  • “p”: p-value,

  • “p_ci”: binomial 95% confidence interval for the p-value,

  • “null_distribution”: numpy.ndarray containing the permuted null distribution or None depending on teh value of

    return_null_dist.

Notes

Performs the ‘full confounder test’, a statistical test described in [1]_, based based on the conditional permutation test for independence [2]_, using a linear or a general additive model (for numerical y, based on the parameter cond_dist_method) and a multinomial logit model (for categorical y, undependent on cond_dist_method) to estimate the y|c conditional distribution. This allows handling non-normal and non-linear dependencies between the target variable, the model output and the confounder variable. The null hypothesis of the test is that the model predictions are independent of the target variable, given the confounder variable, i.e. the model is entirely driven by the confounder. A low p-value therefore indicates that the model is not fully driven by the confounder.

References

[1] Spisak, Tamas “A conditional permutation-based approach to test confounder effect

and center-bias in machine learning models”, in prep.

[2] Berrett, Thomas B., et al. “The conditional permutation test for independence while controlling for confounders.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 82.1 (2020): 175-197.

Examples

See notebooks/quickstart.ipynb for more detailed examples.

>>> full_confound_test(y=[1,2,3,4,5,6], yhat=[1.5,2.3,2.9,4.2,5,5.7], c=[3,5,4,6,1,2], random_state=42, num_perms=100).p
1.0

Module ‘mlconfound.stats’

class mlconfound.stats.CptResults(r2_x_z, r2_x_y, r2_y_z, p, p_ci, null_distribution)
Attributes
null_distribution

Alias for field number 5

p

Alias for field number 3

p_ci

Alias for field number 4

r2_x_y

Alias for field number 1

r2_x_z

Alias for field number 0

r2_y_z

Alias for field number 2

Methods

count(value, /)

Return number of occurrences of value.

index(value[, start, stop])

Return first index of value.

property null_distribution

Alias for field number 5

property p

Alias for field number 3

property p_ci

Alias for field number 4

property r2_x_y

Alias for field number 1

property r2_x_z

Alias for field number 0

property r2_y_z

Alias for field number 2

class mlconfound.stats.ResultsFullyConfounded(r2_y_c, r2_y_yhat, r2_yhat_c, p, p_ci, null_distribution)
Attributes
null_distribution

Alias for field number 5

p

Alias for field number 3

p_ci

Alias for field number 4

r2_y_c

Alias for field number 0

r2_y_yhat

Alias for field number 1

r2_yhat_c

Alias for field number 2

Methods

count(value, /)

Return number of occurrences of value.

index(value[, start, stop])

Return first index of value.

property null_distribution

Alias for field number 5

property p

Alias for field number 3

property p_ci

Alias for field number 4

property r2_y_c

Alias for field number 0

property r2_y_yhat

Alias for field number 1

property r2_yhat_c

Alias for field number 2

class mlconfound.stats.ResultsPartiallyConfounded(r2_y_c, r2_yhat_c, r2_y_yhat, p, p_ci, null_distribution)
Attributes
null_distribution

Alias for field number 5

p

Alias for field number 3

p_ci

Alias for field number 4

r2_y_c

Alias for field number 0

r2_y_yhat

Alias for field number 2

r2_yhat_c

Alias for field number 1

Methods

count(value, /)

Return number of occurrences of value.

index(value[, start, stop])

Return first index of value.

property null_distribution

Alias for field number 5

property p

Alias for field number 3

property p_ci

Alias for field number 4

property r2_y_c

Alias for field number 0

property r2_y_yhat

Alias for field number 2

property r2_yhat_c

Alias for field number 1

mlconfound.stats.full_confound_test(y, yhat, c, num_perms=1000, cat_y=False, cat_yhat=False, cat_c=False, mcmc_steps=50, cond_dist_method='gam', return_null_dist=False, random_state=None, progress=True, n_jobs=-1)[source]

The Full Confounder Test, to test for full confounder bias in machine learning models, based on the target variable, the confounder and the model predictions. Low p-value indicates that the model predictions are not fully driven by the confoudner.

Parameters
yarray_like

Target variable.

yhatarray_like

Predictions.

carray_like

Confounder variable.

num_permsint

Number of conditional permutations.

cat_ybool

Flag for categorical target variable (classification).

cat_yhatbool

Flag for categorical predictions (classification). Must be false, if class probabilities are used.

cat_cbool

Flag for categorical confounder variable (e.g center, sex or batch).

mcmc_stepsint

Number of steps for the Markov-chain Monte Carlo sampling.

cond_dist_methodstr

Method to estimate the y|c conditional distribution. Can be “linear” or “gam”. Recommended: “gam”.

return_null_distbool

Whether permuted null distribution should be returned (e.g. for plotting purposes).

random_stateint or None

Random state for the conditional permutation.

progressbool

Whether to print out progress bar.

n_jobsint

Number of parallel jobs (-1: use all available processors).

Returns
ResultsFullyConfounded

Named tuple with the fields:

  • “r2_y_c”: coefficient-of-determination between y and c,

  • “r2_y_yhat”: coefficient-of-determination between y and yhat,

  • “r2_yhat_c”: coefficient-of-determination between yhat and c,

  • “p”: p-value,

  • “p_ci”: binomial 95% confidence interval for the p-value,

  • “null_distribution”: numpy.ndarray containing the permuted null distribution or None depending on teh value of

    return_null_dist.

Notes

Performs the ‘full confounder test’, a statistical test described in [1]_, based based on the conditional permutation test for independence [2]_, using a linear or a general additive model (for numerical y, based on the parameter cond_dist_method) and a multinomial logit model (for categorical y, undependent on cond_dist_method) to estimate the y|c conditional distribution. This allows handling non-normal and non-linear dependencies between the target variable, the model output and the confounder variable. The null hypothesis of the test is that the model predictions are independent of the target variable, given the confounder variable, i.e. the model is entirely driven by the confounder. A low p-value therefore indicates that the model is not fully driven by the confounder.

References

[1] Spisak, Tamas “A conditional permutation-based approach to test confounder effect

and center-bias in machine learning models”, in prep.

[2] Berrett, Thomas B., et al. “The conditional permutation test for independence while controlling for confounders.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 82.1 (2020): 175-197.

Examples

See notebooks/quickstart.ipynb for more detailed examples.

>>> full_confound_test(y=[1,2,3,4,5,6], yhat=[1.5,2.3,2.9,4.2,5,5.7], c=[3,5,4,6,1,2], random_state=42, num_perms=100).p
1.0
mlconfound.stats.generalization_test(y, yhat, c, num_perms=1000, cat_y=False, cat_yhat=False, cat_c=False, mcmc_steps=50, cond_dist_method='gam', return_null_dist=False, random_state=None, progress=True, n_jobs=-1)[source]

The Generalization Test is simply an alias for the Partial Confounder Test, to test if the machine learning model generalizes to a third, “positive validator” variable. Low p-value indicates that the model predictions explain more variance in the “positive validator” variable than expected from its correlation to the target, indicating that the model is driven by signal that is directly related to the positive validator.

Parameters
yarray_like

Target variable.

yhatarray_like

Predictions.

carray_like

Positive validator variable.

num_permsint

Number of conditional permutations.

cat_ybool

Flag for categorical target variable (classification).

cat_yhatbool

Flag for categorical predictions (classification). Must be false, if class probabilities are used.

cat_cbool

Flag for categorical confounder variable (e.g center, sex or batch).

mcmc_stepsint

Number of steps for the Markov-chain Monte Carlo sampling.

cond_dist_methodstr

Method to estimate the c|y conditional distribution. Can be “linear” or “gam”. Recommended: “gam”.

return_null_distbool

Whether permuted null distribution should be returned (e.g. for plotting purposes).

random_stateint or None

Random state for the conditional permutation.

progressbool

Whether to print out progress bar.

n_jobsint

Number of parallel jobs (-1: use all available processors).

Returns
ResultsPartiallyConfounded

Named tuple with the fields:

  • “r2_y_c”: coefficient-of-determination between y and c,

  • “r2_yhat_c”: coefficient-of-determination between yhat and c,

  • “r2_y_yhat”: coefficient-of-determination between y and yhat,

  • “p”: p-value,

  • “p_ci”: binomial 95% confidence interval for the p-value,

  • “null_distribution”: numpy.ndarray containing the permuted null distribution or None depending on teh value of

    return_null_dist.

Notes

Under the hood, the Generalization Test simply performs the ‘partial confounder test’, a statistical test described in [1]_, based on the conditional permutation test for independence [2]_, using a linear or a general additive model (for numerical y, based on the parameter cond_dist_method) and a multinomial logit model (for categorical c, undependent on cond_dist_method) to estimate the c|y conditional distribution. This allows handling non-normal and non-linear dependencies between the target variable, the model output and the confounder variable. The null hypothesis of the test is that the model predictions are independent of the positive validator variable, given the target variable, i.e. there is true generalization. A low p-value therefore indicates a significant generalization value.

References

[1] Spisak, Tamas “A conditional permutation-based approach to test confounder effect

and center-bias in machine learning models”, in prep.

[2] Berrett, Thomas B., et al. “The conditional permutation test for independence while controlling for confounders.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 82.1 (2020): 175-197.

Examples

See notebooks/quickstart.ipynb for more detailed examples.

>>> generalization_test(y=[1,2,3,4,5,6], yhat=[1.5,2.3,2.9,4.2,5,5.7], c=[3,5,4,6,1,2], random_state=42, num_perms=100).p
1.0
mlconfound.stats.partial_confound_test(y, yhat, c, num_perms=1000, cat_y=False, cat_yhat=False, cat_c=False, mcmc_steps=50, cond_dist_method='gam', return_null_dist=False, random_state=None, progress=True, n_jobs=-1)[source]

The Partial Confounder Test, to test partial confounder bias in machine learning models, based on the target variable, the confounder and the model predictions. Low p-value indicates that the model predictions are partially driven by the confoudner.

Parameters
yarray_like

Target variable.

yhatarray_like

Predictions.

carray_like

Confounder variable.

num_permsint

Number of conditional permutations.

cat_ybool

Flag for categorical target variable (classification).

cat_yhatbool

Flag for categorical predictions (classification). Must be false, if class probabilities are used.

cat_cbool

Flag for categorical confounder variable (e.g center, sex or batch).

mcmc_stepsint

Number of steps for the Markov-chain Monte Carlo sampling.

cond_dist_methodstr

Method to estimate the c|y conditional distribution. Can be “linear” or “gam”. Recommended: “gam”.

return_null_distbool

Whether permuted null distribution should be returned (e.g. for plotting purposes).

random_stateint or None

Random state for the conditional permutation.

progressbool

Whether to print out progress bar.

n_jobsint

Number of parallel jobs (-1: use all available processors).

Returns
ResultsPartiallyConfounded

Named tuple with the fields:

  • “r2_y_c”: coefficient-of-determination between y and c,

  • “r2_yhat_c”: coefficient-of-determination between yhat and c,

  • “r2_y_yhat”: coefficient-of-determination between y and yhat,

  • “p”: p-value,

  • “p_ci”: binomial 95% confidence interval for the p-value,

  • “null_distribution”: numpy.ndarray containing the permuted null distribution or None depending on teh value of

    return_null_dist.

Notes

Performs the ‘partial confounder test’, a statistical test described in [1]_, based on the conditional permutation test for independence [2]_, using a linear or a general additive model (for numerical y, based on the parameter cond_dist_method) and a multinomial logit model (for categorical c, undependent on cond_dist_method) to estimate the c|y conditional distribution. This allows handling non-normal and non-linear dependencies between the target variable, the model output and the confounder variable. The null hypothesis of the test is that the model predictions are independent of the confounder, given the target variable, i.e. there is no-confounder bias. A low p-value therefore indicates significant confounder bias.

References

[1] Spisak, Tamas “A conditional permutation-based approach to test confounder effect

and center-bias in machine learning models”, in prep.

[2] Berrett, Thomas B., et al. “The conditional permutation test for independence while controlling for confounders.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 82.1 (2020): 175-197.

Examples

See notebooks/quickstart.ipynb for more detailed examples.

>>> partial_confound_test(y=[1,2,3,4,5,6], yhat=[1.5,2.3,2.9,4.2,5,5.7], c=[3,5,4,6,1,2], random_state=42, num_perms=100).p
1.0

Module ‘mlconfound.plot’

mlconfound.plot.plot_graph(confound_test_results, y_name='y', yhat_name='<y&#770;>', c_name='c', outfile_base=None, precision=3)[source]

Plot confounder test results as a graph depicting the involved variables.

Parameters
confound_test_resultsnamedtuple

The object returned by test_partially_confounded or test_fully_confounded.

y_name: str

Name of the target variable.

yhat_name: str

Name for the model predictions.

c_name: str

Name of the confounder variable.

outfile_base: str

Path for output files, without extension (None: figure not saved).

precision: int

Precision for r squared values.

Returns
dot

The graphviz object to plot.

See also

plot_null_dist
mlconfound.plot.plot_null_dist(confound_test_results, **kwargs)[source]

Plot the histogram of the permutation-based null distribution of the confounder test.

Parameters
confound_test_results: namedtuple

The object returned by test_partially_confounded or test_fully_confounded.

kwargsdict

Additional named argumnets, passed to seaborn.histplot.

Returns
matplotlib.axes.Axes

The matplotlib axes containing the plot.

See also

plot_graph

Module ‘mlconfound.simulate’

mlconfound.simulate.identity(x)[source]

Identity transformation

Parameters
x
Returns

Notes

To be used as a transformation function in simulate_y_c_yhat.

mlconfound.simulate.simulate_y_c_yhat(w_yc, w_yyhat, w_cyhat, n, delta=1, epsilon=0, nonlin_trf_fun=<function identity>, bin_y=False, bin_c=False, bin_yhat=False, scale=True, random_state=None)[source]

Simulate normally distributed target (y), confounder (c) and predictions (yhat).

Parameters
w_yc: float

The weight of y in c.

w_yyhat: float

The weight of y in yhat.

w_cyhat: float

The weight of c in yhat. Set it to zero for H0.

n: int

Number of observations.

delta: float

The delta param of the sinh_archsin transformation on y’s contribution in yhat (only affects yhat).

epsilon: float

The epsilon param of the sinh_archsin transformation on y’s contribution in yhat (only affects yhat).

nonlin_trf_fun: callable

Callable to introduce non-linearity in the conditional distributions. (default: no non-linearity).

scale: bool

Scale variables to zero mean and unit variance.

random_state: int

Numpy random state.

Returns
tuple
  • y: the simulated target variable

  • c: the simulated confounder variable

  • yhat: the simulated predictions

See also

sinh_arcsinh

Notes

\[y \sim \mathcal{N}(0, 1)\]
\[c | y_i \sim \mathcal{N}(w_{y,c} f(y_i), 1)\]
\[\hat{y} | y_i, c_i \sim sinh(\delta sinh^{-1}( \mathcal{N}(w_{y,c} f(y_i), 1))-\epsilon)\]

Examples

>>> y, c, yhat = simulate_y_c_yhat(0.3, 0.2, 0.2, n=3, random_state=42)
>>> print(y, c, yhat)
[ 0.30471708 -1.03998411  0.7504512 ] [ 1.32193037 -1.09613765 -0.22579272] [ 1.03955979 -1.35013318  0.31057339]
mlconfound.simulate.sinh_arcsinh(x, delta=1, epsilon=0)[source]

Sinh-arcsinh transformation

Parameters
xarray_like

Normally distributed input data.

deltafloat

Parameter to control kurtosis, delta=1 means no change.

epsilonfloat

Parameter to control skewness, epsilon=0 means no change.

Returns
——-
array_like

Transformed data.

Notes

The sinh-arcsinh transformation of Jones and Pewsey [1]_ can be used to transfrom Normal distribution to non-normal.

References

[1] Jones, M. C. and Pewsey A. (2009). Sinh-arcsinh distributions. Biometrika 96: 761–780

Examples

See validation/simulation.py for an application example.

>>> result = sinh_arcsinh([-1, -0.5, -0.1, 0.1, 0.5, 1], delta=2, epsilon=1)
>>> print(result)
[-7.8900947  -3.48801839 -1.50886059 -0.88854985 -0.03758519  0.83888754]
>>> result = sinh_arcsinh([-1, -0.5, -0.1, 0.1, 0.5, 1], delta=1, epsilon=0)
>>> print(result)
[-1.  -0.5 -0.1  0.1  0.5  1. ]