7 - ‘All Data Is Credit Data’: Personalised Consumer Credit Score and Anti-Discrimination Law

Summary

Credit-score models provide one of the many contexts through which the big data micro-segmentation or ‘personalisation’ phenomenon can be analysed and critiqued. This chapter approaches the issue through the lens of anti-discrimination law, and in particular the concept of indirect discrimination. The argument presented is that, despite its initial promise based on its focus on impact, ‘indirect discrimination’ is after all unlikely to deliver a mechanism to intervene and curb the excesses of the personalised service model. The reason for its failure does not lie in its inherent weaknesses but rather in the 'shortcomings' (entrenched biases) of empirical reality itself which any 'accurate' (or useful) statistical analysis cannot but reflect. Still, the anti-discrimination context offers insights that are valuable beyond its own disciplinary boundaries. For example, the opportunities for oversight and review based on correlations within outputs rather than analysis of inputs is fundamentally at odds with the current trend that demands greater transparency of AI but may after all be more practical and realistic considering the ‘natural’ opacity of learning algorithms and businesses’ ‘natural’ secrecy. The credit risk score context also provides a low-key yet powerful illustration of the oppressive potential of a world in which individual behaviour from ANY sphere or domain may be used for ANY purpose; where a bank, insurance company, employer, health care provider, or indeed any government authority can tap into our social DNA to pre-judge us, should it be considered appropriate and necessary for their manifold objectives.