In ‘Rawlsian algorithmic fairness and a missing aggregation property of the difference Principle’, the authors argue that there is a false assumption in algorithmic fairness interventions inspired by John Rawls’ theory of justice. Rawslian justice involves two principles – equal basic liberties and the difference principle – which should be upheld in and through the ‘basic structure’ of society, which includes the constitution, the legal system, the economy, etc. The difference principle says, roughly, that social and economic inequalities are permissible so long as they maximally benefit the least advantaged in society (this is also known as ‘maximin’, short for ‘maximising the minimum’). The authors cite various work in algorithmic fairness which operationalises the difference principle as a constraint on the outputs of a machine learning model.

The authors argue that applying the difference principle at the level of a local algorithmic decision-making context (what they term a ‘constituent situation’), is neither necessary nor sufficient for the difference principle to be upheld at the aggregate level of society at large.

That satisfying the difference principle at a local level is not necessary for satisfying it at the aggregate level seems obvious and intuitive. If one decision-making process (say, a firm making hiring decisions) fails to maximise benefits for the least advantaged, some other decision-making process (say, public healthcare allocation) might sufficiently compensate, such that the overall material position of the least advantage in society is better than they would have been in another set of social arrangements. Indeed, most societies already engage in various mechanisms designed to address social and economic inequalities, such as progressive taxation, the welfare state, and funding public goods.

But even if maximin at the local level isn’t strictly necessary for it to obtain at the societal level, one might still assume it would at least be sufficient. However, the authors provide a simple counterexample which convincingly demonstrates this is not the case. They then go on to simulate a range of probabilistic situations in which the same situation occurs, which show that the simple counterexample is not just an edge case but more likely a pervasive phenomenon. These findings suggest that even if every decision-making context (algorithmic or otherwise) throughout society were to implement the difference principle, this would not result in the difference principle being upheld at large.

I find these arguments compelling. They are in line with long-standing concerns about Rawlsian justice in general, and its application in ‘fair’ machine learning in particular. The difference principle is a creature of ideal theory; it is an articulation of what kinds of distribution of resources could be rationally justified according to Rawlsian theory; it is not a policy design mechanism to deploy in particular decision-making contexts. If we are trying to address inequality, starting from our present, non-ideal world, there are a whole range of measures we might try, from taxing billionaires to universal basic income. Tweaking parameters in machine learning models doesn’t seem like the obvious place to start.

Overall, this article is a very welcome contribution to the literature. In this commentary, I would like to briefly highlight some limitations of these arguments, and some of their broader implications for the field of political philosophy-inspired fair machine learning.

1 Only Some Algorithmic Fairness Proposals Seek to Implement the Difference Principle

First, it is worth noting that the majority of algorithmic fairness literature, even when arguably inspired by Rawls, has not been primarily concerned with implementing the difference principle. The major classes of group fairness measure aim merely to equalise outcomes, errors, calibration, etc., between protected groups. If we want to place these within a Rawlsian framework, they might be best understood as merely implementing the first principle – equal basic liberty – which, as the authors argue, does have the desired aggregation property. In which case, local-level algorithmic fairness is still a coherent endeavour under the Rawlsian paradigm. Even those few works which do seriously engage with formally implementing the difference principle – such as (Heidari et al., 2019) – would still be valuable in so far as they implement the first principle of equal basic liberty.

2 Machine Learning as Basic Structure?

The missing aggregation problem identified in the article arises because of the multiple local contexts in which ML is deployed, between which aggregation is expected to happen. But what if there were some ML systems which operate on so vast a scale, on such a large portion of society, that they have direct and substantial distributive impacts regardless of how they are aggregated with other decision-making contexts? In such cases, the gap between the local and societal level might be small enough that the aggregation problem fails to meaningfully arise. Imagine a single decision making algorithm deployed by the state to determine all the taxes levied and benefits allocated to every citizen. It seems plausible that applying the difference principle would have the capacity to approximate the ‘correct’ distribution at the societal level.

While no such algorithm exists, some models are deployed at a similarly national or global scale with significant distibutional impacts. In the UK in 2020, when exams could not be held due to Covid-19, the government proposed to automatically predict high school student grades using predictive algorithms. This would arguably have seriously impacted the life chances of the vast majority of teenagers in the country with substantial effects on the distribution of resources for those affected. Similarly, many modern AI systems – including search, classification, and recommendation – are deployed by big tech companies on billions of users. It is not inconceivable that they may also have substantial distributive implications.

Such examples suggest that algorithms could be part of the ‘basic structure’ of society. Under the traditional Rawlsian paradigm, the basic structure includes the constitution, legal system, and major institutions. Iason Gabriel draws on sociotechnical perspectives to argue that AI systems are also increasingly ‘part of the major sociotechnical practices that make up the basic structure of society’(Gabriel, 2022). As such, he argues, principles of distributive justice should apply. Creel & Hellman’s concept of the ‘Algorithmic Leviathon’ is also useful here (Creel & Hellman, 2022). They point to examples where the same algorithm is used by many actors, with the effect of systematically excluding certain people from spheres of society (e.g. certain job markets); they argue that such exclusion may be of moral concern, including potentially on Rawlsian grounds. If some algorithms are part of the basic structure, then the difference principle should equally apply to them as it does to the constitution, the legal system, and other institutions that are part of the basic structure.

But this line of thinking raises a more general question for any attempt to apply Rawlsian principles to real life (whether to algorithms or regular human contexts); what if the problem of the non-aggregating difference principle reappears when we consider how the various elements of the basic structure work (or don’t work) together? If the basic structure is made up of various constituent components (including the constitution, institutions, and perhaps sufficiently pervasive algorithms), should we apply the difference principle to each component, or evaluate bundles of components for their overall distributional effects?

I think such questions illustrate the limited nature of the difference principle as an artefact of ideal theory. It is an abstract principle, not a practical guide to designing just societies. The authors critique of fair machine learning seems to shed doubt on the practical utility of Rawlsian principles of justice more broadly; how do we know any given institutional design will actually shift real societal distributions towards the maximin ideal?

3 Intervening Elsewhere

Nevertheless, the authors do take their critique of fair ML to provide support for alternative policies which don’t involve fiddling with ML models but instead require us to ‘intervene elsewhere’. Quoting Hedden, they include examples like ‘reparations, criminal justice reforms, or changes in the tax code’ (Hedden, 2021).

This expanded view of egalitarian justice in machine learning would mean looking at broader patterns of (un)just distribution, instead of only considering the allocative potential of the algorithmic decision-making context. This does not necessarily mean that we should leave ML models untouched; as I have argued previously, considering wider patterns of injustice experienced by a disadvantaged group might ground claims to particular kinds to treatment in a particular decision making context (Binns, 2018, Sect. 3.3). But it does mean the standard suite of fair ML interventions should be considered as at most one small part of a wider approach to addressing injustice.

This illustrates the substantial gap between theories of just distribution from political philosophy, and their implications for real world interventions, including in the context of algorithmic fairness. The authors, and Hedden, are right to point to ‘intervening elsewhere’ as a more effective measure than tinkering with constraints on machine learning models.

But while intervening elsewhere might be more effective, whose job is it? And what interventions should be pursued? Such questions require a deeper consideration of moral responsibility for justice, and political strategy. But there is very little in Rawlsian theory which can meaningfully guide what such interventions could and should look like and who could make them. To illustrate the need to go beyond Rawlsian ideal theorising, the next two sections will sketch possible answers to these questions of responsibility and strategy. My aim is not to convince the reader of these particular answers, but rather to highlight the kinds of non-ideal theorising that an adequate response to unjust machine learning might require.

4 Responsibility for Justice

On the Rawlsian view, ultimate responsibility for justice lies with the state. It is assumed that the state can set up constitutional arrangements and other elements of the basic structure such as to bring about just distributions of resources. By contrast, other egalitarians believe justice to also apply to the domain of personal actions and decisions; people owe justice to each other, interpersonally (Cohen, 1989). On this view, we all have a duty to try to bring about justice through our own actions, not just by voting for governments that will do so via the ballot box, but through our personal lives: depending on one’s broader political outlook, this might include voluntary work, organising for social justice, mutual aid, trade unions, philanthropy, protest, revolution, etc.

Of course, placing all responsibility for justice on the backs of individual people seems too demanding. On the other hand, making justice the sole responsibility of the state is likely insufficient, especially if that state is captured by elite and business interests. And even those very powerful private actors operate within systems which significantly constrain their actions. Corporations operate exploitative sweatshops because they are locked into a globalised capitalist system in which they are legally obligated to maximise shareholder profits; changing practices would typically mean ceding profits to competitors. As a result, neither individuals, nor states, nor even powerful corporations seem able to meet what otherwise appear to be valid demands of justice.

In response to this dilemma, Iris Marion Young proposes a ‘social connections model’; on this view, we all bear responsibility in so far as we all causally contribute to structural processes that produce injustice (Young, 2006). We aren’t limited to considering responsibility for discrete events (e.g. a particular police shooting); we must consider background conditions (e.g. racism, capitalism, austerity, etc.). No one individual is powerful enough to transform those conditions, but collectively we may be able to. Robin Zheng builds on Young’s model, arguing that we are each responsible for structural injustice through and in virtue of our social roles, i.e. our roles as parents, colleagues, workers, citizens, etc., because roles are the site where structure meets agency (Zheng, 2018). The social roles you have in life determine what agency you have and therefore how you can change the system. The point here is not to let governments and corporations off the hook; rather, the social connection model acknowledges that the agents we might otherwise point to as being responsible for addressing injustice, ultimately derive their constitution from larger social orders that we all collectively reproduce – and could change.

So what does this Young / Zheng model of responsibility mean for the application of egalitarian principles to ML models?

First, we have to acknowledge the unjust, oppressive structures which operate around and within the context of algorithm deployment. They influence what models get deployed, where; they shape the distributions of data we feed into those models, and the social meanings of the labels we apply, including a label like ‘good employee’, and of protected characteristics like gender, race, religion, and so on, that we might use to assess fairness. Second, while these social structures are unjust, no one actor is capable of changing them. Even if a deployer of an ML model sincerely wanted to apply some intervention in their own context, in a way that might somehow meaningfully address structural injustice, they would be heavily curtailed by the coercive laws of competition in a capitalist economy. Third, different actors have different roles. A worker, a voter, a government regulator, a CEO, will all have different potential roles to play in reproducing or unmaking the oppressive structures surrounding a deployed algorithm.

Rawls provides a theory of what a just distribution would look like, but provides little detail on who is responsible for bringing it about in the real, non-ideal, world, where the state falls short. The Young / Zheng model provides an answer to that question of who bears responsibility for bringing justice about given structural forces and diffused agency. But the latter is also insufficient; it does not provide detail on what kind of interventions any actor ought to take. That is the realm of politics, not philosophy.

5 The Political ‘Gamespace’ of Algorithmic Fairness

Given such limitations of political philosophy, especially in its ideal form, to guide interventions, we may need to turn to a more strategic perspective on the politics of machine learning and egalitarian justice. To briefly sketch the contours of possible interventions, I draw on a framework proposed by the late Erik Olin Wright (Wright, 2015). Wright took a sociological approach to understand which agents in society have which kinds of power, and to consider in more detail the variety of political strategies that they might deploy at different levels.

Wright analogised the capitalist social system we live in as a kind of game, which can be interacted with on three levels. The situational level involves accepting the rules of the game and deciding what moves to make within it; interventions here include using existing laws, influencing policy via democratic engagement, and so on. At the institutional level, we accept the game we are playing, but not necessarily the current rules of the game. We can redesign institutions in quite substantial ways to give different players within the game more or less advantages, and change the moves available to them, to maximise the rewards they can get. The conflicts here are between reformers and reactionaries. At the systemic level, the very game we are playing is open to contestation. This is level at which Marxists like Wright see class struggle playing out, through an historical process whereby the internal contradictions within the current ‘game’ (capitalism) lead to its replacement by the working class with a different game (socialism).

While not all will agree with Wright’s exact distinctions, nor their Marxist underpinnings, they provide a picture of the different possible depths at which various interventions might operate in the context of algorithmic fairness.

First, many algorithmic fairness interventions appear to be at the situational level, concerned with the application of existing rules to algorithms. Sometimes compliance with existing rules might involve ceasing deployment of a system altogether; at other times, it may mean applying algorithmic fairness measures to, for instance, reduce unequal effects on protected groups. While inherently limited, the value of work at this level should not be understated; with many current uses of algorithms not even compliant with existing law, there is plenty of scope to challenge unjust algorithm deployment at this level.

Second, some algorithmic fairness interventions could be characterised as operating at the institutional level, where they involve new rules or institutional change. For instance, the creation of new laws, such as bans on facial recognition, or the incorporation of previously unconsidered values like dignity or autonomy in the context of algorithmic workplace management.

Third, we might consider what algorithmic ‘fairness’ might look like at the systemic level. It may be hard to imagine such interventions resembling mainstream algorithmic fairness measures; here, interventions are more likely to take the form of resisting and counteracting the existing power structures in which algorithms are embedded, rather than tinkering with their objective functions. Systemic level interventions may also be more speculative, imagining what role algorithmic decision making might play in alternative economic systems which organise production around human needs rather than the profit motive.

Some strategies cut across these different levels, and action at a lower level may help lay the groundwork or create alliances for work at higher levels.Footnote 1 For instance, we might be able to use the rules of the game to create structural changes; e.g. using data protection law in novel ways to challenge the deployment of AI in the workplace (situational level), thus preserving the capacity of organised labour to resist exploitation (institutional level) and build power to challenge the status quo (systemic level).

Finally, it may be that many of the injustices surfaced within algorithmic fairness work are best addressed through interventions which have very little to do with algorithms themselves, operating at deeper levels of societal structures than those at which algorithms currently operate (e.g. reparations, prison abolition, universal basic income, etc.). In such cases, algorithmic fairness researchers might concede that they have less to offer by way of technical research, while continuing to support such measures politically and materially in other ways.

But there are also many possible interventions which would benefit from the socio-technical perspective of algorithmic fairness researchers, beyond the traditional approach of formulating fairness as an optimisation goal. The point of sketching such possibilities is to expand the horizon of ‘algorithmic fairness’ to encompass the full gamut of possible interventions at different levels.

5.1 Stepping Out of Rawls’ Shadow

Given the missing aggregation property they identified, the authors seek to divert attention to the more promising strategy of ‘intervening elsewhere’; arguing this is ‘an important part of the Rawlsian toolbox’. I agree that interventions elsewhere seem more promising, but for the reasons given above, I am less confident that the Rawlsian toolbox will prove useful.

Rawlsianism has cast a long shadow in political philosophy (Forrester, 2019). By focusing only on ideal theory – what distributions of resources ought ideally to look like – it has failed to tackle the harder questions of how to get there. In the process, it has often been used to defend the (increasingly empty) promise of ‘trickle down’ economics, forestalling more radical forms of social justice, and has had little to say about historic and ongoing racial injustice (Shelby, 2003). There is a risk that this long shadow extends into our attempts to apply political philosophy to algorithmic fairness, if it is confined to formalising distributive principles as optimisation criteria. Thankfully, political philosophy is far broader than abstract distributive principles; hopefully, algorithmic fairness can be too.