AGI and the tyranny of the majority (presentation)

Abstract: AI company CEOs variously predict the achievement of artificial general intelligence in the next 2 to 10 years. Soon thereafter, an enormous proportion of worldly decisions will be made by AI systems. How should those decisions be made? A widely endorsed view is that they should be made democratically. This leads to the further question which democratic mechanism should be used to elicit and aggregate group judgments. This talk defends a democratic account of AI alignment whose starting point is John Nash’s influential account of bargaining theory. On this account AI delegates negotiate on behalf of users in a fair marketplace of values, arriving at the Nash product of negotiators’ value systems. The resulting scheme of governance avoids the tyranny of the majority, leading to negotiators getting much more of what they want overall when compared to alternative schemes. Partly in light of this, it also better satisfies the normative ideals of democratic theory including freedom and equality, and leads to a lower likelihood of a specific kind of existential catastrophe whereby the world loses nearly everything that humans value. I conclude by showing how the account can make progress on intractable questions such as AI consciousness and value lock-in and by reflecting on open questions about and objections to the account. The resulting solution is a promising normative target for high-stakes alignment decisions and illuminates the promise of egalitarian, market-inspired approaches to normative alignment decisions.