Coordination Forum

In game theory, assumptions of rationality imply that any "solution" of a game must be an equilibrium.* However, most games have many equilibria, and realistic agents don't always know which equilibrium they are in. Certain equilibrium strategies, such as tit-for-tat in iterated prisoner's dilemma, can also be seen in this broader context as coordination strategies: adopting them teaches others to adopt them, because you punish anyone playing some other strategy. In a narrow sense, these strategies solve both the game itself and the equilibrium selection problem. (Technically, such strategies are the evolutionarily stable ones.)

I want to make an informal point about two very different ways this can work out in real life: coordination strategies in which it feels like everyone is fighting to pull the system in different directions but it all cancels out, vs situations where it feels like the coordination strategy is your friend because it saves everyone effort. I believe the second case exists, but it is rather puzzling in terms of the existing literature.

*Different rationality assumptions give different equilibrium concepts; Nash equilibria are the most popular. Correlated equilibria are the second most popular and somewhat more relevant to the discussion here, but I won't get into enough technical details for it to matter.

This post made possible by discussions with Steve Rayhawk, Harmanas Chopra, Jennifer RM, Anna Salamon, and Andrew Critch. Added some edits proposed by Elo.

Schelling Negotiations

Schelling discussed agents solving the equilibrium selection problem by choosing points which other agents are most likely to choose based on prominence, and the term Schelling point was coined to describe such likely equilibria. The classic examples revolve around agents who cannot communicate with one another (highlighting the need for guesswork about each other's behavior), but adding the ability to communicate does not eliminate the equilibrium-selection problem; our community tends to use the term 'Schelling fence' to refer to the analogous concept when open negotiation is involved -- though my impression is that economic literature uses Schelling point for this case as well.

In A Positive Account of Property Rights, David Friedman** explains the emergence of property rights through the negotiation of Schelling fences. Negotiators want as many resources as possible, but also want to minimize the costs of conflict. If there were nothing special about any one patch of land, then both sides could always demand a little more land -- there's no good stopping-point for the bargaining. However, natural divisions in the land such as rivers can serve as Schelling fences. Once such a solution has been proposed, neither side wants to demand a little more for themselves, because breaking the Schelling fence opens up the door for the other person to do the same.

Even if the territory is blank, Schelling fences can be made with abstract reasoning: a half-half split, for example, is simpler than other options.

The coordination problem does re-assert itself at a higher level: gerrymandering conceptual categories. Is the river or the rocky ridge the more natural dividing line? What really counts as the "border" of the rocky ridge, exactly? Participants will tend to engage in motivated cognition to justify whichever potential Schelling fence is a better fit for them.

The side-taking hypothesis of morality, discussed in The Side-Taking Hypothesis for Moral Judgement by Peter DeScioli and A Solution to the Mysteries of Morality by DeScioli and Kurzban, gives a similar account of where our moral intuitions come from. This time, rather than just thinking about two people negotiating a conflict about property rights, we think about the bystanders. Bystanders may get caught up in a conflict, as the two contestants call on their allies for support. People may have their own interests in the outcome, but they also prefer to end up on the winning side rather than the losing side. So, everyone is trying to predict which side others will take, in order to choose sides. This creates an equilibrium-selection situation, so simple rules about right and wrong can dominate complicated social calculations. (As before, complex social considerations come back due to the possibility of gerrymandering the concepts which make up the Schelling points.)

Ziz's blog has good discussions of social order and justice in a Schelling-type framework, and coins the term Schelling reach to quantify a population's coordination power (how complex can one's reasoning be and still settle on the same Schelling point as others?). We can also understand language as an equilibrium-selection game: when you try to say something, you have to balance the various plausible interpretations which your audience might place on the various options of language at your disposal. People will gerrymander word meanings to make their preferred arguments more compelling. Ziz discusses consequences of this in DRM'd Ontology.

Willpower as Self-Coordination

Now let's relate this to arguments you have within yourself, via Ainslie's explanation of willpower in Breakdown of Will. Ainslie gives evidence that humans have systematically different preferences at different points in time. Moods and drives make different things desirable. Easy pleasure gets more tempting as it gets nearer. You set an alarm at night thinking it would be good to get up early and get more done, yet you prefer to shut it off and sleep in in the morning.

Ainslie suggests that we view this as a negotiation between instances of you across time. He calls these "interests" to avoid constantly sounding like he's talking about multiple personality disorder. Ainslie's definition of willpower is successful coordination between these interests via bright-line rules. Each interest knows that if it breaks the rules, it breaks your ability to coordinate with yourself; although each interest has somewhat different goals, the threat of global inability to coordinate is great enough to balance against almost any temptation.

This is why willpower feels sort of like a top-down imposition: some interests are blackmailing other interests with the threat of global discoordination, to make yourself do something which you don't want to do right now but which fits with your concept of "what you want to do".

One problem with this is that you have to use simple rules. Why? It's a lot like the Schelling negotiations discussed earlier. The interests have to coordinate on an equilibrium, which means it must be simple enough that there aren't plausible alternatives to bargain for. (Although, interests may try to bend even the simplest rules. There's a special Schelling-art to calculating excuses. "This is a special occasion, I can break my diet just this once!")

Anna Salamon has described these systems of rules as ego structures rather than willpower (private communication). I really like thinking of it this way, and had written a post draft about it, but it wasn't written very well so I may or may not post it.

A second problem is that in order to threaten yourself with a global coordination failure, you have to be willing to follow through. This isn't such a problem with humans, because habit-building works this way already: it isn't very plausible that you'll be able to build healthy habits in the future if you keep breaking them in the present. However, greater threats provide greater incentives. This makes some people engage in more directly self-punishing behavior if they don't live up to their own standards, such as mentally beating themselves up and feeling awful about things, depriving themselves of other pleasures, etc.

Two Coordination Styles

Ziz strongly advises against Ainslie's willpower strategy, calling it self-blackmail. Nate Soares seems to do the same in the replacing guilt sequence. Most of Ziz's blog is about an alternative technique called fusion. (I don't recommend just reading that link; read Ziz from the beginning. The posts before the fusion posts are prerequisites.) Nate similarly spends most of his blog explaining how to do better. Both of them use self-coordination strategies which have aspects of Ainslie's approach: they view themselves as made up of sub-agents, and explicitly think about the coordination of those sub-agents. However, the flavor is much more like building up trust between sub-agents rather than blackmail. Other people in the bay area rationality community also seem to advocate similar approaches, particularly Andrew Critch (in in-person conversations). It seems like the people who do this have more Getting Stuff Done power. But how could this work? Schelling's framework seems rather compelling. Is there some way around it?

So, I've finally got to the point I promised to make at the beginning: it seems like there are two different sorts of coordination strategies. I don't have any better terminology lined up, so as per Schelling-nature, I'll borrow Ziz's: treaties vs fusion.

Treaties: Ainslie's, Friedman's, and DeScioli & Kurzban's coordination mechanism involves bright-line rules, guilt vs innocence, retributional justice, and inefficient compromises (though Friedman argues in Law's Order that approximate efficiency is often achieved). Norms are established by historical precedence. Gerrymandering and casuistry abound.
Fusion: Nate's, Critch's, and Ziz's coordination style involves totality of circumstances (the legal term for the opposite of bright-line rules -- careful consideration of the circumstance). Their parts cooperate more fully, rather than spending a lot of the time fighting against each other. Decisions look closer to utility calculations rather than historical precedence. Action comes from intrinsic motivation rather than self-imposed structure.

How is this possible? What are they doing differently? If I buy Ainslie's psychological model even approximately, this seems rather difficult to explain.

Here are several ideas:

A legalistic system with punitive justice is more relevant to a society made up of a large number of agents who come and go, and so are only able to build trust to a limited degree. All the parts of a human live in the one head, so it is possible, though by no means universal, that a human can come to trust themselves.
People play tit-for-tat with other people in lots of ways, tracking how much they owe and are owed in dollars and favors, social status and personal trust. However, while this accounting facilitates an efficient and fair society, tracking every parties' balance can itself be costly. With certain people, there's a kind of mutual wink that happens, and you implicitly or explicitly agree to drop the whole accounting structure and just optimize the joint utility function. When you do that, new things become possible; you no longer have to make sure that each person gets their due in the end, which is actually better for both of you in expected value (potentially much better). This is described well, in my estimation, by true friendship is being counterfactually hugged by vampires on Compass Rose.
Ainslie's model recommends that we make a very important conceptual shift. Don’t just view actions as isolated. See them as instances of the class of similar actions (as in my instance of class post). Why? If you see your action as a part of a pattern, you'll consider its consequences on the coordination game. Only then can self-blackmail work. This is similar to CDT vs EDT in twin prisoner's dilemma: CDT thinks as if it can change its action in isolation without changing the probability of its copy cooperating, whereas EDT thinks of its action as correlated to its other self. In the same way, you have to recognize that your actions now are correlated with your actions later. However, perhaps Ainslie is missing another level of this kind of thinking: you know that if you gerrymander a rule to serve your narrow interests right now, your selves across time will similarly gerrymander. If you stop gerrymandering, you can apply totality-of-circumstances rather than bright-line thinking; it is more easily manipulated, but you have agreed with all your selves to stop trying to manipulate things. The very realization that you can achieve fusion rather than treaty, and that fusion is much better than treaty, is enough to incentivise all your parts to coordinate in this way.

**David Friedman is an expert in the economic analysis of law, and draws striking relationships between what rules are economically efficient, what's intuitively just, and what's used in practice. His book Law's Order contains more in this direction.