Habryka on philosopher kings
From Oliver Habryka's shortform feed:
When I started studying rationality and philosophy, I had the perspective that people who were in positions of power and influence should primarily focus on how to make good decisions in general and that we should generally give power to people who have demonstrated a good track record of general rationality. I also thought of power as this mostly unconstrained resource, similar to having money in your bank account, and that we should make sure to primarily allocate power to the people who are good at thinking and making decisions.
That picture has changed a lot over the years. While I think there is still a lot of value in the idea of "philosopher kings," I've made a variety of updates that significantly changed my relationship to allocating power in this way:
I have come to believe that people's ability to come to correct opinions about important questions is in large part a result of whether their social and monetary incentives reward them when they have accurate models in a specific domain. This means a person can have extremely good opinions in one domain of reality, because they are subject to good incentives, while having highly inaccurate models in a large variety of other domains in which their incentives are not well optimized.
...
The point about rationality being domain-specific, such that it responds to social & monetary incentives, seems very right. See Sinclair: "It is difficult to get a man to understand something, when his salary depends upon his not understanding it!"