Heuristics for evaluating performance
A lot of the thinking I do occurs in poorly defined domains – places where the feedback loops are fuzzy, long, and subjective; places where, by default, it's not clear how well a particular project is going.[1]
I think a lot of interesting and important work lies in places like this, so it's important to have methods for evaluating performance when operating in poorly defined domains.
To that end, here are some heuristics I've thought of for evaluating performance when working in fuzzy domains. These aren't battle-tested, so take them with a grain of salt.[2]
- Knowing what you know about how this project went, would you expect to be successful when undertaking a similar project in the future?
- Would other people expect you to be successful on a future, similar project if you explained the story of this project to them?
- Would you expect one of your peers to be successful on a similar project? If not, what missteps would you expect your peer make? Why would you expect to avoid these missteps when your peers didn't? (If the answer is that you are more capable than your peers in some relevant way, then you're not really comparing yourself to a peer group, just another group. If you think you don't have a peer group, you'd probably benefit from a change of scene.)
- Knowing what you know about this project, would you confidently take on a more difficult (i.e. larger, longer, more fuzzily defined, or more cognitively intense) project in the future? Would you expect to be successful at the harder project? If not, what missteps would you expect to make?
- Gut check: how do you instinctually feel about your performance on the project? Try to cash this out – if you feel bad about it, why do you feel bad? If you feel good about it, why?
- (When working in an organization with a hierarchical management structure) How much time did your work save your managers?
- Assess your work through two frames –
(1) Objective: How did your work product to compare to the work product that was required or aimed at?
(2) Subjective: How did your work product compare to your expectation of how you would do on this project? (Or, if in an organization) How did your work product compare to what is expected of a person at your capability level? - Can you identify common, thematic strengths in your work? What are they? Similarly, can you identify common, thematic weaknesses in your work? What are they?
- If you can identify thematic weaknesses in your work, what could you do to address these weaknesses during future projects? If you can identify thematic strengths, what can be done to maintain or amplify these strengths on future projects?
[1]: Writing a philosophical essay, doing historical research, and assigning subjective probability judgements all occur in poorly defined domains – a sound philosophical argument is not obviously more valid than a fallacious one, an incomplete historical narrative is not obviously less compelling than a more complete narrative (especially when the incompleteness is unknown), and an incorrect assessment of an event's probability can feel about the same as an accurate assessment.
In contrast, domains like motorcycle riding, programming, and rhetoric are well-defined – success is clearly differentiated from failure and the difference is immediately communicated to the agent. In well defined domains, expending effort to nail down accurate performance evaluations is less important because so much high-quality feedback is intrinsically available.
[2]: Take battled-tested business advice with a grain of salt too – anecdotes of business strategy obscure the role that luck played in the success story you're hearing; see Thinking Fast and Slow p. 205-208
[rereads: 3, edits: formatting and grammar tweaks, "most" –> "a lot of", fixed Kahneman page numbers]