Jul 26, 2015

Research methodology for a pseudostatistical analysis of Understanding Power

As I mentioned previously, I'm interested in digging into some of the claims Chomsky makes in Understanding Power.

The book is lengthy (401 pages) and the footnote appendix lengthier (449 pages!). Researching each claim would be an enormous project (read: a project I would never finish). Some sampling is in order.

Sample size

Out of 756 footnotes, I decided a random sample of 20 would be sufficiently powered for my purposes. I didn't think too hard about what "power" means for a study like this; there is a clear trade-off between the confidence I have in my conclusion and the work I'm willing to put into this. Checking one or two footnotes would not be thorough enough to convince me of anything, checking 200 is beyond the limits of my patience.

Sampling procedure

On my first crack at this a couple of months ago, I just started flipping through the main text of UP and then read the footnotes for claims that caught my interest.

This method is far too susceptible to bias – something more rigorous is in order. (Pseudo)randomly selecting 20 chapter numbers and footnote numbers should do the trick.

This excel file contains the methodology I used. Essentially, I applied a RAND function on the range 1-10 (to select the chapter), then applied another RAND function on the range of the number of footnotes in the randomly selected chapter. This procedure is sort of ad hoc (I had to manually code each footnote-selecting RAND function), and I am sure there is a more elegant way to program this, but I think it preserves good randomization on both levels.

Here are the results of that procedure:

  • Chapter 1, Footnote 23
  • Chapter 1, Footnote 36
  • Chapter 2, Footnote 10
  • Chapter 3, Footnote 22
  • Chapter 3, Footnote 35
  • Chapter 4, Footnote 73
  • Chapter 4, Footnote 75
  • Chapter 5, Footnote 57
  • Chapter 5, Footnote 96
  • Chapter 6, Footnote 8
  • Chapter 6, Footnote 9
  • Chapter 6, Footnote 10
  • Chapter 6, Footnote 38
  • Chapter 8, Footnote 10
  • Chapter 8, Footnote 59
  • Chapter 8, Footnote 65
  • Chapter 9, Footnote 18
  • Chapter 9, Footnote 22
  • Chapter 9, Footnote 27
  • Chapter 10, Footnote 86

Research Questions

I'm interested in two research questions:

  1. Do the citations in each footnote support the factual claims Chomsky makes in the text?
  2. Do the factual claims Chomsky makes support his broader conceptual point?

#2 is the squishier question – at best, the factual claims will provide anecdotal support for Chomsky's theoretical claims, and alternative theories will likely fit with the facts to some degree. However, if Chomsky's theories consistently line up with the facts across a random sample (perhaps not always the "best-fit" theory, but at least always a "competitive-fit" theory), I will be persuaded that there is something to his position. This feels very subjective and dependent on my prior beliefs going in, but I don't think further procedural rigor will rescue us from that.

Keeping my methodology and reasoning transparent will provide some guard against this subjectivity.

Research Procedure

For question #1, I'm planning to:

  • Read the footnote and the related in-text passage in UP.
  • Track down each citation at its given source, and read 1-2 pages around the cited passage for context.
  • Search the cited topic on google, google scholar, JSTOR, and the New York Times archive. From this search, find 2-3 associated sources and evaluate whether they confirm or disfirm the cited passage.
  • Write up a brief summary of this search and my conclusion about the factual claim.

For question #2, I'm planning to:

  • Read roughly 2-5 pages of the UP text around the footnote.
  • Write up a summary of Chomsky's conceptual point to make sure I understand it.
  • Try to brainstorm alternative theories that line up with Chomsky's point. Possibly dig into some outside sources here (this is probably a rabbit hole, so I'll probably put a time cap on it).
  • Consider a counterfactual position (i.e., how strong is Chomsky's claim X if cited event Y never happened? Does claim X heavily depend on the evidence, or is claim X detached from the evidence such that it remains plausible for any imagined history?).
  • Write up my conclusion about the relationship between Chomsky's claim and the evidence he cites.

Ideally, I'll do this in chunks of 5 footnotes apiece. Some of the footnotes are pretty hairy, so if I've selected one of those, I might just handle that one its own.

I don't have a timeline for conducting this research or publishing the results. If no progress has been made within a year, I would be sad.

Preregistration of prior beliefs

In the vein of subjective squishiness, it seems appropriate to state my current understanding of UP before diving in.

I am basically sympathetic to Chomsky's thesis, which I parse as:

Modern America is an imperial power that is not notably better than previous empires. America conducts military and economic interventions abroad with the purpose of maintaining its position, often to the detriment of other countries and their people. Domestically, the corporate power-elite conspires with the government to maintain their socio-economic status and decision-making ability.

The American public has a distorted concept of America, and consider their country and system of government to be vaguely "better" than other countries. This conception is factually incorrect, but the media and higher education system reinforce and perpetuate it.

Although this situation is discouraging, it has been slowly improving over time, which can be demonstrated by the limiting of the power-elite's explicit means of control (large police actions, imprisonment and blackballing of opposition) and its recourse to secretive means of control (mass media dispersion of propaganda, widespread government surveillance, accessible provision of numbing entertainments). This shift is cause for optimism about the future.

My parse is pretty rough, but I think the basic thrust is correct. His position rings true for some reason I can't put my finger on.

I believe that I am open to changing my view about the validity of Chomsky's thesis after digging into some of the evidence he cites, though I might be fooling myself here. I am also open to the evidence-digging reinforcing my current view, or not moving me at all.

[rereads: 3, edits: rephrased "political propaganda" clause, "cheap" –>"accessible", "own" –> "one"]