Confidence Intervals

When you do a study on how something is causing something else you generally report two findings, whether your evidence is precise enough to detect if an effect exists and whether the effect is important (this part of subjective).

The second one is the important one but because it is subjective people prefer to put the emphasis on the "objective" test for statistical significance. Objective is a relative term here because, in true, personal biases can have an enormous impact on the result of the test and, more importantly, its interpretation.

Consider "The Effect of File Sharing on Record Sales An Empirical Analysis" a really terrible written paper by a professor at Harvard Business School and a professor at the University of North Carolina. The question they are interested in is whether (and how much) file sharing of music reduces CD sales for music.

Basic economic theory says that being able to get music "for free" should reduce sales of music that costs money. The baseline model would ignore transportation costs and whether consumers care think about the ethics of piracy and just focus on costs. In 2004 songs on CD costs on average $1.5 per song while storage for a 3 MB .mp3 file cost about $0.002 and the time to search for the MP3 (say 20 seconds) is worth about 5 cents . In other words downloaded songs cost about 1/20 as much as music purchased on CD.

If we estimate the elasticity of demand for music and the number of CDs the average person buys we can extrapolate how many songs the average downloader would download. Lets just use 1 for the elasticity and 2 for the number of CDs per year so that the average college student downloads 400 songs per year.

Since it is assumed downloaders do not per for any music the lost sales are about 20 songs and the lost sales per download is 20/400 = 0.05

In the paper the authors find have very imprecise estimates so they find sales increased by up to about .5 units per download or decreased by up to about .5 songs per download. Since one extreme would imply that sales jumped by a factor of over 100x and the other implies negative music sales, all else equal, it is safe to rule both out. Whether the impact is -0.05 as theory predict, -0.027 as they estimate, or 0 as they assume for no apparent reason, not much can be said. Nevertheless they told plenty of newspapers that the effect was 0.