Thursday, September 16, 2010

Do critics or viewers better predict how good a movie is?

One of my friends said he likes to look up the Rotten Tomatoes rating for a movie to decide whether to see it. I suggested he look at the Flixster ratings instead because they'll tend to better predict whether he'll like the movie.

This is something we can test empirically, as long as there is a standard for what "like the movie" means. A sensible definition is that people like a movie in proportion of the average rating of the movie on Netflix.*

So I gathered data on the ten or eleven most popular movies over the past three years and regressed the Netflix ratings on both the Rottan Tomatoes ratings and the Flixster ratings. The results (in technical terms):

(1) Netflix = 0.006*RT + 3.47
R^2 = 0.3436

(2) Netflix = 0.0191*Flixster + 2.39
R^2 = 0.6893

Both predict the Netflix ratings decently, but the Flixster rating is easily the better predictor. It can explain about 69% of the spread (variance) in the Netflix scores compared with less than 35% for the RT ratings.

Here's a nice graph:

The conclusion is that you should never use RT when deciding whether to watch an old movie. You can just use Flixster or Netflix. On the other hand, when new movies are released the Flixster ratings are probably biased because only the bigger fans see movies on opening weekend. You'll have to wait a few weeks before you can get an accurate judgment from Flixster. So maybe Rotten Tomatoes has a use for very new releases.

* - Technical appendix: There are some biases to the Netflix ratings--Netflix customers aren't the "typical American" because they're more likely to be younger and more educated. But it's the only good available data, and with millions of ratings shouldn't be too biased. The other problem is that movies with a small number of ratings have a selection bias: only people who expect to like the movie will watch it so all the ratings are probably slightly inflated compared to what they would be if a random sample of people were forced to watch and rate the movies. But that isn't such a big problem since, if you're considering watching the movie, you're implicitly in the set (or close to being in the set) of people who would watch the given movie.

No comments:

Post a Comment