We were talking to someone in the market research field about the credibility of Ranker’s aggregated rankings, and they were intruiged and suggested that we validate our data by comparing the aggregated results of one of our lists to the results achieved by a traditional research company using traditional market research methodologies. Companies like Gallup often do not survey the same types of questions that we ask at Ranker, in part due to the inherent difficulties of open ended polling via random digit dialing. You can’t realistically call someone up at dinner time and ask them to list their 50 favorite TV shows. You could ask them to name one favorite, but doing that, you can end up with headlines like “Americans admire Glenn Beck more than they admire the Pope.” However, one question that both Gallup and Ranker have asked concerns the nation’s top colleges/universities. How do Ranker’s results compare to Gallup’s data? Below are our results, side by side.
From a market researcher’s perspective, this is good news for Ranker data. Our algorithms have successfully replicated the top 4 results from the Gallup poll exactly, at a fraction of the cost. This likely occurs because Ranker data is largely collected from users who find our website via organic search, so while our data is not a representative probability sample (assuming such a thing still exists in a world where people screen their calls on cellphones), our users tend to be more representative than the motivated Yelp user or the intellectual Quora user. If you compare how representative Ranker’s best movies list is compared to Rotten Tomatoes aggregated opinion list (Toy Story 2 and Man on Wire are #1 & #2!?!?), you get a sense of the importance of having relatively representative data.
In addition, the fact that our lists are derived from a combination of methodologies (listing, reranking, + voting), means that the error associated with each method somewhat cancels out. Indeed, one might argue that Ranker’s top dream colleges list is better than Gallup’s for precisely this reason as individuals are often tempted to list their alma mater or their local school as the best college, and the long tail of answers might actually contain more pertinent information. Aggregating ranked lists from motivated users and combining that data with casual voters might actually be the best way to answer a question like this.
– Ravi Iyer