by    in Rankings

Rankings are the Future of Mobile Search

Did you know that Ranker is one of the top 100 web destinations for mobile per Quantcast, ahead of household names like The Onion and People magazine?  We are ranked #520 in the non-mobile world.  Why do we do better with mobile users as opposed to people using a desktop computer?  I’ve made this argument for awhile, but I’m hardly an authority, so I was heartened to see Google making a similar argument.

This embrace of mobile computing impacts search behavior in a number of important ways.

First, it makes the process of refining search queries much more tiresome. …While refining queries is never a great user experience, on a mobile device (and particularly on a mobile phone) it is especially onerous.  This has provided the search engines with a compelling incentive to ensure that the right search results are delivered to users on the first go, freeing them of laborious refinements.

Second, the process of navigating to web pages (is) a royal pain on a hand-held mobile device.

This situation provides a compelling incentive for the search engines to circumvent additional web page visits altogether, and instead present answers to queries – especially straightforward informational queries – directly in the search results.  While many in the search marketing field have suggested that the search engines have increasingly introduced direct answers in the search results to rob publishers of clicks, there’s more than a trivial case to be made that this is in the best interest of mobile users.  Is it really a good thing to compel an iPhone user to browse to a web page – which may or may not be optimized for mobile – and wait for it to load in order to learn the height of the Eiffel Tower?

As a result, if you ask your mobile phone for the height of a famous building (Taipei 101 in the below case), it doesn’t direct you to a web page.  Instead it answers the question itself.

That’s great for a question that has a single answer, but an increasing number of searches are not for objective facts with a single answer, but rather for subjective opinions where a ranked list is the best result.  Consider the below chart showing the increase in searches for the term “best”.  A similar pattern can be found for most any adjective.

So if consumers are increasingly doing searches on mobile phones, requiring a concise list of potential answers to questions with more than one answer, they naturally are going to end up at sites which have ranked lists…like Ranker. As such, a lot of Ranker’s future growth is likely to parallel the growth of mobile and the growth of searches for opinion based questions.

– Ravi Iyer

Why We Still Play Board Games: An Opinion Graph Analysis

It’s hard reading studies about people my age when research scientists haven’t agreed upon a term for us yet. In one study I’m a member of “Gen Y” (lazy), in another I’m from the “iGeneration” (Orwellian), or worse still, a “Millennial” (…no). You beleaguered and cynical 30-somethings had things easy with the “Generation X” thing. Let the record reflect that no one from my generation is even remotely okay with any of these terms. Furthermore, we all collectively check out whenever we hear the term “aughties”.

I’m whining about the nomenclature only because there’s a clear need for distinction between my generation and those who have/will come before/after us. This isn’t just from a cultural standpoint (although calling us “Generation Spongebob” might be the most ubiquitous touchstone you could get), but from a technical one. If this Kaiser Family Foundation study is to be believed (via NYT), 8-18 year olds today are the first to spend the majority of their waking hours interacting with the internet.

Yet despite this monumental change, there are still many childhood staples that have not been forsaken by an increasingly digital generation. One of the most compelling examples of this anomaly lies in board games. In a day and age where Apple is selling two billion apps a month (Apple), companies peddling games for our increasingly elusive away-from-keyboard time are still holding their own. For example, Hasbro’s board-and-card game based revenue grew to $1.19b dollars over the course of the last fiscal year (a 2% gain from last year).

What drove this growth? Hasbro’s earnings reports primarily accredits this growth to three products: Magic: The Gathering, Twister, and Battleship. All of these products have been mainstays of their line-up for quite some time (prepare to feel old: if Magic: The Gathering was a child, it could buy booze this year), so what’s compelling people to keep buying? Fortunately, Ranker has some pretty in-depth data on all of these products, based on people who vote on it’s best board games list, which receives thousands of opinions each month, as well as voting on other Ranker lists.

Twister’s continuous sales were the easiest to explain: users who expressed interest in the game were most likely to be a fan of other board games (Candy Land, Chutes and Ladders, Monopoly and so forth). Twister also correlated with many other programs/products with fairly universal appeal (Friends, Gremlins). This would seem to indicate that the chief reason for Twister’s continued high sales lies in its simplicity and ubiquity. The game is a cultural touchstone for that reason: more than any other game on the list, it’s the one hardest to picture a childhood without.

Battleship’s success lies in the same roots: our data shows great overlap between fans of the game and fans of Mouse Trap, Monopoly, etc. But Battleship has attracted fans of a different stripe, interest in films such as Doom, Independence Day, and Terminator were highly correlated with the game. In all likelihood, this is due to the recent silver-screen adaptation of the game. Although the movie only faired modestly within the United States, the film clearly did propel the game back into the public consciousness, which translated nicely into sales.

Finally, Magic: The Gathering’s success came from support of another nature. Interest in Magic correlated primarily with other role-play and strategy games (Settlers of Catan, Dominion, Heroscape). Simply put, most fans of Magic are likely to enjoy other traditionally “nerdy” games. The large correlation overlap between Magic and other role-playing games is a testament to how voraciously this group consumes these products.

The crowd-sourced information we have here neatly divides the consumers of each game into three pools. With this sort of individualized knowledge, targeting and marketing to each archetype of consumer is a far easier task.

– Eamon Levesque

by    in Data Science, prediction

Recent Celebrity Deaths as Predicted by the Wisdom of Ranker Crowds

At the end of each year, there are usually media stories that compile lists of famous people who have passed away. These lists usually cause us to pause and reflect. Lists like Celebrity Death Pool 2013 on Ranker, however, give us an opportunity to make (macabre) predictions about recent celebrity deaths.

We were interested in whether “wisdom of the crowd” methods could be applied to aggregate the individual predictions. The wisdom of the crowd is about making more complete and more accurate predictions, and both completeness and accuracy seem relevant here. Being complete means building an aggregate list that identifies as many celebrity deaths as possible. Being accurate means, in a list where only some predictions are borne out, placing those who do die near the top of the list.

Our Ranker data involved the lists provided by a total of 27 users up until early in 2013. (Some them were done after at least one celebrity, Patti Page, had passed away, but we thought they still provided useful predictions about other celebrities). Some users predicted as many as 25 deaths, while some made a single prediction. The median number of predictions was eight, and, in total, 99 celebrities were included in at least one list. At the time of posting, six of the 99 celebrities have passed away.

One way to measure how well a user made predictions is to work down their list, keeping track of every time they correctly predicted a recent celebrity death. This approach to scoring is shown for all 27 users in the graph below. Each blue circle corresponds to a user, and represents their final tally. The location of the circle on the x-axis corresponds to the total length of their list, and the location on the y-axis corresponds to the total number of correct predictions they made. The blue lines leading up to the circles track the progress for each user, working down their ranked lists. We can see that the best any user did was predict two out or the current six deaths, and most users currently have none or one correct predictions in their list.

To try and find some wisdom in this crowd of users, we applied an approach to combining rank data developed as part of our general research into human decision-making, memory, and individual differences. The approach is based on classic models in psychology that go all the way back to the work of Thurstone in 1931, but has some modern tweaks. Our approach allows for individual differences, and naturally identifies expert users, upweighting their opinions in determining the aggregated crowd list. A paper describing the nuts and bolts of our modeling approach can be found here (but note we used a modified version for this problem, because users only provide their “Top-N” responses, and they get to choose N, which is the length of their list).

The net result of our modeling is a list of all 99 celebrities, in an order that combines the rankings provided by everybody. The top 5 in our aggregated list, for the morbidly curious, are Hugo Chavez (already a correct prediction), Fidel Castro, Zsa Zsa Gabor, Abe Vigoda, and Kirk Douglas. We can assess the wisdom of the crowd in the same way we did individuals, by working down the list, and keeping track of correct predictions. This assessment is shown by the green line in the graph below. Because the list includes all 99 celebrities, it will always find the six who have already recently passed away, and the names of those celebrities are shown at the top, in the place they occur in the aggregated list.

Recent Celebrity Deaths and Predictions

The interesting part assessing the wisdom of the crowd is how early in the list it makes correct predictions about recent celebrity deaths. Thus, the more quickly the green line goes up as it moves to the right, the better the predictions of the crowd. From the graph, we can see that the crowd is currently performing quite well, and is certainly about the “chance” line, represented by the dotted diagonal. (This line corresponds to the average performance of a randomly-ordered list).

We can also see that the crowd is performing as well as, or better than, all but one of the individual users. Their blue circles are shown again along with crowd performance. Circles that lie above and to the left of the green line indicate users outperforming the crowd, and there is only one of these. Interestingly, predicting celebrity deaths by using age, and starting with the oldest celebrity first, does not perform well. This seemingly sensible heuristic is assessed by the red line, but is outperformed by the crowd and many users.

Of course, it is only May, so the predictions made by users on Ranker have time to be borne out. Our wisdom of the crowd predictions are locked in, and we will continue to update the assessment graphs.

– Michael Lee

An Opinion Graph of the World’s Beers

One of the strengths of Ranker‘s data is that we collect such a wide variety of opinions from users that we can put opinions about a wide variety of subjects into a graph format.  Graphs are useful as they let you go beyond the individual relationships between items and see overall patterns.  In anticipation of Cinco de Mayo, I produced the below opinion graph of beers, based on votes on lists such as our Best World Beers list.  Connections in this graph represent significant correlations between sentiment towards connected beers, which vary in terms of strength.  A layout algorithm (force atlas in Gephi) placed beers that were more related closer to each other and beers that had fewer/weaker connections further apart.  I also ran a classification algorithm that clustered beers according to preference and colored the graph according to these clusters.  Click on the below graph to expand it.

Ranker's Beer Opinion Graph

One of the fun things about graphs is that different people will see different patterns.  Among the things I learned from this exercise are:

  • •The opposite of light beer, from a taste perspective, isn’t dark beer.  Rather, light beers like Miller Lite are most opposite craft beers like Stone IPA and Chimay.
  • •Coors light is the light beer that is closest to the mainstream cluster.  Stella Artois, Corona, and Heineken are also reasonable bridge beers between the main cluster and the light beer world.
  • •The classification algorithm revealed six main taste/opinion clusters, which I would label: Really Light Beers (e.g. Natural Light), Lighter Mainstream Beers (e.g. Blue Moon), Stout Beers (e.g. Guinness), Craft Beers (e.g. Stone IPA), Darker European Beers (e.g. Chimay), and Lighter European Beers (e.g. Leffe Blonde).  The interesting parts about the classifications are the cases on the edge, such as how Newcastle Brown Ale appeals to both Guinness and Heineken drinkers.
  • •Seeing beers graphed according to opinions made me wonder if companies consciously position their beers accordingly.  Is Pyramid Hefeweizen successfully appealing to the Sam Adams drinker who wants a bit of European flavor?  Is Anchor Steam supposed to appeal to both the Guinness drinker and the craft beer drinker?  I’m not sure if I know enough about the marketing of beers to know the answer to this, but I’d be curious if beer companies place their beers in the same space that this opinion graph does.

These are just a few observations based on my own limited beer drinking experience.  I tend to be more of a whiskey drinker, and hope more of you will vote on our Best Tasting Whiskey list, so I can graph that next.  I’d love to hear comments about other observations that you might make from this graph.

– Ravi Iyer

Ranker Uses Big Data to Rank the World’s 25 Best Film Schools

NYU, USC, UCLA, Yale, Julliard, Columbia, and Harvard top the Rankings.

Does USC or NYU have a better film school?  “Big data” can provide an answer to this question by linking data about movies and the actors, directors, and producers who have worked on specific movies, to data about universities and the graduates of those universities.  As such, one can use semantic data from sources like Freebase, DBPedia, and IMDB to figure out which schools have produced the most working graduates.  However, what if you cared about the quality of the movies they worked on rather than just the quantity?  Educating a student who went on to work on The Godfather must certainly be worth more than producing a student who received a credit on Gigli.

Leveraging opinion data from Ranker’s Best Movies of All-Time list in addition to widely available semantic data, Ranker recently produced a ranked list of the world’s 25 best film schools, based on credits on movies within the top 500 movies of all-time.  USC produces the most film credits by graduates overall, but when film quality is taken into account, NYU (208 credits) actually produces more credits among the top 500 movies of all-time, compared to USC (186 credits).  UCLA, Yale, Julliard, Columbia, and Harvard take places 3 through 7 on the Ranker’s list.  Several professional schools that focus on the arts also place in the top 25 (e.g. London’s Royal Academy of Dramatic Art) as well as some well-located high schools (New York’s Fiorello H. Laguardia High School & Beverly Hills High School).

The World’s Top 25 Film Schools

  1. New York University (208 credits)
  2. University of Southern California (186 credits)
  3. University of California – Los Angeles (165 credits)
  4. Yale University (110 credits)
  5. Julliard School (106 credits)
  6. Columbia University (100 credits)
  7. Harvard University (90 credits)
  8. Royal Academy of Dramatic Art (86 credits)
  9. Fiorello H. Laguardia High School of Music & Art (64 credits)
  10. American Academy of Dramatic Arts (51 credits)
  11. London Academy of Music and Dramatic Art (51 credits)
  12. Stanford University (50 credits)
  13. HB Studio (49 credits)
  14. Northwestern University (47 credits)
  15. The Actors Studio (44 credits)
  16. Brown University (43 credits)
  17. University of Texas – Austin (40 credits)
  18. Central School of Speech and Drama (39 credits)
  19. Cornell University (39 credits)
  20. Guildhall School of Music and Drama (38 credits)
  21. University of California – Berkeley (38 credits)
  22. California Institute of the Arts (38 credits)
  23. University of Michigan (37 credits)
  24. Beverly Hills High School (36 credits)
  25. Boston University (35 credits)

“Clearly, there is a huge effect of geography, as prominent New York and Los Angeles based high schools appear to produce more graduates who work on quality films compared to many colleges and universities,“ says Ravi Iyer, Ranker’s Principal Data Scientist, a graduate of the University of Southern California.

Ranker is able to combine factual semantic data with an opinion layer because Ranker is powered by a Virtuoso triple store with over 700 million triples of information that are processed into an entertaining list format for users on Ranker’s consumer facing website, Ranker.com.  Each month, over 7 million unique users interact with this data – ranking, listing and voting on various objects – effectively adding a layer of opinion data on top of the factual data from Ranker’s triple store. The result is a continually growing opinion graph that connects factual and opinion data.  As of January 2013, Ranker’s opinion graph included over 30,000 nodes with over 5 million edges connecting these nodes.

– Ravi Iyer

Predicting Box Office Success a Year in Advance from Ranker Data

A number of data scientists have attempted to predict movie box office success from various datasets.  For example, researchers at HP labs were able to use tweets around the release date plus the number of theaters that a movie was released in to predict 97.3% of movie box office revenue in the first weekend.  The Hollywood Stock Exchange, which lets participants bet on the box office revenues and infers a prediction, predicts 96.5% of box office revenue in the opening weekend.  Wikipedia activity predicts 77% of box office revenue according to a collaboration of European researchers.  Ranker runs lists of anticipated movies each year, often for more than a year in advance, and so the question I wanted to analyze in our data was how predictive is Ranker data of box office success.

However, since the above researchers have already shown that online activity at the time of the opening weekend predicts box office success during that weekend, I wanted to build upon that work and see if Ranker data could predict box office receipts well in advance of opening weekend.  Below is a simple scatterplot of results, showing that Ranker data from the previous year predicts 82% of variance in movie box office revenue for movies released in the next year.

Predicting Box Office Success from Ranker Data
Predicting Box Office Success from Ranker Data

The above graph uses votes cast in 2011 to predict revenues from our Most Anticipated 2012 Films list.  While our data is not as predictive as twitter data collected leading up to opening weekend, the remarkable thing about this result is that most votes (8,200 votes from 1,146 voters) were cast 7-13 months before the actual release date.  I look forward to doing the same analysis on our Most Anticipated 2013 Films list at the end of this year.

– Ravi Iyer

by    in Data Science

Crowdsourcing Objective Answers to Subjective Questions – Nerd Nite Los Angeles

A lot of the questions on Ranker are subjective, but that doesn’t mean that we cannot use data to bring some objectivity to this analysis.  In the same way that Yelp crowdsources answers to subjective questions about restaurants and TripAdvisor crowdsources answers to subjective questions about hotels, Ranker crowdsources answers to a broader assortment of relatively subjective questions such as the Tastiest Pizza Toppings, the Best Cruise Destination, and the Worst Way to Die.

A few weeks ago, I did an informal talk on the Wisdom of Crowds approach that Ranker takes to crowdsource such answers at a Los Angeles bar as part of “Nerd Nite”.  The gist of it is that one can crowdsource objective answers to subjective questions by asking diverse groups of people questions in diverse ways.  Greater diversity, when aggregated effectively, enables the error inherent in answering any subjective question to be minimized.  For example, we know intuitively that relying on only the young or only the elderly or only people in cities or only people who live in rural areas gives us biased answers to subjective questions.  But when all of these diverse groups agree on a subjective question, there is reason to believe that there is an objective truth that they are responding to.  Below is the video of that talk.

If you want to see a more formal version of this talk, I’ll be speaking at greater length on Ranker’s methodologies at the Big Data Innovation Summit in San Francisco this Friday.

– Ravi Iyer

by    in Data Science

How Crowdsourcing can uncover Niche/Trending shows

At Ranker, people give us their opinions in various different ways. Some people vote.  Other people make long lists.  Still others make really short lists.  Some people tell us their absolute favorite things, while others list everything they’ve ever experienced.  One of the advantages of this diversity is that it allows us to examine patterns within these divergent types of opinions.  For example, some things are really popular, meaning that everyone lists them (e.g. Michael Jordan is on everyone’s best basketball players list).  Most popular things are also things that people generally list high on their lists and also get lots of positive votes (e.g. Michael Jordan).  However, there are some things that don’t get listed very often, but when they do get listed, people are passionate about them, meaning that they get listed high on people’s lists.  We highlight these items in our system using the niche symbol.

I’ve recently been examining our “niche” tag, which signifies when something is not particularly popular, but people are passionate about it.  There are many reasons why things can be niche.  Some things appeal specifically to younger (e.g. Rugrats) or older crowds (e.g.  The Rockford Files).  Other things have natural audiences (e.g.baseball fans who appreciate defense and think Ozzie Smith is one of the greatest players of all time).  The most interesting case is when something that I can’t identify starts showing on the niche list (see the list at the time of this writing here).

This is especially helpful for someone like me, who doesn’t always know what is ‘hot’ and naturally looks to data to find new quality entertainment.  Awhile back, the show Community consistently was showing highest on our niche algorithm.  Few people listed it as one of the best recent TV shows, but those who listed it tended to think very highly of it.  I was intruiged enough to watch the pilot on Hulu and have since become hooked.  Community has since graduated from our niche algorithm as it became popular.  Sometimes passion amongst a small group is how a trend starts.

As Margaret Mead believed that only a small group of citizens could change the world, so Malcolm Gladwell has shown how a small group of trendsetters can signal changes in pop culture.  Not everything on our niche list will become the next big thing, but it’s certainly a good place to search for candidates.

Among the things that people seem to be passionate about now, that aren’t so popular, are several good candidates for up and coming movies, bands, or TV shows.  Pappillon is currently hot, scoring over 2 standard deviations higher in terms of list position on our best movie list, despite being less popular than most movies.  Another Earth and 13 Assassins,  seem like potentially interesting and under the radar films from 2011. Real Time with Bill Maher‘s niche status may be due to appeal particular ideological group, but Warehouse 13 appealed to just my niche as it had passionate fans on both the best recent TV shows list and the best Sci-Fi TV shows list (it has since graduated from the list due to increased popularity).  Warehouse 13’s highest correlated show is one of my favorites, Battlestar Galactica, so I’m definitely going to check it out.

I tend to be a late adopter of pop culture, but thanks to the niche tag, maybe I can be a little hipper going forward.  Take a look at our niche items as of October 20, 2012 and any comments on other things to consider checking out would be appreciated. Or perhaps take a look in a few months time and consider whether our niche tag successfully captured coming trends in a few cases.

– Ravi Iyer

by    in Opinion Graph

The Best Possible Answers To Opinion-Based Questions

Ranker, as an openended platform for ranking people/places/things, is a lot of different (awesome) things to different people. But the overarching goal for Ranker has always been to provide the best possible answer to opinion-based questions like “What are the best _____?”

Popular sports and entertainment vote lists often grow into being a great answer within 12-72 hours as they get lots of traffic quickly, but the majority of Ranker lists take 1 – 3 months to build to full credibility as visitors on Ranker and from search engines find them and shape them with votes and re-ranks.

I thought it would be fun to showcase some Ultimate Lists and Vote Lists in other categories that haven’t gone viral, but through the participation of lots of Rankers over a few months have indeed become “the best possible answer” to this question.

Food

You all clearly love to weigh in on the start of the day, and the 5 o’clock hour:

Best Breakfast Cereals

The Best Cocktails

But you also have strong opinions on hydration during the day:

Best Sodas (and for the more calorie-conscious among you The Best Diet Sodas)

And even specific Gatorade flavors (thanks for the list Lucas)

Snacking, whether it be on a particular type of cheese, candy bar, or even as granular as a specific Jelly Belly flavor (thanks for the list Samantha but what’s with all the chocolate pudding haters?)

Dining out, specifically at Italian chain restaurants

A list I am not authorized to vote on, pregnancy cravings

And hundreds more, including perhaps a new category entirely – food nostalgia (I do miss those Crispy M&Ms myself)

Fashion/Beauty

Not categories that I personally check up on much, so I was psyched to see quite a few solid rankings here, some of them high-end but mostly stuff you can find at the mall:

Best women’s shoe brands

Best denim brands

Top handbag designers

Fashion Blogs

Sulfate-free shampoos

And even a men’s facial moisturizers list (have only tried 3 or 4 myself, but agree with their relative positions on the list)

Travel

Rankers, I know from a number of you that as we’ve been adding datasets of “rank-able objects” over the last year, one of the most-requested ones that we don’t yet have is hotels/resorts. Trust me, it’s still on the list. But in the meantime, it’s been heartening to see how many of you have participated in these great resources for travel destinations and attractions, like these:

Best US cities for vacations

Honeymoon destinations

Coolest cities in America

Theme parks for roller coaster addicts

And my personal faves, “bucket lists” of the world’s most beautiful natural wonders and historical landmarks.

Great stuff – these lists and 1000s more like them are true testimonials to the “wisdom of crowds”. Thanks, crowds!

by    in Data Science, Market Research

Validating Ranker’s Aggregated Data vs. a Gallup Poll of Best Colleges

We were talking to someone in the market research field about the credibility of Ranker’s aggregated rankings, and they were intruiged and suggested that we validate our data by comparing the aggregated results of one of our lists to the results achieved by a traditional research company using traditional market research methodologies.  Companies like Gallup often do not survey the same types of questions that we ask at Ranker, in part due to the inherent difficulties of open ended polling via random digit dialing.  You can’t realistically call someone up at dinner time and ask them to list their 50 favorite TV shows.  You could ask them to name one favorite, but doing that, you can end up with headlines like “Americans admire Glenn Beck more than they admire the Pope.”  However, one question that both Gallup and Ranker have asked concerns the nation’s top colleges/universities.  How do Ranker’s results compare to Gallup’s data?  Below are our results, side by side.

Ranker vs Gallup Best US Colleges

From a market researcher’s perspective, this is good news for Ranker data.  Our algorithms have successfully replicated the top 4 results from the Gallup poll exactly, at a fraction of the cost.  This likely occurs because Ranker data is largely collected from users who find our website via organic search, so while our data is not a representative probability sample (assuming such a thing still exists in a world where people screen their calls on cellphones), our users tend to be more representative than the motivated Yelp user or the intellectual Quora user.  If you compare how representative Ranker’s best movies list is compared to Rotten Tomatoes aggregated opinion list (Toy Story 2 and Man on Wire are #1 & #2!?!?), you get a sense of the importance of having relatively representative data.

In addition, the fact that our lists are derived from a combination of methodologies (listing, reranking, + voting), means that the error associated with each method somewhat cancels out.  Indeed, one might argue that Ranker’s top dream colleges list is better than Gallup’s for precisely this reason as individuals are often tempted to list their alma mater or their local school as the best college, and the long tail of answers might actually contain more pertinent information.  Aggregating ranked lists from motivated users and combining that data with casual voters might actually be the best way to answer a question like this.

– Ravi Iyer