Today, Google announced the introduction of the “knowledge graph”, which introduces facts into Google searches. So now, when you search for an object that Google understands, search results reflect Google’s actual understanding, leveraging what they know about each object. Here is a video with more detail.
At Ranker, we know things about specific objects too, as most every item in the Ranker system maps to a Freebase object, which is a company (MetaWeb) that Google bought in order to provide these features. We know a lot of the same information that Google knows, since we leverage the Freebase dataset. For example, on our Godfather page, we present facts such as who directed the movie, when it was released, and what it’s rating was. However, we also present other facts that people traditionally do not think of as part of the knowledge graph, but are actually just as essential to understanding the world. We tell you that it’s one of the best movies of all time. We also tell you that people who like the Godfather also tend to like Goodfellas, the Shawshank Redemption, and Scarlett Johansson.
Is this “knowledge”? These aren’t “hard” facts, but it is a fact that people generally think of The Godfather as a good movie and Gilgi as a bad movie. Moreover, knowledge about people’s opinions is essential for understanding the world in the way that the “Star Trek computer” that is referred to in Google’s blog post understands the world. Could you pick a college based on factual information about enrollment and majors offered? Could you hold an intelligent conversation about Harvard without knowing it’s place in the universe of universities? Could you choose a neighborhood to live in based solely on statistics about the neighborhood, or would understanding what neighborhoods people like you also tend to like help you make the right choice? If the broader mission of a search engine is to help you answer questions, then information about people’s opinions about colleges and neighborhoods is essential in these cases. The knowledge graph isn’t just about facts, it’s about opinions as well. Much of the knowledge you use in everyday reasoning concerns opinions, and if the internet is to get smarter, it needs this knowledge just as much as it needs to know factual information.
NBC’s trend-setting ensemble sitcom “30 Rock” is wrapping up its sixth season, and remains one of the most discussed shows on all of Ranker. The series currently ranks 17th on our list of History’s Greatest Sitcoms as well as having a strong showing on the Funniest Shows of 2011 round-up. As well, main characters Tracy Jordan and Jack Donaghy BOTH cracked the Top 20 on our Funniest TV Characters Of All Time list. (No Liz Lemon until #35? Come on, gang!)
This many votes on “30 Rock” spanning this many lists gives us A LOT of data to sift through for interesting correlations. And wouldn’t you know, we found one. Namely, fans of “30 Rock” by and large seem to enjoy surprisingly dark film entertainment. More so than you’d think from a show about the wacky behind-the-scenes hijinks on a sketch comedy show that contains this many fart jokes and Werewolf Bar Mitzvahs.
The 2 films “30 Rock” aficionados are most likely to enjoy? You guessed it, “The Deer Hunter” and “Raging Bull.” “30 Rock” fans are… get this… nearly 2000% more likely to enjoy “Deer Hunter” than some schmo off the street, and almost as enthusiastic about Scorsese’s boxing biopic.
Aside from the presence of Robert De Niro, and generally being really really awesome, these films have in common an unsettling, gritty outlook, not to mention protagonists who may not always be relateable. It’s kind of hard picturing people sympathizing with Jake LaMotta’s violent temper and fits of jealous rage, then switching over to chuckle at Jack Donaghy’s doppelganger, El Comandante. Yet that’s apparently just what’s happening.
Grab some Sabor de Soledad, niños, cause we’re gonna watch Bobby D get tortured in a Vietnamese POW camp.
It doesn’t stop there. We also noticed that some of “30 Rock” fans’ favorite film and TV characters are not what you’d expect from people who can’t get enough of Kenneth Parcell’s down-home folksy wisdom. For example, aside from Liz Lemon herself, and “Arrested Development’s” Buster Bluth, the most popular fictional character among “30 Rock” fans is Kurt Russell’s Snake Plissken from the John Carpenter “Escape” movies. Now, granted, those movies are sort of funny, but not quite in the same way that “30 Rock” is funny. Although both projects do involve a love of shoddy greenscreen effects:
Sarah Connor from “The Terminator” films also wins surprisingly favorable reviews from loyal “30 Rock” viewers. No word on whether they like the more girly mall-rat version from “The Terminator” or her later, tormented and also super-buff self. Maybe we’ll dig that up for a future post.
In a previous post, we talked about a bit about how Ranker collects users into like-minded “clusters” that allowed for statistical analysis. This method is how we were able to look at “Game of Thrones” fans and figure out other shows, characters, games and movies they might like.
Now, let’s dig a bit deeper into how this analysis works, and what sort of things we can learn from it. Essentially, breaking down the users who vote on our lists into clusters of people with similar taste lets us predict how fans of one thing will feel about some other thing.
We use the advertising term “Lift %” to represent this idea, but it basically boils down to an odds ratio. We’re measuring the projected increase in someone’s interest level for something, based on their preference for something else. Therefore, we don’t even just have to compare fans of one show to another, or fans of one movie to another. Sure, we can tell what TV shows you’ll probably like if you like “Game of Thrones,” but we can also tell what people you’ll respond to positively, or what websites you prefer, or your favorite athlete.
For another example, let’s look at the 1998 comedy-drama “Rushmore.” Along with “Bottle Rocket,” this was really the film that made Wes Anderson a household name, and also contains one of Bill Murray’s most beloved and iconic performances.
The first big trend we noticed among this like-minded cluster of “Rushmore” fans was that they tended to like other comedy films, too. Which you’d sort of expect. Except these fans tended to prefer classic comedies to more contemporary films. In fact, all of these films had a greater “Lift %” among “Rushmore” fans than any films made in the 1990s, when the film actually came out:
“Dr. Strangelove” (1964)
“The General” (1926)
“Modern Times” (1936)
“The Lady Eve” (1941)
“A Night at the Opera” (1935)
As well, all of these films had a Lift % of OVER 500%, which means someone who likes “Rushmore” is 500% more likely to enjoy, say, “A Night at the Opera,” than someone who is ambivalent about “Rushmore.” That strikes us as statistically significant. (The numbers are even higher the further up the list you go. A “Rushmore” fan is 1000% more likely to enjoy “Dr. Strangelove” than a random person.)
From what we can tell, it works the other way, too. “Rushmore” is the most popular overall film among “Annie Hall” fans and #4 overall among fans of Charlie Chaplin’s “City Lights.” Exactly WHY Wes Anderson’s coming-of-age dramedy scores so well among lovers of old movies is up for debate, but the correlation itself is not, really, based on the numbers.
We’re continuing to develop and fine-tune our reports, of course. And it’s worth remembering that we get the BEST results on popular stuff that gets voted on all the time. It’s not too hard to tell what kind of music Jay-Z fans will like (though we’ll save that for another blog post), but we won’t do nearly as well for Captain Beefheart fans. Yet.
As part of our effort to promote Ranker’s unique dataset, I recently attended the Data 2.0. conference in San Francisco. “Data 2.0.” is a relatively vague term, and as Ranker’s resident Data Scientist, I have a particular perspective on what constitutes the future of data. My PhD is in psychology, not computer science, and so for me, data has always been a means, rather than an end. One thing that became readily apparent at the first few talks I saw, was that a lot of the emphasis of the conference was on dealing with bigger data sets, but without much consideration of what one could do with this data. It goes without saying that larger sample sizes allow for more statistical power than smaller sample sizes, but as the person who has collected some of the larger samples of psychological data (via YourMorals.org and BeyondThePurchase.org), I have often found that what holds me back from predictive power with my data is not the volume of data, but rather the diversity of variables in my dataset. What I often need is not bigger data, it’s better data.
The same premise has informed much of our data decision making at Ranker, where we emphasize the quality of our semantic, linked data, as opposed to the quantity. Again, both quality and quantity are important, but my thought going through the conference was that there was an over-emphasis on quantity. I didn’t find anyone talking about semantic data, which is one of the primary “Data 2.0.” concepts that relates more to quality than quantity.
I tested this idea out with a few people at the conference, framed as “better data beats better algorithms” and generally got positive feedback about the phrase. I was heartened when the moderator of a panel entitled “Data Science and Predicting the Future”, which included Alex Gray, Anthony Goldbloom, and Josh Wills, specifically proposed the question as to what was more important, data, people, or algorithms. It wasn’t quite the question I had in mind, but it served as a great jumping off point for a great discussion. Josh Wills, who worked as a data scientist at Google previously actually said the following, which I’m paraphrasing, as I didn’t take exact notes:
“Google and Facebook both have really smart people. They use essentially the same algorithms. The reason why Google can target ads better than Facebook is purely a function of better data. There is more intent in the data related to the Google user, who is actively searching for something, and so there is more predictive power. If I had a choice between asking my team to work on better algorithms or joining the data we have with other data, I’d want my team joining my data with other data, as that is what will lead to the most value.”
Again, that is paraphrased. Some of the panelists disagreed a bit. Alex Gray works on algorithms and so emphasized the importance of algorithms. To be fair, I work with relatively precise data, so I have the same bias in emphasizing the importance of quality data. Daniel Tunkelang, Principal Data Scientist of LinkedIn, supported Josh, in saying that better data was indeed more important than bigger data, a point his colleague, Monica Rogati, had made recently at a conference. I was excited to hear that others had been having similar thoughts about the need for better, not bigger, data.
I ended up asking a question myself about the Netflix challenge, where the algorithms and collective intelligence addressing the problem (reducing error of prediction) were maximized, but the goal was a relatively modest 10% gain, which was won by a truly complex algorithm that Netflix itself found too costly to use, relative to the gains. Surely better data (e.g. user opinions about different genres or user opinions about more dimensions of each movie) would have led to much greater than a 10% gain. There seemed to be general agreement, though Anthony Goldbloom rightly pointed out that you need the right people to help figure out how to get better data.
In the end, we all have our perspectives, based perhaps on what we work on, but I do think that the “better data” perspective is often lost in the rush toward larger datasets with more complex algorithms. For more on this perspective, here and here are two blog posts I found interesting on the subject. Daniel Tunkelang blogged about the same panel here.
Last week, we published an info graphic with lots of “taste data” about “Game of Thrones” fans. Basically, we used all the data we’re collecting about people’s preferences in Ranker to make some educated guesses about what else people who like “Game of Thrones” might like. Why? Mostly because we can, but also because we figured people could potentially find it interesting.
After we showed the infographic to the world, a lot of people wrote to us asking how we actually arrived at these conclusions. (And yes, some of them just wanted to be sure we weren’t just making the whole thing up.)
It all starts with votes. Thousands of people have voted on Ranker lists on which “Game of Thrones” appears. If they’re on a list that’s “positive” (for example, “Best Premium Cable Shows”) and they vote “Game of Thrones” up, we know they like the show. If we notice they also vote for “Game of Thrones” on other lists (“Most Loving Caresses of Dragon Eggs in TV History,” for example), we know they REALLY like the show.
Then we look at all the other Ranker lists where that person has voted, and get a sense for what else they like, and what else they hate.
But we don’t stop there. The next step is to arrange people into clusters based on their specific preferences. If 80% of the people who vote on Ranker lists like “The Simpsons,” and 80% of “Game of Thrones” fans like “The Simpsons,” that’s not very meaningful at all. But if only 20% of people who vote like “The Simpsons,” and 80% of “Game of Thrones” fans like “The Simpsons,” then we’ve learned something statistically significant about these people.
But what about fans of “Simpsons” parodies of “Game of Thrones,” you might ask… if you were purposefully trying to confuse me.
These “clusters” of people with tastes that are aligned will teach us basically everything we need to know to make educated guesses about what random Ranker users will like. In our next post, we’ll explore exactly how we use these “taste clusters” to draw conclusions.
At Ranker HQ, we’re constantly monitoring the topics that get ranked a lot. It’s pretty easy to tell when a certain book or movie or musical artist is getting popular or hitting critical mass just based on how frequently the name is mentioned on lists. This is especially true of TV, where the start of a new season for a popular show means an eruption of lists mentioning that show. (Don’t believe me? Check out all the “Mad Men” lists streaming in!)
We weren’t necessarily surprised that HBO viewers were losing their heads for “Game of Thrones.” (See what I did there?) It’s back for Season 2, and obviously Rankers are going to have fun making tons of lists about the sword-and-sorcery-and-skin fantasy series based on George R. R. Martin’s novels. Instead, we were intrigued because the data reveals Game of Thrones fans are just as… idiosyncratic as the show they love. (Yes, idiosyncratic is a nice way of putting it. But hey, we’re not here to INSULT our users.)
And we say this not just because they watch a show in which incest happens as often as other series take commercial breaks. Also because they overwhelming love villainous characters and anti-heroes and they prefer a lot of lesser-known shows that failed to ever find an audience.
Read on for more insight into the weird, even twisted world of “Game of Thrones” fans (or Throne-heads, as we’ve dubbed them.)
Like the graphic? Feel free to repost it anywhere you like. Spread the word throughout the Seven Kingdoms!
Below is a narrated powerpoint from a presentation I gave at South by Southwest Interactive on March 11, 2012. The point of this presentation was to explore the intersection of technology and psychology, and hopefully to convince technologists to try to use our data to examine intangible things like values. While the talk focuses more on psychology, many of the ideas were inspired by the semantic datasets we work with at Ranker. Working with semantic datasets puts one in the mindset of considering synergy among different fields with different kinds of data.
Perhaps you didn’t know Ranker had a whole large laboratory full of scientists in neatly pressed white coats doing crazy, some might even say Willy Wonka-esque experiments. We try to keep that sort of thing fairly under wraps. The government’s been sort of cracking down on evil science ever since that Freeze Ray incident a few years back… you know the one I mean…
A rare glimpse behind the curtain at how Ranker lists are made. Photo by RDECOM.
Anyway, recently, our list technicians have been playing around with CrowdRanked lists. We get a lot of Ranker users giving us their opinion on these lists.
(Ranker’s CrowdRankings invite our community members to all gather together and make lists about one topic. Then everyone else can come in and vote on what they think. When it’s all been going on for a while, and a bunch of people have participated, you get a list that’s a fairly definitive guide to that topic.)
One list that has interested us in particular is this one: The Worst Movies of All Time. Almost 70 people have contributed their own lists of the worst films ever, and thousands of other members of the Ranker community have voted.
And what do we learn from this list? Everyone really, really, really hates “Gigli.” I mean, hates it. That movie is no good at all.
Ben Affleck does his impression of everyone watching more than 5 minutes of ‘Gigli.’
It comes in #2 right now, with almost 700 votes upholding its general crapitude. The only movie topping it in votes right now is Mariah Carey’s vanity project, “Glitter,” which, to be fair, barely qualifies as “a movie.”
But our scientists – because they are seriously all about science – thought, there must be something more we can do with this data now that we’ve collected it. And wouldn’t you know, they came up with something. They call it “FactorAnalysis.” I call it “The thing on my desk I’m supposed to write about after I have a few more cups of coffee.”
So What Is FactorAnalysis Anyway?
Here’s how the technicians explained it to me…
We’re going to perform a statistical analysis of the votes we collected on the “Worst Movies Ever” list. (Just the votes, not the lists people made nominating movies.) To do this, we’re going to break up the list of movies into groups based on similarities in people’s voting patterns. (That is, if a lot of people voted for both “Twilight” and “From Justin to Kelly,” we might group them together. If a lot of those same people voted against “Catwoman,” we’d put that in a separate group.)
Sometimes, you’ll be able to look at the grouping and the common thread between those choices will be obvious. Of course the same people hated “Lady in the Water’ and “The Last Airbender.” They can’t stand M. Night Shyamalan (or, perhaps more accurately, they can’t stand what he has become.) Not exactly a shocking twist there.
The Airbender gains his abilities by harnessing the power of constant downvotes.
But other times, the groupings will not be quite as obvious, and that’s where the analysis can get more intriguing. Once we collect enough data, we’ll be able to make all kinds of weird connections between movies, and maybe figure out a more Unified Theory of Bad Movies than currently exists! (Hey, a blogger can dream…)
When doing this kind of factor analysis, you first must determine the number of groups that exist in your data. We used something called a Catell’s Scree Test to determine the number of groups. (This is fancy-talk for saying: “We plot everything on a graph like the one below, and look for the elbow – the point where the steepness of the dropoff between factors is the greatest.”)
The “Eigenvalue” that you see along the Y axis there is a measure of the importance of each factor. It helps us to differentiate between significant factors (the “signal”) from insignificant ones (the “noise”).
Once we decide how many factors we have, it’s time to actually extract factors whereby we determine which movies load on which factors. It sounds precise and mathematical, but there’s some amount of subjectivity that still comes into play. For example, let’s say you were talking about your favorite foods. (Yes, yes, we all love “bacon,” but be serious.)
One way to group them would be on a spectrum from spicy to bland foods. But you could also choose to go from very exotic foods to more ordinary, everyday ones. Or starting with healthy foods and moving into junk food. Each view would be a legitimate way to classify food, so a decision must be made on some level about how to “rotate” the factor solution.
In our case, we chose what’s called the “varimax rotation,” which maximizes the independence of each factor and tries to prevent a ton of overlap. This allows us to break up the movies into interesting sub-groups, rather than just having one big list of “bad” films (which is where we started out.)
Doing that yields the below chart.
Along the top, you can see the factors that were extracted. The higher the number a film gets for a certain component, the more closely aligned it is with that component. Using these charts, we can then place movies in “Factors,” or categories, with relative ease.
Unfortunately, the program can only get us this far – we can see the factors, but we can’t tell why certain items apply to certain factors and not others.
So What Can FactorAnalysis Tell Us About the Worst Movies?
First, our lab rats managed to split the entire Worst Movies List (containing 70 total films) into 5 different categories.
Category 1 (we called it “Factor 1”) contained the most movies overall, so whatever the common thread was, we knew that it must be something that people immediately identified with “bad movies.” Some of the titles that most closely correlated with Factor 1 were:
– “Monster a Go-Go” – “Manos: The Hands of Fate” – “Crossover” – “The Final Sacrifice” – “Zombie Nation”
We decided that “Classic B-Movie Horror” was the best way to describe this grouping. Of the group, 1965’s “Monster a Go-Go” was the most representative item, and it didn’t really overlap with any of the other groups. The film is a fairly standard horror/sci-fi matinee of the time. An astronaut crashes back to Earth having suffered radiation poisoning, and then goes on a rampage.
So when most Rankers think about what makes a movie “bad,” they tend to think of older, low budget movies that fail at being scary, and maybe have a sci-fi element as well.
Factor 2 was a bit harder to pin down. Lots more movies seemed to fall into or overlap with this category, but it was a bit tricky to pinpoint what they had in common. Representative Factor 2 movies included:
and the most representative of all for Factor 2 was “Gigli.” (See all the movies relating to Factor 2 here.)
We settled on “Cheesiness” as a good common thread for these movies. (Especially if you continue on down the list: “Battlefield Earth,” “The Room,” “Batman and Robin,” “Superman IV: The Quest for Peace”…yeesh…)
Note here that “Gigli” was the film that most closely correlated to Factor 2 (what we have deemed “cheesy movies”), and “Glitter” was also considered highly cheesy. Yet “Glitter” is the overall most popular “Worst Movie” on the list, when going by straight votes. This seems to indicate that “Gigli” was hated SOLELY because it is cheesy, while “Glitter” commits numerous cinematic crimes, including cheesiness.
Factor 3 had even fewer films that closely correlated, but it was very simple to figure out what they all had in common. Consider the movies that were most representative of Factor 3:
– “The English Patient” – “The Family Stone” – “Far and Away” – “Legends of the Fall” – “The Fountain” – “Eyes Wide Shut” (oh come on are you guys kidding it’s freaking Kubrick!) – “What Dreams May Come”
Let’s call this the “Self-Important Pretension” group. People who hate movies that are self-consciously “artsy” and “important” REALLY hate those movies, and will pretty much always pick them over other bad movies from other genres. These folks are just outnumbered by the people who think it’s worse to be old-fashioned or cheesy than pompous. (At least, people ON RANKER.)
Factors 4 and 5 are sort of interesting. It’s definitely harder to make a clear-cut distinction between these two groups when you’re just looking at the films. We know they are distinct, because of the voting patterns that created them. But consider the actual movies:
– “Star Wars: Episode I: The Phantom Menace” – “Transformers: Revenge of the Fallen” – “Indiana Jones and the Kingdom of the Crystal Skull” – “Spider-Man” – “Godzilla” (the 1998 Matthew Broderick version) – “Star Wars: Episode II: Attack of the Clones” – “Pearl Harbor”
Certainly, if you didn’t like Best Picture winners “Forrest Gump,” “Million Dollar Baby” and “Avatar,” you considered them disappointments? “Quantum of Solace” was the lukewarm follow-up to “Casino Royale,” one of the best Bond films of all time. And “Temple of Doom” is the sequel to arguably the best adventure movie ever made, “Raiders of the Lost Ark.”
So how come the movies in Factor 4 closely correlated with one another, and the movies in Factor 5 closely correlated with one another, if they’re BOTH groups of disappointing films? Maybe they disappointed different people, or they disappointed people in different ways?
One theory: Factor 4 films are entries in above-average franchises that are considered not as good as the other films. (This doesn’t quite apply to “Pearl Harbor,” unless you consider Michael Bay Movies to be a franchise. As I do.) The people who agreed on voting for these films felt that the worst thing a movie can do is disappoint fans of other, similar movies.
For example, movies starring Ben Affleck…
This would make Factor 5 the “overhyped” category. Everyone’s “supposed” to love “Million Dollar Baby” and “Avatar” and “Forrest Gump.” And the people who don’t like them feel a curmudgeonly sense of kinship around some of these titles. (One would expect “The English Patient,” then, to fall into this factor. Unfortunately for our theory, it’s most closely aligned with Factor 3, the “Long and Boring” category.)
More theories as to the strange circumstances of Factor 4 and 5 are certainly welcome. We just thought it was kind of an intriguing puzzle.
There were 3 movies that seemed to coalesce into a “Factor 6,” but we didn’t have enough data and enough films didn’t correlate to create a true category in any meaningful sense. So it may forever elude us what “Waterworld,” “The Postman” and “Road House” have in common. Aside from kicking ass, amiright? R-r-right?
Movies That Scored High in Multiple Factors
Some movies didn’t closely align with any single group, but nonetheless scored high for numerous different factors. For example, “Masters of the Universe,” the ill-fated live-action ’80s adaptation of the He-Man line of toys. “Masters of the Universe” was somewhat aligned with Factor 1 – the “dated B-movie genre” group – as well as Factor 3 – the self-important pretension group. Now that is just weird. I mean, yes, He-Man is kind of a blowhard, with all that “I Have the Power!” stuff. But I don’t really think of it as terribly similar to “The English Patient” when all is said and done.
Also, consider “Lady in the Water.” It aligns fairly closely with Factors 1, 2 AND 3, and even makes a showing in Factor 4. This is a movie upon which haters of every kind of movie can agree.
A Look at Things to Come
So, that’s how we’ve gotten started with using FactorAnalysis on some of our CrowdRanked lists. Isn’t it very very very interesting, such that you’d like to tell all of your friends about what you’ve just read? If only there were some kind of digital environment where people could socially interact and share hypertextual links to information that they enjoy with their friends…
Be sure to check out the next edition of Ranker Labs, coming in a few weeks, when we’ll apply some FactorAnalysis to ANOTHER one of our big CrowdRanked lists – History’s Worst People.