Hidden features: how movie recommendation systems extract insights from your ratings

🎞️ 57 channels and nothing on; A look inside a movie recommendation system; I guess I'm going to watch Wallace & Gromit

🎵 57 channels and nothing on…

In 1992 Bruce Springsteen released his song about having so many TV channels and still feeling like there’s nothing to watch.

🎵 57 channels and nothing on

It's called “the paradox of choice": given 3 movie ideas, we pick the one we like the most and get going. But with 15 options it gets harder: we take longer to decide and can’t quite enjoy whatever we end up picking.

The algorithm

Modern problems, modern solutions. Today online video platforms use recommendation engines to help with the decision overhead. (TikTok perfected this approach: the user no longer needs to make any choices, they just keep scrolling.)

But how do such recommendation algorithms work? Let’s find out by building one!

Movie ratings

We’re going to make a movie recommendation system. It will use the wisdom of the crowd (what other people liked and disliked) to create personalized suggestions of what to watch and what to avoid.

“Personalized” is the key here. People who love “Casablanca” tend to have different style preferences from those who love “Sonic 2”.

We’re going to use the MovieLens 100k dataset, where 943 users already rated 1682 movies on the scale from 1 to 5:

Hidden features

Our system will use nothing more than those simple ratings. No information about genre, cast, duration, nor any other metadata.

Instead, we’re going to extract information that is implicitly stored in the rating data. Let’s call this information “hidden features”. (It’s a less intimidating name than the official jargon: “latent factors”.)

Just to illustrate what the hidden features could be:

1. 🍿 mass appeal: -1.0 for ”The Room”, 1.0 for hollywood blockbusters

2. 👽 sci-fi: -1.0 for “Amadeus”, 1.0 for “2001: A Space Odyssey”

3. 😲 dialogue-driven: -1.0 for “Die Hard”, 1.0 for Woody Allen movies

… etc.

Similarly, each user would have a taste profile defined in terms of those features:

1. 🍿 how much they like “mass appeal” movies

2. 👽 how much they like “sci-fi”

3. 😲 how much they like films that are “dialogue-driven”

Having those features, we could easily estimate how good of a match there is between a given movie and a given user, just by comparing their profiles feature by feature.

Squeezing the data set

Of course, we don’t know what those features are. And here comes the magical part: we don’t know, but we can pretend that we do.

We will arbitrarily decide that there are 10 (or 3, or 50) features, and set them to random values. For all users and for all movies, across all features. Completely made up values.

We then calculate the quality of the movie predictions we would get using those feature values… and start fiddling with the numbers.

We change the feature values little by little, in a way that improves the resulting accuracy. In ML jargon this technique is called “gradient descent”. We keep doing this until we get diminishing returns. This technique may seems desperate, but it’s a foundational method for all Deep Learning systems big and small. It works 💫.(We already saw Gradient descent when calculating Titanic survival predictions.)

Results

To get individual recommendations, we need to include our own ratings in the data, so that the algorithm learns our taste, along with everyone else’s.

I added 30 ratings based on my letterboxd history (finally useful for something):

The model takes less than a minute to train (without GPU acceleration), and just like that, we get the resulting recommendations. The top 3 movies predicted for my taste:

The winner is … a 1995 British stop-motion animation about a guy and his dog operating a window-cleaning business 🤔 ?

Time to stop writing and start watching :).

More on this

• ⭐️ This exercise is based on PDL4C lesson 7 by Jeremy Howard

• 👨‍💻 Source code

• 📝 When Choice is Demotivating [2000], a paper describing the famous Paradox of choice jam experiment

Postcard from Paris

Night walk by Canal de l'Ourcq ❤️. It’s now below zero in Paris, “snow and ice cause chaos”. Following a treatment of dumpling soup and massive amounts of green tea until the morale improves 💫.

Have a great week,
– Przemek