Recommender systems

1 · What it is

A recommender system chooses which items, from a catalogue far too big to browse, each person sees first. The items might be videos on YouTube or apps in the Google Play store. EU law defines one as an automated system that a platform uses to suggest or prioritise information, including setting its order or prominence. A common design works in three stages. A quick first pass finds likely items, a richer model scores them, and a last pass adjusts the order.

Two families of method find the candidates. Content-based filtering compares item features with what one person already liked, so it needs no data about anyone else. Collaborative filtering looks at people and items together, so it can suggest a video to you because someone with similar taste liked it. Amazon’s item-to-item version, described in a 2003 paper, links products instead: B is related to A if buyers of A are unusually likely to buy B. Both approaches usually turn people and items into embeddings, lists of numbers arranged so that similar things sit close together.

The evidence comes in two kinds. Explicit feedback is a rating the person gives. Implicit feedback is behaviour, such as watching a film, from which the system infers interest. YouTube’s 2016 paper says it trains on watches because there is far more of that history than explicit feedback. For new profiles, Netflix asks for a few liked titles to jump start recommendations, or starts with a diverse, popular set.

What the scorer predicts shapes what people see. Google’s course warns that scoring on clicks can favour click-bait, and scoring on watch time can favour very long videos. YouTube’s paper reports ranking mainly on expected watch time per impression, because ranking on click-through rate promoted videos people did not finish. Its authors also say that training only on watches the recommender produced itself would make new videos hard to surface and push the system toward exploitation. It is the trade-off between exploring new videos and exploiting well-established ones. In the EU, online platforms must explain the main parameters of their recommender systems in their terms and conditions. Very large platforms must also offer at least one option that is not based on profiling.