RStudio AI Weblog: Collaborative filtering with embeddings


What’s your first affiliation once you learn the phrase embeddings? For many of us, the reply will in all probability be phrase embeddings, or phrase vectors. A fast seek for current papers on arxiv exhibits what else might be embedded: equations(Krstovski and Blei 2018), car sensor knowledge(Hallac et al. 2018), graphs(Ahmed et al. 2018), code(Alon et al. 2018), spatial knowledge(Jean et al. 2018), organic entities(Zohra Smaili, Gao, and Hoehndorf 2018) … – and what not.

What’s so enticing about this idea? Embeddings incorporate the idea of distributed representations, an encoding of knowledge not at specialised areas (devoted neurons, say), however as a sample of activations unfold out over a community. No higher supply to quote than Geoffrey Hinton, who performed an vital position within the improvement of the idea(Rumelhart, McClelland, and PDP Analysis Group 1986):

Distributed illustration means a many to many relationship between two sorts of illustration (akin to ideas and neurons). Every idea is represented by many neurons. Every neuron participates within the illustration of many ideas.

The benefits are manifold. Maybe probably the most well-known impact of utilizing embeddings is that we are able to be taught and make use of semantic similarity.

Let’s take a process like sentiment evaluation. Initially, what we feed the community are sequences of phrases, primarily encoded as components. On this setup, all phrases are equidistant: Orange is as completely different from kiwi as it’s from thunderstorm. An ensuing embedding layer then maps these representations to dense vectors of floating level numbers, which might be checked for mutual similarity by way of varied similarity measures akin to cosine distance.

We hope that once we feed these “significant” vectors to the subsequent layer(s), higher classification will end result. As well as, we could also be curious about exploring that semantic area for its personal sake, or use it in multi-modal switch studying (Frome et al. 2013).

On this publish, we’d love to do two issues: First, we wish to present an attention-grabbing software of embeddings past pure language processing, specifically, their use in collaborative filtering. On this, we comply with concepts developed in lesson5-movielens.ipynb which is a part of quick.ai’s Deep Studying for Coders class. Second, to collect extra instinct, we’d like to have a look “underneath the hood” at how a easy embedding layer might be applied.

So first, let’s soar into collaborative filtering. Similar to the pocket book that impressed us, we’ll predict film rankings. We are going to use the 2016 ml-latest-small dataset from MovieLens that accommodates ~100000 rankings of ~9900 motion pictures, rated by ~700 customers.

Embeddings for collaborative filtering

In collaborative filtering, we attempt to generate suggestions primarily based not on elaborate information about our customers and never on detailed profiles of our merchandise, however on how customers and merchandise go collectively. Is product (mathbf{p}) a match for person (mathbf{u})? If that’s the case, we’ll suggest it.

Typically, that is completed by way of matrix factorization. See, for instance, this good article by the winners of the 2009 Netflix prize, introducing the why and the way of matrix factorization methods as utilized in collaborative filtering.

Right here’s the overall precept. Whereas different methods like non-negative matrix factorization could also be extra fashionable, this diagram of singular worth decomposition (SVD) discovered on Fb Analysis is especially instructive.

The diagram takes its instance from the context of textual content evaluation, assuming a co-occurrence matrix of hashtags and customers ((mathbf{A})). As said above, we’ll as an alternative work with a dataset of film rankings.

Had been we doing matrix factorization, we would wish to one way or the other deal with the truth that not each person has rated each film. As we’ll be utilizing embeddings as an alternative, we gained’t have that downside. For the sake of argumentation, although, let’s assume for a second the rankings have been a matrix, not a dataframe in tidy format.

In that case, (mathbf{A}) would retailer the rankings, with every row containing the rankings one person gave to all motion pictures.

This matrix then will get decomposed into three matrices:

  • (mathbf{Sigma}) shops the significance of the latent components governing the connection between customers and films.
  • (mathbf{U}) accommodates info on how customers rating on these latent components. It’s a illustration (embedding) of customers by the rankings they gave to the films.
  • (mathbf{V}) shops how motion pictures rating on these similar latent components. It’s a illustration (embedding) of flicks by how they acquired rated by stated customers.

As quickly as we now have a illustration of flicks in addition to customers in the identical latent area, we are able to decide their mutual match by a easy dot product (mathbf{m^ t}mathbf{u}). Assuming the person and film vectors have been normalized to size 1, that is equal to calculating the cosine similarity

[cos(theta) = frac{mathbf{x^ t}mathbf{y}}{mathbfspacemathbf}]

What does all this should do with embeddings?

Properly, the identical general ideas apply once we work with person resp. film embeddings, as an alternative of vectors obtained from matrix factorization. We’ll have one layer_embedding for customers, one layer_embedding for motion pictures, and a layer_lambda that calculates the dot product.

Right here’s a minimal customized mannequin that does precisely this:

simple_dot <- operate(embedding_dim,
                       n_users,
                       n_movies,
                       identify = "simple_dot") {
  
  keras_model_custom(identify = identify, operate(self) {
    self$user_embedding <-
      layer_embedding(
        input_dim = n_users + 1,
        output_dim = embedding_dim,
        embeddings_initializer = initializer_random_uniform(minval = 0, maxval = 0.05),
        identify = "user_embedding"
      )
    self$movie_embedding <-
      layer_embedding(
        input_dim = n_movies + 1,
        output_dim = embedding_dim,
        embeddings_initializer = initializer_random_uniform(minval = 0, maxval = 0.05),
        identify = "movie_embedding"
      )
    self$dot <-
      layer_lambda(
        f = operate(x) {
          k_batch_dot(x[[1]], x[[2]], axes = 2)
        }
      )
    
    operate(x, masks = NULL) {
      customers <- x[, 1]
      motion pictures <- x[, 2]
      user_embedding <- self$user_embedding(customers)
      movie_embedding <- self$movie_embedding(motion pictures)
      self$dot(checklist(user_embedding, movie_embedding))
    }
  })
}

We’re nonetheless lacking the info although! Let’s load it. In addition to the rankings themselves, we’ll additionally get the titles from motion pictures.csv.

data_dir <- "ml-latest-small"
motion pictures <- read_csv(file.path(data_dir, "motion pictures.csv"))
rankings <- read_csv(file.path(data_dir, "rankings.csv"))

Whereas person ids haven’t any gaps on this pattern, that’s completely different for film ids. We due to this fact convert them to consecutive numbers, so we are able to later specify an sufficient dimension for the lookup matrix.

dense_movies <- rankings %>% choose(movieId) %>% distinct() %>% rowid_to_column()
rankings <- rankings %>% inner_join(dense_movies) %>% rename(movieIdDense = rowid)
rankings <- rankings %>% inner_join(motion pictures) %>% choose(userId, movieIdDense, score, title, genres)

Let’s take a word, then, of what number of customers resp. motion pictures we now have.

n_movies <- rankings %>% choose(movieIdDense) %>% distinct() %>% nrow()
n_users <- rankings %>% choose(userId) %>% distinct() %>% nrow()

We’ll cut up off 20% of the info for validation. After coaching, in all probability all customers can have been seen by the community, whereas very possible, not all motion pictures can have occurred within the coaching pattern.

train_indices <- pattern(1:nrow(rankings), 0.8 * nrow(rankings))
train_ratings <- rankings[train_indices,]
valid_ratings <- rankings[-train_indices,]

x_train <- train_ratings %>% choose(c(userId, movieIdDense)) %>% as.matrix()
y_train <- train_ratings %>% choose(score) %>% as.matrix()
x_valid <- valid_ratings %>% choose(c(userId, movieIdDense)) %>% as.matrix()
y_valid <- valid_ratings %>% choose(score) %>% as.matrix()

Coaching a easy dot product mannequin

We’re prepared to start out the coaching course of. Be happy to experiment with completely different embedding dimensionalities.

embedding_dim <- 64

mannequin <- simple_dot(embedding_dim, n_users, n_movies)

mannequin %>% compile(
  loss = "mse",
  optimizer = "adam"
)

historical past <- mannequin %>% match(
  x_train,
  y_train,
  epochs = 10,
  batch_size = 32,
  validation_data = checklist(x_valid, y_valid),
  callbacks = checklist(callback_early_stopping(endurance = 2))
)

How properly does this work? Ultimate RMSE (the sq. root of the MSE loss we have been utilizing) on the validation set is round 1.08 , whereas fashionable benchmarks (e.g., of the LibRec recommender system) lie round 0.91. Additionally, we’re overfitting early. It seems like we’d like a barely extra subtle system.

Training curve for simple dot product model

Accounting for person and film biases

An issue with our technique is that we attribute the score as a complete to user-movie interplay. Nonetheless, some customers are intrinsically extra important, whereas others are usually extra lenient. Analogously, movies differ by common score. We hope to get higher predictions when factoring in these biases.

Conceptually, we then calculate a prediction like this:

[pred = avg + bias_m + bias_u + mathbf{m^ t}mathbf{u}]

The corresponding Keras mannequin will get simply barely extra advanced. Along with the person and film embeddings we’ve already been working with, the under mannequin embeds the common person and the common film in 1-d area. We then add each biases to the dot product encoding user-movie interplay. A sigmoid activation normalizes to a worth between 0 and 1, which then will get mapped again to the unique area.

Notice how on this mannequin, we additionally use dropout on the person and film embeddings (once more, one of the best dropout charge is open to experimentation).

max_rating <- rankings %>% summarise(max_rating = max(score)) %>% pull()
min_rating <- rankings %>% summarise(min_rating = min(score)) %>% pull()

dot_with_bias <- operate(embedding_dim,
                          n_users,
                          n_movies,
                          max_rating,
                          min_rating,
                          identify = "dot_with_bias"
                          ) {
  keras_model_custom(identify = identify, operate(self) {
    
    self$user_embedding <-
      layer_embedding(input_dim = n_users + 1,
                      output_dim = embedding_dim,
                      identify = "user_embedding")
    self$movie_embedding <-
      layer_embedding(input_dim = n_movies + 1,
                      output_dim = embedding_dim,
                      identify = "movie_embedding")
    self$user_bias <-
      layer_embedding(input_dim = n_users + 1,
                      output_dim = 1,
                      identify = "user_bias")
    self$movie_bias <-
      layer_embedding(input_dim = n_movies + 1,
                      output_dim = 1,
                      identify = "movie_bias")
    self$user_dropout <- layer_dropout(charge = 0.3)
    self$movie_dropout <- layer_dropout(charge = 0.6)
    self$dot <-
      layer_lambda(
        f = operate(x)
          k_batch_dot(x[[1]], x[[2]], axes = 2),
        identify = "dot"
      )
    self$dot_bias <-
      layer_lambda(
        f = operate(x)
          k_sigmoid(x[[1]] + x[[2]] + x[[3]]),
        identify = "dot_bias"
      )
    self$pred <- layer_lambda(
      f = operate(x)
        x * (self$max_rating - self$min_rating) + self$min_rating,
      identify = "pred"
    )
    self$max_rating <- max_rating
    self$min_rating <- min_rating
    
    operate(x, masks = NULL) {
      
      customers <- x[, 1]
      motion pictures <- x[, 2]
      user_embedding <-
        self$user_embedding(customers) %>% self$user_dropout()
      movie_embedding <-
        self$movie_embedding(motion pictures) %>% self$movie_dropout()
      dot <- self$dot(checklist(user_embedding, movie_embedding))
      dot_bias <-
        self$dot_bias(checklist(dot, self$user_bias(customers), self$movie_bias(motion pictures)))
      self$pred(dot_bias)
    }
  })
}

How properly does this mannequin carry out?

mannequin <- dot_with_bias(embedding_dim,
                       n_users,
                       n_movies,
                       max_rating,
                       min_rating)

mannequin %>% compile(
  loss = "mse",
  optimizer = "adam"
)

historical past <- mannequin %>% match(
  x_train,
  y_train,
  epochs = 10,
  batch_size = 32,
  validation_data = checklist(x_valid, y_valid),
  callbacks = checklist(callback_early_stopping(endurance = 2))
)

Not solely does it overfit later, it truly reaches a manner higher RMSE of 0.88 on the validation set!

Training curve for dot product model with biases

Spending a while on hyperparameter optimization may very properly result in even higher outcomes. As this publish focuses on the conceptual facet although, we wish to see what else we are able to do with these embeddings.

Embeddings: a better look

We will simply extract the embedding matrices from the respective layers. Let’s do that for motion pictures now.

movie_embeddings <- (mannequin %>% get_layer("movie_embedding") %>% get_weights())[[1]]

How are they distributed? Right here’s a heatmap of the primary 20 motion pictures. (Notice how we increment the row indices by 1, as a result of the very first row within the embedding matrix belongs to a film id 0 which doesn’t exist in our dataset.) We see that the embeddings look somewhat uniformly distributed between -0.5 and 0.5.

levelplot(
  t(movie_embeddings[2:21, 1:64]),
  xlab = "",
  ylab = "",
  scale = (checklist(draw = FALSE)))
Embeddings for first 20 movies

Naturally, we is likely to be curious about dimensionality discount, and see how particular motion pictures rating on the dominant components. A attainable solution to obtain that is PCA:

movie_pca <- movie_embeddings %>% prcomp(middle = FALSE)
elements <- movie_pca$x %>% as.knowledge.body() %>% rowid_to_column()

plot(movie_pca)
PCA: Variance explained by component

Let’s simply have a look at the primary principal part as the second already explains a lot much less variance.

Listed below are the ten motion pictures (out of all that have been rated no less than 20 occasions) that scored lowest on the primary issue:

ratings_with_pc12 <-
  rankings %>% inner_join(elements %>% choose(rowid, PC1, PC2),
                         by = c("movieIdDense" = "rowid"))

ratings_grouped <-
  ratings_with_pc12 %>%
  group_by(title) %>%
  summarize(
    PC1 = max(PC1),
    PC2 = max(PC2),
    score = imply(score),
    genres = max(genres),
    num_ratings = n()
  )

ratings_grouped %>% filter(num_ratings > 20) %>% organize(PC1) %>% print(n = 10)
# A tibble: 1,247 x 6
   title                                   PC1      PC2 score genres                   num_ratings
   <chr>                                 <dbl>    <dbl>  <dbl> <chr>                          <int>
 1 Starman (1984)                       -1.15  -0.400     3.45 Journey|Drama|Romance…          22
 2 Bulworth (1998)                      -0.820  0.218     3.29 Comedy|Drama|Romance              31
 3 Cable Man, The (1996)                -0.801 -0.00333   2.55 Comedy|Thriller                   59
 4 Species (1995)                       -0.772 -0.126     2.81 Horror|Sci-Fi                     55
 5 Save the Final Dance (2001)           -0.765  0.0302    3.36 Drama|Romance                     21
 6 Spanish Prisoner, The (1997)         -0.760  0.435     3.91 Crime|Drama|Thriller|Thr…          23
 7 Sgt. Bilko (1996)                    -0.757  0.249     2.76 Comedy                            29
 8 Bare Gun 2 1/2: The Scent of Concern,… -0.749  0.140     3.44 Comedy                            27
 9 Swordfish (2001)                     -0.694  0.328     2.92 Motion|Crime|Drama                33
10 Addams Household Values (1993)          -0.693  0.251     3.15 Kids|Comedy|Fantasy           73
# ... with 1,237 extra rows

And right here, inversely, are people who scored highest:

ratings_grouped %>% filter(num_ratings > 20) %>% organize(desc(PC1)) %>% print(n = 10)
 A tibble: 1,247 x 6
   title                                PC1        PC2 score genres                    num_ratings
   <chr>                              <dbl>      <dbl>  <dbl> <chr>                           <int>
 1 Graduate, The (1967)                1.41  0.0432      4.12 Comedy|Drama|Romance               89
 2 Vertigo (1958)                      1.38 -0.0000246   4.22 Drama|Thriller|Romance|Th…          69
 3 Breakfast at Tiffany's (1961)       1.28  0.278       3.59 Drama|Romance                      44
 4 Treasure of the Sierra Madre, The…  1.28 -0.496       4.3  Motion|Journey|Drama|W…          30
 5 Boot, Das (Boat, The) (1981)        1.26  0.238       4.17 Motion|Drama|Conflict                   51
 6 Flintstones, The (1994)             1.18  0.762       2.21 Kids|Comedy|Fantasy            39
 7 Rock, The (1996)                    1.17 -0.269       3.74 Motion|Journey|Thriller         135
 8 Within the Warmth of the Night time (1967)     1.15 -0.110       3.91 Drama|Thriller                      22
 9 Quiz Present (1994)                    1.14 -0.166       3.75 Drama                              90
10 Striptease (1996)                   1.14 -0.681       2.46 Comedy|Crime                       39
# ... with 1,237 extra rows

We’ll depart it to the educated reader to call these components, and proceed to our second matter: How does an embedding layer do what it does?

Do-it-yourself embeddings

You could have heard individuals say all an embedding layer did was only a lookup. Think about you had a dataset that, along with steady variables like temperature or barometric stress, contained a categorical column characterization consisting of tags like “foggy” or “cloudy.” Say characterization had 7 attainable values, encoded as an element with ranges 1-7.

Had been we going to feed this variable to a non-embedding layer, layer_dense say, we’d should take care that these numbers don’t get taken for integers, thus falsely implying an interval (or no less than ordered) scale. However once we use an embedding as the primary layer in a Keras mannequin, we feed in integers on a regular basis! For instance, in textual content classification, a sentence may get encoded as a vector padded with zeroes, like this:

2  77   4   5 122   55  1  3   0   0  

The factor that makes this work is that the embedding layer truly does carry out a lookup. Under, you’ll discover a quite simple customized layer that does primarily the identical factor as Keras’ layer_embedding:

  • It has a weight matrix self$embeddings that maps from an enter area (motion pictures, say) to the output area of latent components (embeddings).
  • Once we name the layer, as in

x <- k_gather(self$embeddings, x)

it seems up the passed-in row quantity within the weight matrix, thus retrieving an merchandise’s distributed illustration from the matrix.

SimpleEmbedding <- R6::R6Class(
  "SimpleEmbedding",
  
  inherit = KerasLayer,
  
  public = checklist(
    output_dim = NULL,
    emb_input_dim = NULL,
    embeddings = NULL,
    
    initialize = operate(emb_input_dim, output_dim) {
      self$emb_input_dim <- emb_input_dim
      self$output_dim <- output_dim
    },
    
    construct = operate(input_shape) {
      self$embeddings <- self$add_weight(
        identify = 'embeddings',
        form = checklist(self$emb_input_dim, self$output_dim),
        initializer = initializer_random_uniform(),
        trainable = TRUE
      )
    },
    
    name = operate(x, masks = NULL) {
      x <- k_cast(x, "int32")
      k_gather(self$embeddings, x)
    },
    
    compute_output_shape = operate(input_shape) {
      checklist(self$output_dim)
    }
  )
)

As standard with customized layers, we nonetheless want a wrapper that takes care of instantiation.

layer_simple_embedding <-
  operate(object,
           emb_input_dim,
           output_dim,
           identify = NULL,
           trainable = TRUE) {
    create_layer(
      SimpleEmbedding,
      object,
      checklist(
        emb_input_dim = as.integer(emb_input_dim),
        output_dim = as.integer(output_dim),
        identify = identify,
        trainable = trainable
      )
    )
  }

Does this work? Let’s check it on the rankings prediction process! We’ll simply substitute the customized layer within the easy dot product mannequin we began out with, and test if we get out the same RMSE.

Placing the customized embedding layer to check

Right here’s the easy dot product mannequin once more, this time utilizing our customized embedding layer.

simple_dot2 <- operate(embedding_dim,
                       n_users,
                       n_movies,
                       identify = "simple_dot2") {
  
  keras_model_custom(identify = identify, operate(self) {
    self$embedding_dim <- embedding_dim
    
    self$user_embedding <-
      layer_simple_embedding(
        emb_input_dim = checklist(n_users + 1),
        output_dim = embedding_dim,
        identify = "user_embedding"
      )
    self$movie_embedding <-
      layer_simple_embedding(
        emb_input_dim = checklist(n_movies + 1),
        output_dim = embedding_dim,
        identify = "movie_embedding"
      )
    self$dot <-
      layer_lambda(
        output_shape = self$embedding_dim,
        f = operate(x) {
          k_batch_dot(x[[1]], x[[2]], axes = 2)
        }
      )
    
    operate(x, masks = NULL) {
      customers <- x[, 1]
      motion pictures <- x[, 2]
      user_embedding <- self$user_embedding(customers)
      movie_embedding <- self$movie_embedding(motion pictures)
      self$dot(checklist(user_embedding, movie_embedding))
    }
  })
}

mannequin <- simple_dot2(embedding_dim, n_users, n_movies)

mannequin %>% compile(
  loss = "mse",
  optimizer = "adam"
)

historical past <- mannequin %>% match(
  x_train,
  y_train,
  epochs = 10,
  batch_size = 32,
  validation_data = checklist(x_valid, y_valid),
  callbacks = checklist(callback_early_stopping(endurance = 2))
)

We find yourself with a RMSE of 1.13 on the validation set, which isn’t removed from the 1.08 we obtained when utilizing layer_embedding. A minimum of, this could inform us that we efficiently reproduced the method.

Conclusion

Our targets on this publish have been twofold: Shed some gentle on how an embedding layer might be applied, and present how embeddings calculated by a neural community can be utilized as an alternative choice to part matrices obtained from matrix decomposition. After all, this isn’t the one factor that’s fascinating about embeddings!

For instance, a really sensible query is how a lot precise predictions might be improved through the use of embeddings as an alternative of one-hot vectors; one other is how realized embeddings may differ relying on what process they have been educated on. Final not least – how do latent components realized by way of embeddings differ from these realized by an autoencoder?

In that spirit, there isn’t a lack of matters for exploration and poking round …

Ahmed, N. Okay., R. Rossi, J. Boaz Lee, T. L. Willke, R. Zhou, X. Kong, and H. Eldardiry. 2018. “Studying Function-Based mostly Graph Embeddings.” ArXiv e-Prints, February. https://arxiv.org/abs/1802.02896.
Alon, Uri, Meital Zilberstein, Omer Levy, and Eran Yahav. 2018. “Code2vec: Studying Distributed Representations of Code.” CoRR abs/1803.09473. http://arxiv.org/abs/1803.09473.

Frome, Andrea, Gregory S. Corrado, Jonathon Shlens, Samy Bengio, Jeffrey Dean, Marc’Aurelio Ranzato, and Tomas Mikolov. 2013. “DeViSE: A Deep Visible-Semantic Embedding Mannequin.” In NIPS, 2121–29.

Hallac, D., S. Bhooshan, M. Chen, Okay. Abida, R. Sosic, and J. Leskovec. 2018. “Drive2Vec: Multiscale State-Area Embedding of Vehicular Sensor Information.” ArXiv e-Prints, June. https://arxiv.org/abs/1806.04795.
Jean, Neal, Sherrie Wang, Anshul Samar, George Azzari, David B. Lobell, and Stefano Ermon. 2018. “Tile2Vec: Unsupervised Illustration Studying for Spatially Distributed Information.” CoRR abs/1805.02855. http://arxiv.org/abs/1805.02855.
Krstovski, Okay., and D. M. Blei. 2018. “Equation Embeddings.” ArXiv e-Prints, March. https://arxiv.org/abs/1803.09123.

Rumelhart, David E., James L. McClelland, and CORPORATE PDP Analysis Group, eds. 1986. Parallel Distributed Processing: Explorations within the Microstructure of Cognition, Vol. 2: Psychological and Organic Fashions. Cambridge, MA, USA: MIT Press.

Zohra Smaili, F., X. Gao, and R. Hoehndorf. 2018. “Onto2Vec: Joint Vector-Based mostly Illustration of Organic Entities and Their Ontology-Based mostly Annotations.” ArXiv e-Prints, January. https://arxiv.org/abs/1802.00864.