skipgram: Continuous skip-gram

#Word2vec

We use the following quote by Ford in Westworld as an example.

I read a theory once that the human intellect is like peacock feathers. Just an extravagant display intended to attract a mate, just an elaborate mating ritual. But, of course, the peacock can barely fly. It lives in the dirt, pecking insects out of the muck, consoling itself with its great beauty.

The word intended is surrunded by extravagant display in the front and to attract after it. The task is to predict the probability of words around the middle word intended, which are the ‘history words’ extravagant, display and ‘future words’ to, attract in our case. [mikolov2013]

For this center word intended, we generate the following data.

Input (Center Word)Output (Context)
intendedextravagant
intendeddisplay
intendedto
intendedattract

We will build the following dataset using the sentence.

Input (Center Word)Output (Context)
Iread
Ia
readI
reada
readtheory
aI
aread
atheory
aonce
theoryread
theorya
theoryonce

It is not required to use two in the front and two words after the middle word. The number of words to choose is a hyperparameter to be decided.

Published: by ;

Table of Contents

Current Ref:

  • cards/machine-learning/embedding/continuous-skip-gram.md