Cosine similarity

Similarity

A commonly used measure of similarity for Continuous embedding vectors. $$\cos(\vec u , \vec v) = \frac{\vec u \cdot \vec v}{\lVert \vec u \rVert \lVert \vec v \rVert}$$ The main justification for this is that Euclidean distance does not work well in high-dimensional space (read more here).

Issues

• Zhou2022problems

Cosine distance vs. angular distance

Cosine similarity is often converted into “distance” (from $\left[ -1,1 \right]$ to $\left[ 0,2 \right]$): $$d(\vec u, \vec v) = 1 - \cos(\vec u, \vec v).$$ However, this may not be a good measure of distance. - Because the cosine function’s derivative is 0 and decreasing around 0, the cosine distance does not have a good “resolution” around 0, which is often the most important domain of interest. - See Michael Trosset’s note (Sec. 1.3 and 2.7)

Alternatively, we can use the angle as the distance: $$\arccos(\vec u , \vec v)$$ Google’s Universal Sentence Encoding paper uses the following similarity: $$\text{sim}(\vec u, \vec v) = 1 - \arccos\Bigg(\frac{\vec u \cdot \vec v}{\lVert \vec u \rVert \lVert \vec v \rVert}\Bigg)/\pi$$