Embedding

What is an Embedding?

An embedding is a way to turn words, sentences, or even images into lists of numbers (called vectors) so that a computer can understand their meaning. Imagine a giant map where every concept has a specific location. Words with similar meanings, like "happy" and "joyful," are placed close together, while unrelated words like "happy" and "volcano" are far apart.

For example, the word "king" might be represented as a list of 768 numbers. These numbers capture various aspects of meaning, such as royalty, gender, and power.

How Does It Work?

During training, a model learns to assign vectors to words based on how they appear in context across billions of sentences. Words that frequently appear in similar contexts end up with similar vectors. This is the core idea behind representation learning: the model discovers useful representations of data on its own, without being explicitly told what each dimension means.

Modern embedding models can encode entire sentences or paragraphs into a single vector, capturing the overall meaning rather than just individual words.

Why Does It Matter?

Embeddings are the backbone of search engines, recommendation systems, and Retrieval-Augmented Generation (RAG). By comparing the distance between vectors, applications can find semantically similar content, even when the exact words differ. This enables powerful features like semantic search, where searching for "how to fix a flat tire" also returns results about "changing a punctured wheel."