Embeddings Explained: A Practical Guide for Developers

I spent weeks trying to understand embeddings from academic papers. Dense mathematical notation. Proof-heavy explanations. Formulas that made my eyes glaze over.

Then someone said: "It is just coordinates for meaning."

Everything clicked. Here is the explanation I wish someone had given me.

Coordinates for Meaning

You know how GPS coordinates represent physical locations? 48.8566, 2.3522 is Paris. 40.7128, -74.0060 is New York. Close coordinates mean close locations.

Embeddings do the same thing for meaning. But instead of two dimensions (latitude, longitude), they use hundreds or thousands of dimensions. Each dimension captures some aspect of meaning that the model learned during training.

"Dog" gets coordinates. "Puppy" gets nearby coordinates. "Cat" gets somewhat nearby coordinates (both animals). "Bankruptcy" gets far-away coordinates (different concept entirely).

The distance between coordinates reflects the semantic distance between concepts. Close coordinates mean similar meanings. Far coordinates mean different meanings. This is literally all embeddings are.

How They Get Created

A neural network processes your text and outputs a fixed-size vector. The model was trained on massive text datasets where it learned that certain words and phrases tend to appear in similar contexts. Words with similar contexts get similar vectors.

You do not train this model. You use a pre-trained one. OpenAI's text-embedding-3-small. Cohere's embed models. Open-source models from Hugging Face. You send text in, you get a vector out.

The vector is an array of floating-point numbers. Something like [0.0234, -0.0891, 0.0456, ...] repeated 1536 times for a 1536-dimensional model. Each number is a coordinate in the meaning space.

You do not need to understand what each dimension represents. Nobody does, including the people who trained the model. The dimensions are abstract learned features. What matters is that the resulting vectors preserve semantic relationships.

The King - Man + Woman = Queen Thing

This is the example everyone uses. And it genuinely demonstrates something profound.

Take the vector for "king." Subtract the vector for "man." Add the vector for "woman." The resulting vector is closest to "queen" in the embedding space.

What this means: the relationship between "king" and "man" is captured in vector space. The model has learned a "gender" direction in the embedding space. Subtracting "man" and adding "woman" moves along this direction, transforming "king" into "queen."

For developers, the practical implication is: vector arithmetic on embeddings produces meaningful results. You can search for analogies, find opposites, identify relationships, and cluster concepts using basic math operations on vectors.

Text Embeddings in Practice

The practical applications are more exciting than the theory.

Semantic search: A user searches for "how to fix a slow website." You embed this query. You find the stored document embeddings closest to the query embedding. The top results include documents titled "Web Performance Optimization," "Speed Up Your Site," and "Reducing Page Load Times." None of these contain the word "fix" or "slow." They match by meaning.

Content recommendation: You have a user who read three articles about machine learning. Embed those articles. Find other articles with similar embeddings. Recommend them. The recommendations are semantically relevant, not just keyword-matched. An article about "statistical learning methods" gets recommended even though it never uses the phrase "machine learning."

Classification: You have 10 labeled categories. Embed a description of each category. When a new item arrives, embed it and find the closest category embedding. Zero-shot classification without training a classifier. Works surprisingly well for many use cases.

Duplicate detection: Embed all your content. Find pairs with very high similarity scores. These are potential duplicates even if they use completely different wording. "How to reset my password" and "I forgot my login credentials, need to change them" have high similarity despite sharing almost no words.

Choosing an Embedding Model

This decision matters more than most people realize. Different models produce different quality embeddings for different domains.

OpenAI text-embedding-3-small is the practical default. Good quality. Reasonable cost. 1536 dimensions. Handles general English text well. If you do not have a specific reason to use something else, start here.

OpenAI text-embedding-3-large produces higher quality embeddings in 3072 dimensions. More expensive. More compute for similarity search. Use it when text-embedding-3-small is not accurate enough for your use case.

Open-source models from Hugging Face (like BGE, E5, or GTE) run locally. No API costs. Full data privacy. Variable quality depending on the model and your domain. Good for teams with strong privacy requirements or high volume that makes API costs prohibitive.

Domain-specific models exist for medical text, legal text, scientific papers, and code. If your content is heavily specialized, a domain-specific model often outperforms general-purpose ones significantly.

Always benchmark on your data. Take 100 representative queries with known relevant documents. Run them through different embedding models. Measure which model ranks the relevant documents highest. This takes an afternoon and prevents months of working with the wrong model.

Practical Implementation

The implementation workflow is straightforward.

Step one: Choose your embedding model and generate embeddings for your content. This is a batch operation. Run it once for existing content and incrementally for new content.

Step two: Store embeddings in a vector database. Pinecone, Weaviate, Chroma, Qdrant, or pgvector. The choice depends on your scale and operational preferences.

Step three: At query time, embed the user's query with the same model. Search the vector database for the nearest embeddings. Return the associated content.

Step four: Measure and iterate. Track which results users click. Track which queries return no good results. Use this data to improve your chunking strategy, switch embedding models, or add hybrid search.

The entire pipeline from zero to production semantic search can be built in a day. Getting it production-quality takes a few weeks of iteration. But the baseline is surprisingly capable from day one.

Gotchas

Embedding models have maximum input lengths. Exceed them and the input is truncated. Chunk your content to fit within the model's limits.

Embeddings from different models are not compatible. You cannot search a Cohere embedding database with an OpenAI query embedding. Pick a model and stick with it, or re-embed everything when you switch.

Embedding quality degrades for content very different from the training data. A model trained on English web text produces poor embeddings for medical Latin or ancient Greek. Domain matters.

Cosine similarity scores are not probabilities. A score of 0.85 does not mean "85% relevant." Scores are relative within a search, not absolute measures of relevance. Calibrate your similarity thresholds empirically based on what scores correspond to genuinely relevant results in your data.

Start simple. Complexity can come later. The basics work better than most people expect.