embed-king, picking the embedding model by evidence
A benchmark that compares embedding models on our own matching task by accuracy, cost and speed, and watches for better ones every week.
I built it on my own, including the benchmark, the reports and the weekly watcher.
- about 20embedding models compared
About this number
Distinct embedding models benchmarked on our matching task (25 configurations counting variants). As of Oct 2026.
- 3kinds of model
About this number
Paid APIs, open models on our own GPU, and simple text matching. As of Oct 2026.
- Weeklyleaderboard check for new models
Why it mattered
Our matching ran on a paid embedding API at large volume. New open models appear almost every week. The question was whether one could match the paid model’s accuracy for less, and how we would know when that changed. Public leaderboards rank models on general tasks, so we had to measure them on our own task.
What it does
- The task is matching raw free text to the right entry in a fixed catalog. School names were our first task.
- Paid APIs and open models on our own GPU go through the same scoring, and so does simple text matching. The score counts how often the right entry comes first, and it also covers running cost and speed.
- I wrote down what would count as a win before seeing any results.
- Every week, an agent checks public embedding leaderboards and alerts me when a promising new model appears. One command then benchmarks it on our task and flags it if it beats the current cost or speed trade-off.
Result
An open model on our own GPU matched the paid reference’s accuracy for a fraction of the running cost; we confirmed it on fresh data before switching.
The trade-off, model by model
- Paid API
- Open model on our GPU
- Tried after the rules were set
Reference: text-embedding-3-large. Picked: F2LLM-v2-14B, int8. Most accurate: gemini-embedding-2.
Each mark is one model setup on our task (some models appear in more than one setup), against the reference (text-embedding-3-large: no gap, 1× cost and speed). Open models are costed at electricity only, on a GPU we already own. Simple text matching scored well below every embedding model and is left out.
| Model | Gap | Cost | Speed |
|---|---|---|---|
| gemini-embedding-2Paid API, most accurate | +0.7 | 2.9× | 0.28× |
| text-embedding-3-largePaid API, reference | 0 | 1× | 1× |
| text-embedding-3-large, 1024-dPaid API | −0.1 | 1× | 1× |
| harrier-oss-v1-27b, nf4Open model | −0.2 | 0.69× | 0.066× |
| F2LLM-v2-14B, int8Open model, picked | −0.3 | 0.21× | 0.17× |
| F2LLM-v2-14BOpen model | −0.3 | 0.29× | 0.16× |
| harrier-oss-v1-27b, nf4, alt. promptOpen model, tried after the rules were set | −0.3 | 0.71× | 0.064× |
| KaLM-Embedding-Gemma3-12BOpen model, tried after the rules were set | −0.4 | 0.18× | 0.22× |
| F2LLM-v2-8BOpen model | −0.7 | 0.15× | 0.31× |
| harrier-oss-v1-27b, int8Open model | −0.7 | 0.67× | 0.038× |
| harrier-oss-v1-27b, int8, alt. promptOpen model, tried after the rules were set | −0.7 | 0.68× | 0.037× |
| gemini-embedding-001Paid API | −0.7 | 1.2× | 0.24× |
| Giga-Embeddings-instruct-10BOpen model | −1.7 | 0.075× | 0.26× |
| F2LLM-v2-4BOpen model | −2.5 | 0.066× | 0.67× |
| Giga-Embeddings-instruct-3BOpen model, tried after the rules were set | −2.6 | 0.074× | 0.58× |
| embeddinggemma-2Open model, tried after the rules were set | −2.7 | 0.0065× | 4.9× |
| F2LLM-v2-1.7BOpen model | −3.2 | 0.04× | 1.1× |
| Octen-Embedding-8BOpen model | −3.2 | 0.15× | 0.31× |
| BidirLM-Omni-2.5B-EmbeddingOpen model, tried after the rules were set | −3.3 | 0.044× | 0.91× |
| Qwen3-Embedding-8BOpen model | −3.4 | 0.11× | 0.4× |
| Qwen3-Embedding-8B, hosted APIPaid API | −3.6 | 0.17× | 0.2× |
| Qwen3-Embedding-4BOpen model | −5.1 | 0.068× | 0.65× |
| harrier-oss-v1-0.6b, alt. promptOpen model, tried after the rules were set | −6.5 | 0.018× | 2.2× |
| harrier-oss-v1-0.6bOpen model | −7.6 | 0.018× | 2.4× |
| harrier-oss-v1-270mOpen model, tried after the rules were set | −8.1 | 0.004× | 7.6× |
What I got wrong
- What I assumed going in
- That the top of a public leaderboard would be the best model for our task.
- What I'd measure first next time
- How much of the input has no right answer in the catalog at all, before scoring any model.
- What it can't do yet
- Tell that a piece of text has no match in the catalog as reliably as it finds the right one.