Aditya Rathi

Production system, 2026

embed-king, picking the embedding model by evidence

A benchmark that compares embedding models on our own matching task by accuracy, cost and speed, and watches for better ones every week.

I built it on my own, including the benchmark, the reports and the weekly watcher.

  • about 20embedding models compared
    About this number

    Distinct embedding models benchmarked on our matching task (25 configurations counting variants). As of Oct 2026.

  • 3kinds of model
    About this number

    Paid APIs, open models on our own GPU, and simple text matching. As of Oct 2026.

  • Weeklyleaderboard check for new models

Why it mattered

Our matching ran on a paid embedding API at large volume. New open models appear almost every week. The question was whether one could match the paid model’s accuracy for less, and how we would know when that changed. Public leaderboards rank models on general tasks, so we had to measure them on our own task.

What it does

  1. The task is matching raw free text to the right entry in a fixed catalog. School names were our first task.
  2. Paid APIs and open models on our own GPU go through the same scoring, and so does simple text matching. The score counts how often the right entry comes first, and it also covers running cost and speed.
  3. I wrote down what would count as a win before seeing any results.
  4. Every week, an agent checks public embedding leaderboards and alerts me when a promising new model appears. One command then benchmarks it on our task and flags it if it beats the current cost or speed trade-off.

Result

An open model on our own GPU matched the paid reference’s accuracy for a fraction of the running cost; we confirmed it on fresh data before switching.

The trade-off, model by model

  • Paid API
  • Open model on our GPU
  • Tried after the rules were set
Accuracy against running costAccuracy gap to text-embedding-3-large in points against running cost, relative to the reference, on a log scale; left is cheaper, up is more accurate. Every value is in the table below.−10−8−6−4−20+20.001×0.01×0.1×1×10×running cost, relative to the reference (left is cheaper)accuracy gap to the reference, points (up is more accurate)gemini-embedding-2 (most accurate): accuracy gap +0.7 points, cost 2.9×, speed 0.28×text-embedding-3-large (reference): accuracy gap 0 points, cost 1×, speed 1×text-embedding-3-large, 1024-d: accuracy gap −0.1 points, cost 1×, speed 1×harrier-oss-v1-27b, nf4: accuracy gap −0.2 points, cost 0.69×, speed 0.066×F2LLM-v2-14B, int8 (picked): accuracy gap −0.3 points, cost 0.21×, speed 0.17×F2LLM-v2-14B: accuracy gap −0.3 points, cost 0.29×, speed 0.16×harrier-oss-v1-27b, nf4, alt. prompt: accuracy gap −0.3 points, cost 0.71×, speed 0.064×, tried after the rules were setKaLM-Embedding-Gemma3-12B: accuracy gap −0.4 points, cost 0.18×, speed 0.22×, tried after the rules were setF2LLM-v2-8B: accuracy gap −0.7 points, cost 0.15×, speed 0.31×harrier-oss-v1-27b, int8: accuracy gap −0.7 points, cost 0.67×, speed 0.038×harrier-oss-v1-27b, int8, alt. prompt: accuracy gap −0.7 points, cost 0.68×, speed 0.037×, tried after the rules were setgemini-embedding-001: accuracy gap −0.7 points, cost 1.2×, speed 0.24×Giga-Embeddings-instruct-10B: accuracy gap −1.7 points, cost 0.075×, speed 0.26×F2LLM-v2-4B: accuracy gap −2.5 points, cost 0.066×, speed 0.67×Giga-Embeddings-instruct-3B: accuracy gap −2.6 points, cost 0.074×, speed 0.58×, tried after the rules were setembeddinggemma-2: accuracy gap −2.7 points, cost 0.0065×, speed 4.9×, tried after the rules were setF2LLM-v2-1.7B: accuracy gap −3.2 points, cost 0.04×, speed 1.1×Octen-Embedding-8B: accuracy gap −3.2 points, cost 0.15×, speed 0.31×BidirLM-Omni-2.5B-Embedding: accuracy gap −3.3 points, cost 0.044×, speed 0.91×, tried after the rules were setQwen3-Embedding-8B: accuracy gap −3.4 points, cost 0.11×, speed 0.4×Qwen3-Embedding-8B, hosted API: accuracy gap −3.6 points, cost 0.17×, speed 0.2×Qwen3-Embedding-4B: accuracy gap −5.1 points, cost 0.068×, speed 0.65×harrier-oss-v1-0.6b, alt. prompt: accuracy gap −6.5 points, cost 0.018×, speed 2.2×, tried after the rules were setharrier-oss-v1-0.6b: accuracy gap −7.6 points, cost 0.018×, speed 2.4×harrier-oss-v1-270m: accuracy gap −8.1 points, cost 0.004×, speed 7.6×, tried after the rules were setmost accuratereferencepicked
Accuracy against running cost, log scale.
Accuracy against speedAccuracy gap to text-embedding-3-large in points against speed, relative to the reference, on a log scale; right is faster, up is more accurate. Every value is in the table below.−10−8−6−4−20+20.01×0.1×1×10×speed, relative to the reference (right is faster)accuracy gap to the reference, points (up is more accurate)gemini-embedding-2 (most accurate): accuracy gap +0.7 points, cost 2.9×, speed 0.28×text-embedding-3-large (reference): accuracy gap 0 points, cost 1×, speed 1×text-embedding-3-large, 1024-d: accuracy gap −0.1 points, cost 1×, speed 1×harrier-oss-v1-27b, nf4: accuracy gap −0.2 points, cost 0.69×, speed 0.066×F2LLM-v2-14B, int8 (picked): accuracy gap −0.3 points, cost 0.21×, speed 0.17×F2LLM-v2-14B: accuracy gap −0.3 points, cost 0.29×, speed 0.16×harrier-oss-v1-27b, nf4, alt. prompt: accuracy gap −0.3 points, cost 0.71×, speed 0.064×, tried after the rules were setKaLM-Embedding-Gemma3-12B: accuracy gap −0.4 points, cost 0.18×, speed 0.22×, tried after the rules were setF2LLM-v2-8B: accuracy gap −0.7 points, cost 0.15×, speed 0.31×harrier-oss-v1-27b, int8: accuracy gap −0.7 points, cost 0.67×, speed 0.038×harrier-oss-v1-27b, int8, alt. prompt: accuracy gap −0.7 points, cost 0.68×, speed 0.037×, tried after the rules were setgemini-embedding-001: accuracy gap −0.7 points, cost 1.2×, speed 0.24×Giga-Embeddings-instruct-10B: accuracy gap −1.7 points, cost 0.075×, speed 0.26×F2LLM-v2-4B: accuracy gap −2.5 points, cost 0.066×, speed 0.67×Giga-Embeddings-instruct-3B: accuracy gap −2.6 points, cost 0.074×, speed 0.58×, tried after the rules were setembeddinggemma-2: accuracy gap −2.7 points, cost 0.0065×, speed 4.9×, tried after the rules were setF2LLM-v2-1.7B: accuracy gap −3.2 points, cost 0.04×, speed 1.1×Octen-Embedding-8B: accuracy gap −3.2 points, cost 0.15×, speed 0.31×BidirLM-Omni-2.5B-Embedding: accuracy gap −3.3 points, cost 0.044×, speed 0.91×, tried after the rules were setQwen3-Embedding-8B: accuracy gap −3.4 points, cost 0.11×, speed 0.4×Qwen3-Embedding-8B, hosted API: accuracy gap −3.6 points, cost 0.17×, speed 0.2×Qwen3-Embedding-4B: accuracy gap −5.1 points, cost 0.068×, speed 0.65×harrier-oss-v1-0.6b, alt. prompt: accuracy gap −6.5 points, cost 0.018×, speed 2.2×, tried after the rules were setharrier-oss-v1-0.6b: accuracy gap −7.6 points, cost 0.018×, speed 2.4×harrier-oss-v1-270m: accuracy gap −8.1 points, cost 0.004×, speed 7.6×, tried after the rules were setmost accuratereferencepicked
Accuracy against speed, log scale.

Reference: text-embedding-3-large. Picked: F2LLM-v2-14B, int8. Most accurate: gemini-embedding-2.

Each mark is one model setup on our task (some models appear in more than one setup), against the reference (text-embedding-3-large: no gap, 1× cost and speed). Open models are costed at electricity only, on a GPU we already own. Simple text matching scored well below every embedding model and is left out.

Every model, relative to the reference (gap in accuracy points)
ModelGapCostSpeed
gemini-embedding-2Paid API, most accurate+0.72.9×0.28×
text-embedding-3-largePaid API, reference01×1×
text-embedding-3-large, 1024-dPaid API−0.11×1×
harrier-oss-v1-27b, nf4Open model−0.20.69×0.066×
F2LLM-v2-14B, int8Open model, picked−0.30.21×0.17×
F2LLM-v2-14BOpen model−0.30.29×0.16×
harrier-oss-v1-27b, nf4, alt. promptOpen model, tried after the rules were set−0.30.71×0.064×
KaLM-Embedding-Gemma3-12BOpen model, tried after the rules were set−0.40.18×0.22×
F2LLM-v2-8BOpen model−0.70.15×0.31×
harrier-oss-v1-27b, int8Open model−0.70.67×0.038×
harrier-oss-v1-27b, int8, alt. promptOpen model, tried after the rules were set−0.70.68×0.037×
gemini-embedding-001Paid API−0.71.2×0.24×
Giga-Embeddings-instruct-10BOpen model−1.70.075×0.26×
F2LLM-v2-4BOpen model−2.50.066×0.67×
Giga-Embeddings-instruct-3BOpen model, tried after the rules were set−2.60.074×0.58×
embeddinggemma-2Open model, tried after the rules were set−2.70.0065×4.9×
F2LLM-v2-1.7BOpen model−3.20.04×1.1×
Octen-Embedding-8BOpen model−3.20.15×0.31×
BidirLM-Omni-2.5B-EmbeddingOpen model, tried after the rules were set−3.30.044×0.91×
Qwen3-Embedding-8BOpen model−3.40.11×0.4×
Qwen3-Embedding-8B, hosted APIPaid API−3.60.17×0.2×
Qwen3-Embedding-4BOpen model−5.10.068×0.65×
harrier-oss-v1-0.6b, alt. promptOpen model, tried after the rules were set−6.50.018×2.2×
harrier-oss-v1-0.6bOpen model−7.60.018×2.4×
harrier-oss-v1-270mOpen model, tried after the rules were set−8.10.004×7.6×

What I got wrong

What I assumed going in
That the top of a public leaderboard would be the best model for our task.
What I'd measure first next time
How much of the input has no right answer in the catalog at all, before scoring any model.
What it can't do yet
Tell that a piece of text has no match in the catalog as reliably as it finds the right one.