MaxSim: How Do We Compare Two Sets of Vectors?
1. Introduction
Some search models keep a separate vector for every token in a document, instead of squashing the whole thing into one vector. That's great for accuracy. You get a much richer, more detailed picture of what the document actually means.
But it also creates a very practical problem. If a query has, say, 5 vectors and a document has 200 vectors, how do you turn all of that into one single number that tells you "this document matches this query well" or "it doesn't"?
With a normal single vector search, this is easy. You just take the cosine similarity or the dot product between two vectors and you're done. But once you have a whole bunch of vectors on both sides, that simple trick doesn't work anymore. You need some way to combine all those individual comparisons into one meaningful score.
That's exactly what MaxSim was built to do.
2. What MaxSim actually does
MaxSim stands for "maximum similarity," and honestly the idea behind it is simpler than it sounds. Here's the basic logic:
- Take one token from the query.
- Compare it against every single token in the document.
- Keep only the best match, the highest similarity score you found.
- Do this for every token in the query.
- Add up all of those "best match" scores.
That final sum is the MaxSim score. In plain terms, every word in your query gets to go find its closest friend in the document, and then we add up how good all those friendships are.
If you want the more formal version of that same idea, it's usually written like this:
It looks a bit dense at first, but every symbol here maps directly to the plain english version above. Let's break it down piece by piece:
- Q represents the query token vectors
- D represents the document token vectors
- sim(qi, dj) is a base similarity function that measures the distance between two vectors
The sim(qi, dj) function can be any distance metric - dot product, cosine similarity, Euclidean distance, or others. Let’s break down what this formula actually computes.
So really, the formula is just two familiar ideas stacked on top of each other: an ordinary similarity function on the inside, wrapped in a max and a sum that let it work across two whole sequences of vectors instead of just two single ones.
You might wonder why we take the maximum instead of, say, averaging every similarity score together. Here's the thing: relevance in a document is rarely spread out evenly. Imagine a long article that spends one paragraph talking about exactly what you searched for, and the rest of it wandering off into loosely related topics. If we averaged everything, that one great paragraph would get diluted by all the mediocre ones around it. Taking the maximum lets the strongest match speak for itself, instead of getting watered down by the parts of the document that aren't relevant.
There's another nice side effect of matching at the token level. Because each token carries some sense of the words around it, the same word can match differently depending on context. A query token can end up matching a completely different word in the document, as long as the two carry similar meaning in context. That's exactly the kind of flexibility a single, one-size-fits-all vector would struggle to capture.
3. Let's walk through a real example
Numbers make more sense with an example, so let's use one.
Query: "doctor appointment" (2 tokens)
Document: "Please schedule a visit with your physician" (6 tokens)
First, take the query token "doctor." It looks at every word in the document and asks, "which one of you is most similar to me?" It matches best with "physician," even though the two words share no letters at all, scoring a similarity of 0.91.
Now take the second query token, "appointment." It does the same search across the document. Its best match turns out to be "schedule," again a different word entirely, but close enough in meaning to score a similarity of 0.85.
Now we just add the two best scores together.
That 1.76 is the final MaxSim score for this query and document pair. Higher scores mean a stronger match.
This picture is basically what's happening under the hood. Query tokens sit on one side, document tokens on the other. We check every pair, pick the best match per query token, and sum those up. That sum is the final score.
Now here's where it gets a little tricky, and where the real challenge shows up.
4. The tricky part: MaxSim isn't symmetric
Here's something that trips a lot of people up the first time they see it. You'd expect that comparing a query to a document should give the same result as comparing that document back to the query. It doesn't.
Think about why. In our example, the query only had 2 tokens, but the document had 6. When we run MaxSim the normal way, we're doing 2 "find the best match" operations, one for each query token. If we flip it around and treat the document as the "query" instead, we'd suddenly be doing 6 "find the best match" operations instead of 2. That's a completely different calculation with a completely different answer.
Look at how different this picture is from the one before. Now every document token is searching for its best match among just 2 query tokens, and we end up summing 6 numbers instead of 2. That's not a small rounding difference, it's a genuinely different computation with its own result. This is a nice, concrete way to see why direction matters so much here.
5. Why this breaks a lot of search systems
You might be wondering why this asymmetry actually matters in practice. Here's the catch: a lot of fast vector search systems, like HNSW, are built assuming distance works both ways equally. In other words, they assume the distance from point A to point B is exactly the same as the distance from B to A.
But actually, there's a more basic mismatch going on here, even before we get to the symmetry issue. HNSW was designed around a very specific idea: every document gets squashed down into a single combined vector, one point sitting somewhere in vector space. Search then becomes a question of "which points are closest to my query point," and HNSW builds a graph of neighbors around exactly that assumption. One document, one point, one fixed location in the graph.
MaxSim doesn't play by those rules at all. A document isn't one point anymore, it's a whole cluster of points, one for every token. There's no single, fixed location you could place a document at in the graph, because its "position" depends entirely on which query token is asking. So even if the symmetry problem didn't exist, you'd still be stuck: HNSW expects one vector per document, and late interaction hands it dozens or hundreds instead. The two approaches are built for fundamentally different shapes of data, one for holistic, whole-document matching, and the other for fine-grained, token-by-token matching.
That assumption is what lets them build a graph of "neighbors" ahead of time and search through it quickly. But MaxSim breaks that assumption completely. Since the distance from A to B doesn't equal the distance from B to A, you simply can't build that kind of static neighbor graph using MaxSim directly. The whole indexing trick falls apart.
There's a deeper practical problem hiding in this too. A document's closest matches aren't fixed anymore, they shift depending on whatever query you happen to be running. Without a precomputed graph to lean on, the only fully accurate option left is to compare the query against every single document at search time. That works fine for a small collection, but the moment you're dealing with millions of documents, checking every one of them individually becomes painfully slow, and sometimes just not practical at all.
6. So how do people work around it?
The common fix is to split the search into two steps instead of relying on one method to do everything.
- First, use a quick, symmetric method to grab a rough shortlist of candidate documents. This step doesn't need to be perfectly accurate, it just needs to be fast and cast a wide net.
- Then, run MaxSim only on that smaller shortlist to rank the candidates properly. Since you're only scoring a handful of documents at this point, the extra computation is no longer a big deal.
This way you get the speed of a simple method for the first pass, and the accuracy of MaxSim where it actually counts, at the final ranking stage.
7. Conclusion
MaxSim gives us a way to properly compare multiple vectors on each side instead of forcing everything into a single number too early. Because every query token gets to independently find its own best match, the final score genuinely rewards documents that satisfy every part of the query well, not just the parts that happen to line up with one dominant vector.
Of course, none of this comes for free. The asymmetry we just walked through causes real headaches for the indexing systems that power fast search, and that's exactly why the two stage approach exists.
It's also worth asking a more basic question: is multi vector search even worth the extra complexity in the first place? That added cost has to earn its keep somewhere, and it usually does in cases where getting every nuance of the query right actually matters.
At the end of the day, MaxSim is what actually makes multi vector search usable. It's the small piece of math that turns two piles of vectors into one score you can rank documents by, and everything else in this space is really just about making that piece of math run faster.