Search: full-text and vector

Full-text search finds texts by their words, via an inverted index and stems. Vector search finds them by meaning, via embeddings and nearest neighbors.

Search is not LIKE

Say a site has a search box. The simplest idea that comes to mind is WHERE title LIKE '%boundary%'. That looks for a substring letter for letter. It finds “boundary lines” but not “boundaries”, because the ending is different. It treats “Boundary” and “boundary” as different words unless you switch to ILIKE. It can’t tell a page about the topic from one that mentions it in passing, it orders nothing by importance, and on a big table it’s slow: an ordinary index doesn’t help when the pattern starts with %. Searching text calls for a device of its own.

Searching by words: how it works

Full-text search prepares the text in advance, like the index at the back of a book (you met that idea in the lesson on indexes).

1. Tokens. The text is cut into words.
2. Normalization. Everything is put in one case, and function words are dropped: “the”, “of”, “and”, “for”, “how”. They are everywhere and tell texts apart from nothing.
3. Stems. Words are reduced to a stem: “survey”, “surveys”, “surveyed” and “surveying” all become “survey”. This is called stemming, or lemmatization (a lemma is the dictionary form). In Russian this matters far more than in English, because a Russian word has dozens of endings, but even English has plurals, tenses and “boundary” against “boundaries”.
4. The index. A table is built: “stem → the texts it appears in”. This is an inverted index.
5. Ranking. The results are put in order. The idea: a word counts for more if it’s frequent in this text and rare in the whole collection (this is what TF-IDF and its improvement BM25 compute), and a match in the title weighs more than one in the body.

Play with the diagram. On the “By words” tab the search runs over your own services. Press “Another form of the word”, then switch off “Match word forms”: that’s what LIKE would do.

By words By meaning

A simplified model: it cuts common English endings and counts matches (a title counts double). A real engine uses full dictionaries.

Query Another form of the word Match word forms

The query became these stems: servic. Function words (the, of, and…) were dropped.

1. 3 First service
   title stems: first, servic
2. 3 Second service
   title stems: second, servic
3. 0 Consultation
   title stems: consultation

With word forms found: 2 of 3. Try “Another form of the word”: the stem is the same, so the result is the same.

Text alternative: word search reduces the words of the query and of the texts to stems, builds an index and ranks by the number of matches; it finds different forms of a word but not synonyms. Search by meaning compares numeric “coordinates of meaning” and finds the texts closest in meaning even when the words differ, but it is poor at exact codes and numbers.

PostgreSQL has all of this built in. The tsvector type holds a text broken into stems, tsquery holds a query, the @@ operator tests for a match, ts_rank computes a score, and a GIN index speeds all of it up on big tables.

SELECT title,
       ts_rank(to_tsvector('english', title || ' ' || text), q) AS rank
FROM items, plainto_tsquery('english', 'survey') AS q
WHERE to_tsvector('english', title || ' ' || text) @@ q
ORDER BY rank DESC;

CREATE INDEX items_fts_idx ON items
  USING GIN (to_tsvector('english', title || ' ' || text));

The word 'english' picks the English configuration: its stemming rules and its list of function words. Name the configuration explicitly, because with the wrong one the words aren’t reduced to the stems you expect.

The built-in search has limits: it doesn’t correct typos and it doesn’t suggest as you type. When that matters, people reach for a separate search engine. Meilisearch and Typesense are easy to install and handle typos and “instant” search. Elasticsearch and OpenSearch are for when the data is huge, the filters and tuning are complex, and there’s somebody to run it. The price is the same as for any new database: one more server, one more copy of the data that has to be kept in step with the main one (remember the deliberate copy from lesson 3).

Check yourself: why are function words dropped before searching?

They appear in almost every text, so they don’t help tell one text from another, and they would make the index swell.

Searching by meaning

Now a question that words can’t answer. A visitor types “how much does it cost”, and the page is called “Prices and billing”. They have no words in common, but the meaning is the same.

Embeddings were invented for this. A special model turns a text into a list of hundreds or thousands of numbers: “coordinates of meaning”. Texts close in meaning get close coordinates, like cities on a map. To answer a question, it’s turned into coordinates too, and the nearest neighbors are taken: the texts that lie next to it. That’s how it finds things even when the words are different.

Switch to the “By meaning” tab. There the map is drawn by hand, with just two axes instead of hundreds, and it shows the idea: a question about price lands next to “Prices and billing” and “Payment methods”, and a question about getting money back lands next to “Refund policy”. Below it you can see what a word search would have found: often nothing.

Storing these numbers and finding the nearest among them is the job of vector databases (Qdrant, Pinecone, Weaviate) and the pgvector extension for PostgreSQL. So as not to compare a question with every text, they build a special approximate-search index (HNSW, IVF), and again they pay for it: the answer is “almost the closest”, but fast.

What search by meaning does badly:

• Exact identifiers. An order number, a SKU or the number of a statute isn’t meaning. Vector search returns “similar” ones, and you need that exact one. That’s why the two methods are often combined in hybrid search.
• Explainability. It’s hard to say why exactly this result came up: similarity is a number.
• Model errors. It may count as close two texts that are opposites (“refunds are allowed” and “refunds are not allowed”).

RAG: how AI answers from your texts

The same embeddings stand behind most chatbots that answer “from a company’s knowledge base”. The scheme is called RAG (retrieval-augmented generation):

1. Your texts are cut into pieces beforehand, and an embedding is computed for each.
2. A question comes in. Its embedding is compared with the pieces, and the few nearest are taken.
3. The model is given the question and the pieces found: “answer using only this”.

The model isn’t retrained, it’s only shown the right pages. So an answer can be checked against its source, and for the bot to know something new, it’s enough to update the texts.

Whose texts leave, and where do they go

A practical question that is often forgotten. Many people compute embeddings on someone else’s service: the texts go over the network to an outside provider. If those texts contain personal data (names, phone numbers, details of requests), then sending them is already processing personal data by a third party, with the requirements of privacy law (the next lesson is about that). What to do:

• Don’t send more than necessary: the pages of services and the answers to frequent questions usually hold no personal data.
• Find out where the provider processes and stores the texts, and on what terms. Under GDPR a provider that handles personal data on your behalf is your processor, and you need a written contract with it. In the US the state privacy laws ask for similar contracts with “service providers” or “processors”.
• If you do need to send personal data, make sure there’s a legal basis (consent is one), or compute the embeddings yourself, on your own server, with an open model.

The decision here isn’t the developer’s. It belongs to the site’s owner, together with an attorney. But the person who builds the search is the one who has to ask: “What exactly are we sending?”

Practice

You’re in the era of 2022: sites now have a search box that understands word forms. Add one to your project.

The page has a demo that searches your services. Write down three forms of one word and their stem, write a PostgreSQL query with the English configuration, ranking and an index, and then answer briefly: when word search is enough for you, and when you would need meaning. The real PostgreSQL in a browser is on the PGlite site; here the text of the query is what gets checked.

What was wrong before

The built-in search of 2010s sites was usually simple: it looked for words in the title and the text and ranked poorly. Many sites plugged in an outside engine, and Apache Solr and Elasticsearch (both built on the Lucene library) were the usual choices. Search by meaning stayed the business of research labs. Today vector search is available to any site, and the main question is no longer “how do we do it” but “what exactly are we sending, and to whom”.