Learning with AI (Part 4) - Model driven discovery

"Creation works ceaselessly through man. But man does not create, he discovers"

— Antoni Gaudí (Gaudí: A Biography)

Each one of the 2,400 entries in IdeaKache taught me something. Since they've lived in my head as I built the collection over nine years, I can go back to them again and again. For everyone else, however, they're going to need help to surface the entries that are helpful to them. That requires great discovery.

Discovery matters because I can't compete with ChatGPT on breadth of content. What I can bet on is that my taste is good enough that people find the collection valuable. That bet only pays off if users can actually find the entries that speak to them. Without good discovery, they'll never get deep enough to see the value.

Of course, the need for good discovery has to be traded off against costs. If every query in the search box on IdeaKache costs me $1, you can see that this could get ugly for me quickly.

Summary

At a high level, when a user types a query into the search box in IdeaKache, the website's code will make four things happen:

Have Postgres run a keyword search;
At the same time as 1, call an embedding model, which turns the query into numbers;
Hand those numbers from the embedding model to Postgres, which compares and ranks the entries (using the keyword approach as a backup);
Send the top-ranked entries plus the original search to the generative model (i.e. Claude), which generates an "answer" for the user.

1. How Postgres runs a typical keyword search

Keyword searching is the traditional way to use search on a website. It's a blunt, rules-based instrument. It works well when a user knows exactly what they want.

At least in the early days of IdeaKache, however, given there's 2,400 entries, users won't always know exactly what they want. IdeaKache's search queries will require more nuance and so a keyword search alone won't be sufficient.

Exhibit 1

Keyword search follows a set of rules, run on Postgres

1. Normalize the words Postgres lowercases everything and strips filler words like "how", "to" and "the" that appear in almost every entry and carry no signal.

2. Stem the words What's left gets cut to its root. As one example, business and busy both stem to busi and so appear to be the same word to keyword search.

3. Match against the index Each entry was stemmed once when it was saved and its stems stored in an index. The search's stems are looked up against that index to find the entries that share them.

4. Qualify the entries The default rule is strict. Only entries that share every stem of the search qualify.

The tradeoff with looser rules is burying the results in noise.

5. Score each entry Each qualifying entry gets a number based on how often the matched stems appear, how close they sit and potentially where they appear.

6. Sort the entries The qualifying entries are ordered by their score, highest first, so the strongest keyword match sits at the top. That ranked list is what the search returns.

7. Example of the keyword process in action "How do I know where to start?" stems to know and start. Six entries qualify and Estée Lauder ranks first with "By the way, it is never too late to start a business, you know, just as it is never too late to make yourself beautiful".

"You don't know, you've just got to do it" (James Dyson) and "Just get started ... Order finds itself through action" (Stewart Resnick) don't surface because they each share only one stem with the query (know and start, respectively), even though they are relevant to the query in a holistic sense.

2. Why an embedding model, and how it works

Having confirmed that keyword searching can't be the only solution, the next decision was between using an embedding model as a first pass or using a generative model end to end. The ~2,400 entries on IdeaKache fit within the context of generative models so, in theory, a generative model could run the whole "search" when a user types in a query.

The issue with using a generative model is that it would read the full IdeaKache collection for every query (i.e. read all ~2,400 entries each time). An embedding model, however, would read the entire collection once the model is ready, and then read each new entry just the once, on its addition to IdeaKache. That means fewer tokens and lower costs for each search.

After the code hands off the query to the embedding model, it breaks the text into individual tokens and each token gets an ID number.

The ID numbers correspond to a row number in the equivalent of a filing cabinet. Each row number holds, say, 1,024 numbers. Think of these 1,024 numbers as different characteristics of the token (almost like height, weight, eye color etc). Each token's 1,024 numbers don't change. They are set by the model during training.

The model takes a copy of the 1,024 numbers. If there were 10 tokens in the original search, the model now holds copies of 10 sets of 1,024 numbers. The model then needs to convert each set into a weighted average that changes the 1,024 numbers.

Exhibit 2

The weighted average process measures how related the sets are, to decide how much each set should blend into one another

1. Pair up the sets Start with a pair of two sets of 1,024 numbers, e.g., set 1 and set 2.

2. Multiply, position by position Position 1 in set 1 times position 1 in set 2, through each position from 1 to 1,024.

3. Add the pair subtotals The 1,024 subtotals sum to the pair's score. The higher the score, the more related the pair.

4. Repeat steps 1 to 3 for every pair For example, pair set 1 and set 3, then set 2 and set 3, then set 3 and set 4 until every pair has been completed. Each pair gets a score.

5. Translate and total, per set Now focus on one set. For set 1, take its score with each other set, translate each with a math constant, and add those translated numbers into set 1's total, say 100. Every set gets its own total this way.

6. Weight each set For set 1, its score with set 3 is negative, so set 3 contributes essentially nothing to weighted set 1. Set 1's score with set 6 is very high, so set 6 could contribute 54.6% of its numbers to weighted set 1.

Applied across all 1,024 positions, that gives set 1 a rewritten set of 1,024 numbers, and the same happens to every set, so you end up with 10 rewritten sets of numbers.

7. Average into a single set, position by position Every position in each set now has a number, after the weighting process that concluded in step 6. For each position, add its value across all 10 weighted sets and divide by 10.

For example, if position 47 totals 300 across the sets, the single set holds 30 (300/10) there. The end result is one set of 1,024 numbers.

For the embedding model provider, I went with Voyage. I considered OpenAI embeddings and Cohere, which both sounded good as well. After picking Voyage, a one-time script goes through the database and feeds each entry to it.

That creates a set of 1,024 numbers, using the same process it runs on a query. For IdeaKache, that means ~2,400 entries each get a set of 1,024 numbers and those numbers are stored in Postgres.

3. How Postgres creates the shortlist

Postgres does a relatedness comparison between the query's set of 1,024 numbers and the numbers for each entry, putting the entry with the highest relatedness score at the top. Postgres will also check the keyword search results as a backup.

Postgres will then hand back the ranked list of entries to the website's code. There will normally be a cap for the number of entries, otherwise Postgres will hand back a ranking of every entry, which defeats the purpose of the embedding model. The website's code then hands the ranked list of entries to the generative model.

4. How the generative model closes out the query

The generative model's job here is to pick the best of the shortlist and frame it for the user.

The generative model takes the ranked list and breaks it up into tokens. Once the result is broken up into tokens, similar to the embedding model, each token gets an ID number that can be used to pull its row of numbers from the generative model's own filing cabinet. That row of numbers could be as large as 8,000+ numbers.

The generative model completes the same pairwise multiplication as in Exhibit 2. For example, if there are 20 tokens, it has copies of 20 sets of numbers. For set 1 and set 2, it multiplies position 1 in set 1 by position 1 in set 2, all the way through to the 8,000+ numbers. It then translates the total for each paired set with a math constant, and weights it according to the total.

At the end of this process, all 20 sets of numbers have been rewritten. If set 1 and set 17 have a very high total, set 17 will get rewritten to have a significant proportion of set 1's numbers. As the last "set", set 20 gets rewritten based on its relatedness with each of the 19 sets before it.

The generative model then compares set 20 to every token in its vocabulary (can be 100,000+ tokens), whether or not those tokens appeared in the original query. It scores each token with the same position-by-position multiplication. The token with the highest total becomes the next token (and the start of the search response). Then the process is run again for each token in its response. A 200-token answer will require the model to run the pairing-and-scoring machine 200 times.

The model stops when the special end-of-response token in its vocabulary becomes the top score. The model sends the answer back to the website's code, which then displays the answer to the user.

For the generative model, I started with Haiku 4.5. After getting some noisy results during testing, I upgraded to Sonnet 4.6, where the results were more in line with what I expected.

The core premise of IdeaKache is attribution, taste and judgement, and so the generative model has to serve answers actually in the collection. This meant putting in guardrails, including giving the model instructions that forbid quoting or restating an entry's text. Generative models want to paraphrase and naturally smooth words, which would break the whole point of IdeaKache. The model must point at the entry in the collection and pull straight from the database.

The result of all this is that when a user types "How do I know where to start?" into the search box, they will get the best of what IdeaKache has to offer. Sign in to try it yourself, and you'll see those James Dyson and Stewart Resnick quotes that didn't make it into Exhibit 1.