I use Emacs org-mode for a variety of tasks like time tracking, keeping work and reference notes, putting together professional documents that require pdf/latex using org-mode, tracking certain expenses with ledger, etc. At some point, I might write about all the different ways I use Emacs, an editor that can truly be customized for personal use cases.

Recently, I was looking for a dedicated note-taking package in Emacs that uses org-mode, and came across three that I considered, Org-roam, Denote and Emacs Knowledge Graph (EKG). I had used Org-Roam before, it implements the Zettelkasten system of interconnected note taking, storing text of the notes in a directory and the metadata of the notes like title, tags and links in a SQLite database. It uses a grep like tool for text searches, and SQL queries for searching through titles, tags, links and backlinks. Denote does not use a database, but stores all notes in the directory and the metadata of the note, including creation dates, title, and tags are incorporated in the file name. This allows for searching both the text content of the notes and the metadata of the notes through grep-like search, thus providing a simple and scalable solution to note taking.

However, the package that intrigued me the most is the Emacs Knowledge Graph (EKG). Everything, text content of the notes, metadata like tags, date, title (which is optional) is stored in the SQLite database, and all searches are done through querying the database. This already reminded me of “bag of documents” approach of storing and retrieving documents based on their metadata, rather than their content (which is also considered as metadata).

However, the kicker in EKG is that it has capability (optional) of sending the entire note, before it is saved, to a Large Language Model (LLM) API (default is OpenAI, with other options too), to get the embedding for the entire note. These embeddings are stored in the database too as another piece of metadata for the note. With embeddings, notes that are similar in terms of their semantic content to a given note, even though they many not share any common words, can be retrieved through similarity search on embeddings.

This also means that queries, which get converted to embeddings through API call to LLM, can now find notes that are semantically related to the query, for example searching for ’espresso’ would retrieve notes that talk about drinking coffee or taste of coffee, but may not include the word espresso (this is example given in an EKG video).

There is also a capability to append response from the LLM in the note, sending the content that has been created in the note so far as a prompt to the LLM API, with possibility to continue that conversation with further questions repeating the process. There are also other interesting features like the storage mechanism in the SQLite database resembles an RDF graph, which is a framework for storing triples (subject, predicate, object), which is the same framework that was used when search engines moved beyond simple text based searches to semantic searches (details coming later).

This search for a note-taking package provided a nice glimpse into the evolution of search, which is what I would like to trace briefly, as we start moving into the AI driven search.

We started with text based search in which documents were scanned to find exact match to the search query. Later, regular expressions based search was introduced where patterns could be searched for in documents. Both these methods were inefficient as search time grew linearly with the the number and size of documents.

Next evolution of search was inverted index search, where terms/words were mapped to locations within documents, enabling much faster retrieval of documents with the search term. Early web search engines like Altavista and Yahoo used this technology for searching through fast growing online content.

Next big innovation in search was Google’s introduction of PageRank algorithm, which not only considered the content of online pages, but also how pages were linked to each other, to determine both relevance and quality of search results.

Further evolution of search was not only to match the search term with documents, but also understand the meaning or semantics of the search term to match documents that may not necessarily contain the search term but still were relevant. A technology that facilitated this was Natural Language Processing (NLP), with models like Word2Vec which converted words (more precisely, tokens) into vectors called embeddings and placed semantically similar words close to each other. Each document was also converted into a vector, by aggregating word vectors in the document in some way like averaging. Search query was also similarly transformed into an embedding and a similiarity calculation was performed to retrieve documents matching the search query.

Another technology used to capture semantics of a word was knowledge graph, which represented relationships between words in form of ’triples’ - subject, predicate and object. So a search for a movie director, would also retrieve the movies they directed, as the relationship between the director and movies was encapsulated in a knowledge graph. Another evolution of adding meaning to search was personalization, where search started taking into account search history, location and preferences of a user.

In December 2017 a research team at Google published a seminal paper “Attention is All You Need” that was to significantly impact the next evolution of search. This paper introduced “Transformer” model or architecture for NLP Deep Learning Neural Networks, which made possible to encode the word into vectors or embeddings, considering the surrounding words, thus enriching the understanding of word meaning with the specific context in which the word was used using so called ‘self-attention’ mechanisms. Various approaches are used to convert longer word sequences, like sentences, paragraphs or documents, into embeddings. With this foundational Transformer architecture, search evolved from not only retrieving semantically relevant documents to the search query more effectively, but also generating human-like responses to the search query.

In 2019 Google introduced BERT (Bidirectional Encoder Representations from Transformers) deep learning model using Transformer architecture, and later another well known model called GPT (Generative Pre-Trained Transformer) was developed by OpenAI, and they started providing access to GPT-3 model through API to developers in around June 2020. However, it wasn’t until November 2022, when OpenAI started providing access directly to consumers on its next iteration of GPT model, calling it ChatGPT. It would not be an understatement to say that this development finally pushed the possibilities of AI into public consciousness.

Related to the development of NLP deep learning models, is the emergence of Vector databases, which specialize in storing the embeddings described above, and providing various methods to do similarity search on the embeddings. These embeddings are generated using pre-existing NLP deep learning models, for example, OpenAI has an API to provide embeddings which takes text as an input and returns embedding as an output. Open source models can also be used for this purpose. Vector databases make it possible to localize document search, so text of a documents (each document is divided into text chunks) is converted into embeddings, and then the Vector database returns chunks of documents that are most semantically similar to a search query (which is also converted into an embedding) by performing embedding similarity search.

Throughout this evolution of search, the goal has always been to provide more relevant, accurate, high quality and efficient search results, and the journey has been marked by significant technological innovations at every step, transforming and at times disrupting businesses along the way.