The Prospera Labs Flexi-Match Knowledge Base is a flexible system that allows your agent to access a large database of knowledge. The Knowledge Base can be used for multiple different use-cases - from answering questions about products and services, through to analyzing documents or situations against best-practices that have been uploaded.

Loading In Knowledge

There are many different ways to load in knowledge to your agent, which will depend a lot on the different use-cases for your agent.

Uploading Documents

One of the simplest ways to load in information is to import documents prepared in conventional Office software, such as PDF or Microsoft Word documents.

You can upload documents by pressing the Import Document button in the top right.

This will bring you to the bulk import page where you can then upload documents either by dragging and dropping, or by pressing the upload button and selecting the files.

Uploading Web Pages

You can similarly import information from web pages. This is done in the Imported Webpage menu.

Then press the “Import Web Page URL” button.

That will bring you to this page. Its a bit terse but you can paste in the URL here:

https://prosperalabs.ai/

How It Works

Here we will explain the basic process of how the knowledge base works in the default Q&A setup.

Document Ingestion

When any type of document is taken into the system, we go through the following process:

1. Convert into raw text

2. Break the text into small sections based on the semantics and headers of the document

3. For each section, we generate a list of questions that the section of text answers. E.g. almost like Jeopardy, we are going backwards from the answer (contained in the document) to questions

4. We fetch the embedding vector for each of the questions

5. Create a summary of the entire document, referred to as the “Qualifying Text” internally because it helps qualify whether a particular knowledge chunk is relevant

6. Write the knowledge chunks out to the database

Querying

When you query the knowledge base, the system goes through the following steps:

1. The original raw query is transformed into a clean query. For the default knowledge base configuration, this means taking an ambiguous sentence, like location of food and turning it into the format of a question, such as Where is the food located? so that the text matches the format of the questions that were generated by the ingestion engine

2. We lookup the embedding vector for the transformed query

3. We use the embedding vector to perform a query on the knowledge base and find the top K knowledge chunks whose matching text (a.k.a the generated question) is the closest to the query text

4. We use a reranking algorithm to rerank the matched knowledge chunks against the query

5. We discard all except the top 5 matching knowledge chunks

6. The matching knowledge chunks are returned. For your standard AI Agents, these matching knowledge chunks are then fed into the conversation history for the agent as a tool-result, which the agent can then use to formulate and write its response

Customizing

The knowledge base can be connected to many different sources of data, combining them together into a homogenous knowledge system. Sources of data include documents and web-pages that you import through the user interface. But may also include custom data schemas that you have created yourself, containing information that the bot has extracted from conversations (this allows the agent to have a ‘memory’ in-between multiple sessions). Eventually we may even allow third-party API services to become sources of data for the knowledge base.

Each source of data can be customized independently. There are six different smart-chains that you can modify to change the behavior of the knowledge base. Those are:

1. Chunker chain - The chunker chain is responsible for taking a large, long document and breaking it up into individual sections of content. The default version just breaks a document into sections based on format meaning, e.g. using the existing headers and paragraphs as section breaks

2. Matching Text Chain - The matching text chain is responsible for taking a chunk of content, and transforming them into bits of text that are embedded and matched against when querying. The default form that matching texts take are questions, e.g. “What kind of food is available in Alaska?”. However, its not necessary and some use-cases of the knowledge base require querying based on other formats of text.

3. Qualifying Text Chain - The qualifying text chain is responsible for taking the chunk of content, along with the document as a whole, and creating a summary of the document as a whole along with how the text of the specific knowledge chunk fits into that larger document. E.g. if the whole document is on the subject of “Arctic Cuisine”, then the qualifying text might be “The section describes Alaskan cuisine within the context of a larger document describing various arctic cuisines”. The qualifying text is used to provide contextual information so that the LLM can correctly interpret the content of the knowledge chunk, which loses some meaning when separated from the larger document

4. Query Transformation Chain - The query transformation chain is responsible for taking the query provided by the user, which might take a variety of different forms, and transform it into the same format that is produced by the Matching Text Chain. By default the matching texts take the format of a question, so the default query transformation chain is designed to take whatever you type in and transform it also into a question. E.g. it might take an abstract bit of text like “location of food” and turn it into a proper question format of “Where can the food be found?”