Page Comparison

...

This will bring you to the bulk import page where you can then upload documents either by dragging and dropping, or by pressing the upload button and selecting the files.

...

Uploading Web Pages

You can similarly import information from web pages. This is done in the Imported Webpage menu.

...

• Document Qualifying Text - the default document qualifying text smart chain produces a summary of the entire document to use as the qualifying text for all the separate knowledge chunks which resulted from the document.

• Document Matching Text - the default matching text smart chain is designed to take a chunk of content and try to come up with the questions that said content is answering. E.g. it works almost like jeopardy, trying to take an answer and go backwards to the questions. It is designed to generate as many questions as possible for the given knowledge chunk as it can. Additionally, it is supposed to take into account the contextual information provided by the qualifying text.

Premade Query Processing Chains

These are pre-made smart chains that are designed for processing a query for a default Q&A style knowledge base. These pre-made smart chains can be found if you search for the prefix knowledge_base_basic_

...

• Basic Query Transformer - The basic query transformer is designed for Q&A style knowledge bases. It will take whatever text was provided by the user for the query, e.g. “location of food”, and then turning it into a question in the same format that the Document Matching Text smart chain produces. e.g. “Where can I find the food?” By turning a potentially ambiguous statement into a specific question in the same format as the matching texts, we give our system the best chance of finding the correct knowledge chunks in the database.

• Basic Reranker - The basic reranker (currently not visible in the above screenshot) is designed for Q&A style knowledge bases. It takes the content of the knowledge chunk, and then uses a Ranked Selection smart chain step in order to assess how well the content of the knowledge chunk matches the question. By using the LLM predicted probabilities, it is able to come up with a final ranking score between 0 and 1. The scoring formula is calibrated so that a score of 0.5 corresponds roughly to a document that the model thinks has a 50-50% chance of being the answer to the users question

• Basic Filterer - The basic filterer is designed for Q&A style knowledge bases. It will remove any knowledge chunks that have a match score lower then 0.5 or a rerank score lower then 0.5. This typically only eliminates knowledge chunks that are very very obviously wrong.

Smart Chain Bindings for Knowledge Base

The knowledge base has a different set of bindings for imported web-pages versus imported documents versus other custom data schemas that are loaded in. This allows you to customize how the knowledge base imports documents separately from how it imports web pages, and so on… These bindings can be found by searching for the prefix knowledge_base_imported_ on the smart chains.

...

The other bindings for the queries are shared for the entire knowledge base. They can be found by searching for the prefix knowledge_base_main_ in the bindings:

...

These allow you to customize the default way that the knowledge base processes queries.

Identity Bindings

For each of the identity smart chain, there is also an identity binding which points to it. These can be found by searching for identity in the smart chain bindings.

...

These are some parts of the knowledge base system that are programmatically designed to use the identity smart chains in order to process knowledge base queries in specific situations that require turning off that components. For example, when you do a query in the Knowledge user interface, by default the filtering chain is turned off so that you can see all of the matches, whether they were good or bad.