my-vault/00_Inbox/Clippings/2023/06/Building Your Own DevSecOps Knowledge Base with OpenAI, LangChain, and LlamaIndex by Wenqi Glantz May, 2023 Better Programming.md

---
page-title: "Building Your Own DevSecOps Knowledge Base with OpenAI, LangChain, and LlamaIndex | by Wenqi Glantz | May, 2023 | Better Programming"
url: https://betterprogramming.pub/building-your-own-devsecops-knowledge-base-with-openai-langchain-and-llamaindex-b28cda15abb7
date: "2023-06-02 12:38:49"
---
## Building Your Own DevSecOps Knowledge Base with OpenAI, LangChain, and LlamaIndex

## Building your custom knowledge base chatbot

[

![Wenqi Glantz](https://miro.medium.com/v2/resize:fill:88:88/1*Ce4jOl6gjeebSiHsknN2-A.jpeg)



](https://medium.com/@wenqiglantz?source=post_page-----b28cda15abb7--------------------------------)[

![Better Programming](https://miro.medium.com/v2/resize:fill:48:48/1*QNoA3XlXLHz22zQazc0syg.png)



](https://betterprogramming.pub/?source=post_page-----b28cda15abb7--------------------------------)

![](https://miro.medium.com/v2/resize:fit:1400/1*ER6RlV0fckoT4swB39MnJA.png)

Diagram by author

DevSecOps is a big part of what I do daily at my current job. I love learning in the DevSecOps space and sharing my knowledge with others through blogging. Often, I find myself searching through my articles for the information I need. Wouldn’t it be nice to build my own custom knowledge base for DevSecOps so I can feed my files or articles to it and search it when needed?

In this article, let’s explore building a custom DevSecOps knowledge base using OpenAI, LangChain, and LlamaIndex (GPT Index).

## High-Level Architecture

We are going first to feed my article files to our knowledge base. Then we query our knowledge base with questions from my article files related to DevSecOps.

Let’s split the architecture into two stages: data ingestion/indexing and data querying.

![](https://miro.medium.com/v2/resize:fit:1400/1*7-3ejsbENu_qMF77PAzHCw.png)

Diagram by author

![](https://miro.medium.com/v2/resize:fit:1400/1*eWgA_U9RAFtm7IS0EmDecQ.png)

Diagram by author

Now, let’s get started building our knowledge base.

## Prerequisites

-   Python installation: assume you have Python installed in your local environment. If not, please refer to the Python [download](https://www.python.org/downloads/) page to download and install Python. Be sure to upgrade pip to the latest version by running the following command:

python -m pip install -U pip

-   OpenAI API key: navigate to OpenAI’s [API Keys](https://platform.openai.com/account/api-keys) page to generate a new API key if you don’t already have one. Suggest you also set up a usage limit through their [Usage Limits](https://platform.openai.com/account/billing/limits) page so you can manage your spending.
-   Create a directory on your local environment, for example `DevSecOpsKB`. You will be running all commands below in this directory.

## Installation of Libraries

To train our custom DevSecOps knowledge base, we need to install a few libraries. Simply navigate to the `DevSecOpsKB` directory, and run:

pip install openai langchain llama\_index==0.6.12 pypdf PyCryptodome gradio

Note: we specified version `0.6.12` for `llama_index`. Without specifying the version, it would install the latest version, `0.6.16` as of this update (May 31 2023), which introduced breaking changes. We cover some of the changes in the latest `llama_index` release in [another blog](https://betterprogramming.pub/a-glimpse-into-the-mechanics-of-llamaindex-apps-through-the-lens-of-observability-9e7c49f4cb32?sk=6bb0a3a8dc496e1f58523991f063550e).

Let’s take a closer look at each library.

## OpenAI library

We are using [OpenAI](https://openai.com/) library for two purposes:

-   Data ingestion/indexing: as depicted in the architecture diagram above, we will be calling OpenAI’s embedding model `text-embedding-ada-002` via LangChain under the hood.
-   Data query: we will call OpenAI’s GPT-3.5 LLM (Large Language Model). GPT-3.5 models can understand and generate natural language or code. We will be using their most capable and cost-effective model in the GPT-3.5 family, `gpt-3.5-turbo`.

## LangChain

[LangChain](https://python.langchain.com/en/latest/index.html) is an open source library that provides developers with the necessary tools to create applications powered by LLMs. It is a framework built around LLMs that can be used for chatbots, Generative Question-Answering (GQA), summarization, and much more. The core idea of the library is that developers can “chain” together different components to create more advanced use cases around LLMs.

LangChain offers a series of modules, which are the core abstractions as the building blocks of any LLM-powered application. These modules include models, prompts, memory, indexes, chains, agents, and callbacks. For our knowledge base chatbot, we will be using LangChain’s `chat_models` module.

## LlamaIndex

[LlamaIndex](https://gpt-index.readthedocs.io/en/latest/) uses LangChain’s LLM modules and allows for customizing the underlying LLM. LlamaIndex is a powerful tool that provides a central interface to connect the LLM with external data and allows you to create a chatbot based on the data you feed it. With LlamaIndex, you don’t need to be an NLP or machine learning expert. You only need to provide the data you want the chatbot to use, and LlamaIndex will take care of the rest.

As [outlined](https://github.com/jerryjliu/llama_index) by [Jerry Liu](https://twitter.com/jerryjliu0), the creator of LlamaIndex, LlamaIndex provides the following tools in an easy-to-use fashion:

-   Offers data connectors to ingest your existing data sources and data formats (APIs, PDFs, docs, SQL, etc.)
-   Provides ways to structure your data (indices, graphs) so that this data can be easily used with LLMs.
-   Provides an advanced retrieval/query interface over your data: Feed in any LLM input prompt, get back retrieved context and knowledge-augmented output.
-   Allows easy integrations with your outer application framework (e.g., LangChain, Flask, Docker, ChatGPT, or anything else).

## pypdf + PyCryptodome

[pypdf](https://pypi.org/project/pypdf/) is a free and open source pure-python PDF library capable of splitting, merging, cropping, and transforming the pages of PDF files. We will be using this library to parse our PDF files. PyCryptodome is another library that helps prevent errors while parsing PDF files.

## Gradio

[Gradio](https://gradio.app/) is an open source Python package that allows you to quickly create easy-to-use, customizable UI components for your ML model, any API, or even an arbitrary Python function using a few lines of code. You can integrate the Gradio GUI directly into your Jupyter Notebook or share it as a link with anyone. Let’s use Gradio to build a simple UI for our knowledge base.

## How to Add Data Source

I converted my articles listed in [The Path to DevOps Self-Service: A Five-Part Series](https://medium.com/@wenqiglantz/the-path-to-devops-self-service-a-five-part-series-5ea5d4552f9e), along with the [Troubleshooting Tips for GitHub Actions Workflows](https://medium.com/better-programming/17-troubleshooting-tips-for-github-actions-workflows-43394e4f1a8a), into PDFs and saved those PDF documents under my `DevSecOpsKB/data` directory. Let’s use these documents to start training our knowledge base chatbot.

![](https://miro.medium.com/v2/resize:fit:1400/1*csTczE3LacgqRFtIoO-0xA.png)

## Implement Python Code

There are many open source Python tutorials online for building custom chatbots, but many contain outdated code as they were built on older versions of the libraries, and hard to get them to work as desired. I recommend follow the instructions on the [LlamaIndex Usage Pattern](https://github.com/jerryjliu/llama_index/blob/main/docs/guides/primer/usage_pattern.md) page as the base framework, then add your custom logic. Let’s dive into the code.

**Step 1**: Import the following modules and classes:

from llama\_index import StorageContext, ServiceContext, GPTVectorStoreIndex, LLMPredictor, PromptHelper, SimpleDirectoryReader, load\_index\_from\_storage  
from langchain.chat\_models import ChatOpenAI  
import gradio as gr  
import sys  
import os

-   `SimpleDirectoryReader`, `LLMPredictor`, `PromptHelper`, `StorageContext`, `ServiceContext`, `GPTVectorStoreIndex`, and `load_index_from_storage` are classes from the `llama_index` module.
-   `ChatOpenAI` is a class from the `langchain.chat_models` module.
-   `gradio` is the library we use for creating web interfaces.
-   `sys` and `os` are standard Python modules for system-related operations.

**Step 2**: The API key for OpenAI is set as an environment variable using `os.environ["OPENAI_API_KEY"]`. You need to replace `'YOUR-OPENAI-API-KEY'` with your actual OpenAI API key for it to work.

os.environ\["OPENAI\_API\_KEY"\] = 'YOUR-OPENAI-API-KEY'

**Step 3**: Define the function `data_ingestion_indexing(directory_path)`. This function is responsible for ingesting the data and creating and saving the index used for data queries in our knowledge base.

def create\_service\_context():

      
    max\_input\_size = 4096  
    num\_outputs = 512  
    max\_chunk\_overlap = 20  
    chunk\_size\_limit = 600

      
    prompt\_helper = PromptHelper(max\_input\_size, num\_outputs, max\_chunk\_overlap, chunk\_size\_limit=chunk\_size\_limit)

      
    llm\_predictor = LLMPredictor(llm=ChatOpenAI(temperature=0.5, model\_name="gpt-3.5-turbo", max\_tokens=num\_outputs))

      
    service\_context = ServiceContext.from\_defaults(llm\_predictor=llm\_predictor, prompt\_helper=prompt\_helper)  
    return service\_context

def data\_ingestion\_indexing(directory\_path):

      
    documents = SimpleDirectoryReader(directory\_path).load\_data()

          
    index = GPTVectorStoreIndex.from\_documents(  
        documents, service\_context=create\_service\_context()  
    )

      
    index.storage\_context.persist()

    return index

-   We define a utility function named `create_service_context`, which creates the `ServiceContext`, a utility container for LlamaIndex index and query classes. The container contains objects that are commonly used for configuring every index and query, such as the `LLMPredictor` (for configuring the LLM, it is a wrapper class around LangChain’s LLMChain that allows easy integration into LlamaIndex), the `PromptHelper` (allows the user to explicitly set certain constraint parameters, such as maximum input size, number of generated output tokens, maximum chunk overlap, etc.), the `BaseEmbedding` (for configuring the embedding model), and more.
-   It uses `SimpleDirectoryReader` to load data from the specified directory path.
-   It creates an instance of `GPTVectorStoreIndex` with the loaded `documents`, and the `service_context` by calling the utility function `create_service_context()`.
-   Finally, it calls the `storage_context` and persists the index to disk under the default `storage` folder, and returns the `index` object.

**Step 4**: Define the function `data_querying(input_text)`. This function is the core of our knowledge base logic.

def data\_querying(input\_text):

      
    storage\_context = StorageContext.from\_defaults(persist\_dir="./storage")

      
    index = load\_index\_from\_storage(storage\_context, service\_context=create\_service\_context())

          
    response = index.as\_query\_engine().query(input\_text)

        return response.response

-   It rebuilds storage context.
-   It loads the index from storage. Since we initialized the index with a custom `ServiceContext` object, we also need to pass in the same `ServiceContext` during `load_index_from_storage`.
-   It queries the index with the input text using `index.as_query_engine().query()`.
-   It returns the response received from the index.

**Step 5**: Define the UI by creating an instance of `gr.Interface`.

iface = gr.Interface(fn=data\_querying,  
                     inputs=gr.components.Textbox(lines=7, label="Enter your text"),  
                     outputs="text",  
                     title="Wenqi's Custom-trained DevSecOps Knowledge Base")

-   The `fn` parameter is set to the `data_querying` function defined earlier.
-   The `inputs` parameter specifies a textbox input component with 7 lines for entering text.
-   The `outputs` parameter specifies that the output will be text-based.
-   The `title` parameter sets the title of the web interface. Customize it to whatever you want your UI title to be.

**Step 6**: The `data_ingestion_indexing` function is called with the argument `data` to create and save the index. Notice this `data` directory is where we store our PDF documents. If you want to name your directory differently, change it here accordingly.

  
index = data\_ingestion\_indexing("data")

**Step 7**: The `iface.launch(share=False)` line launches the UI, making the chatbot accessible through a web browser. You have the option of turning `share` to `True`, which allows Gradio to create a share link so you can share your knowledge base chatbot with others. For this POC, we are disabling this feature for simplicity reason.

iface.launch(share=False)

See the complete code below. Copy this code into a file named `kb.py`, and placed it at the root of our `DevSecOpsKB` directory.

from llama\_index import SimpleDirectoryReader, LLMPredictor, PromptHelper, StorageContext, ServiceContext, GPTVectorStoreIndex, load\_index\_from\_storage  
from langchain.chat\_models import ChatOpenAI  
import gradio as gr  
import sys  
import os

os.environ\["OPENAI\_API\_KEY"\] = 'YOUR-OPENAI-API-KEY'

def create\_service\_context():

      
    max\_input\_size = 4096  
    num\_outputs = 512  
    max\_chunk\_overlap = 20  
    chunk\_size\_limit = 600

      
    prompt\_helper = PromptHelper(max\_input\_size, num\_outputs, max\_chunk\_overlap, chunk\_size\_limit=chunk\_size\_limit)

      
    llm\_predictor = LLMPredictor(llm=ChatOpenAI(temperature=0.5, model\_name="gpt-3.5-turbo", max\_tokens=num\_outputs))

      
    service\_context = ServiceContext.from\_defaults(llm\_predictor=llm\_predictor, prompt\_helper=prompt\_helper)  
    return service\_context

def data\_ingestion\_indexing(directory\_path):

      
    documents = SimpleDirectoryReader(directory\_path).load\_data()

          
    index = GPTVectorStoreIndex.from\_documents(  
        documents, service\_context=create\_service\_context()  
    )

      
    index.storage\_context.persist()

    return index

def data\_querying(input\_text):

      
    storage\_context = StorageContext.from\_defaults(persist\_dir="./storage")

      
    index = load\_index\_from\_storage(storage\_context, service\_context=create\_service\_context())

          
    response = index.as\_query\_engine().query(input\_text)

        return response.response

iface = gr.Interface(fn=data\_querying,  
                     inputs=gr.components.Textbox(lines=7, label="Enter your question"),  
                     outputs="text",  
                     title="Wenqi's Custom-trained DevSecOps Knowledge Base")

  
index = data\_ingestion\_indexing("data")  
iface.launch(share=False)

## Launch DevSecOps Knowledge Base

Now that we have our custom PDF files ready, and the code is ready, let’s launch our DevSecOps knowledge base by running the following command in the `DevSecOpsKB` directory:

python kb.py

Let’s launch the UI of our new knowledge base: [http://127.0.0.1:7860/](http://127.0.0.1:7860/).

Here we go! Our new knowledge base is ready for us to tap into. Let’s ask a generic question on a term I coined in [one of my articles on DevOps self-service model](https://medium.com/better-programming/devops-self-service-pipeline-architecture-and-its-3-2-1-rule-517dc0bbcb4a), in particular, the 3–2–1 rule, and I was happy to see that our new knowledge base outputs the right information I was looking for:

![](https://miro.medium.com/v2/resize:fit:1400/1*EgZiv4JpKxlO79jdbg4YBQ.png)

Asking it with a specific error encountered in the GitHub Actions workflow, we get the desired answer. See the following:

![](https://miro.medium.com/v2/resize:fit:1400/1*MQsWPO4mO5OW1ZTJVhQxXQ.png)

Now, let’s ask if our knowledge base can answer questions on [Harden Runner](https://www.stepsecurity.io/products/harden-runner):

![](https://miro.medium.com/v2/resize:fit:1400/1*vtcPaYfm7U5koa2v86GVXQ.png)

Right on! I am amazed at how accurate the answer is. Next, let’s see if our knowledge base can output a code snippet:

![](https://miro.medium.com/v2/resize:fit:1400/1*nCIXBpZo4oFaFY4a_zBtHQ.png)

This one works like a charm!

Now, let’s attempt a negative scenario: let’s try to ask a question that is not in the provided source documents:

![](https://miro.medium.com/v2/resize:fit:1400/1*uIaWmYJj5Q2lyEQNj_LurQ.png)

Job well done! LlamaIndex seems to have a guardrail in place against hallucination, which is a confident response by an AI that does not seem justified by its training data, either because it is insufficient, biased, or too specialized.

## Does This AI Bot Expose My Private Data to OpenAI?

The answer is no. Per [OpenAI privacy policy on API](https://help.openai.com/en/articles/5722486-how-your-data-is-used-to-improve-model-performance):

> OpenAI does not use data submitted by customers via our API to train OpenAI models or improve OpenAI’s service offering.

Both our functions, `data_ingestion_indexing` for data ingestion/indexing and `data_querying` for Q&A, invoke OpenAI APIs via LangChain, so we can rest assured that OpenAI does not use our private data per their privacy policy on API mentioned above.

## A Note on Cost

As you may already know, using OpenAI models does incur a cost. In our use case, we use its embedding model during data ingestion/indexing, and chat model for data querying. Here are the pricing details:

-   For embedding model `text-embedding-ada-002`: $0.0004 / 1K tokens
-   For chat model `gpt-3.5-turbo`: $0.002 / 1K tokens

Here is a screenshot of my OpenAI usage while working on this chatbot:

![](https://miro.medium.com/v2/resize:fit:1400/1*xRguhQmi62D7Y3YsYJi7Ig.png)

If you plan to use OpenAI LLMs, I strongly encourage you to configure a usage limit on OpenAI’s [Usage Limit page](https://platform.openai.com/account/billing/limits), where you can define a hard limit and soft limit, so you manage your usage properly.

## Summary

This article explored how to build a customized DevSecOps knowledge base chatbot. This is a mere proof of concept. The potential of incorporating LlamaIndex and LangChain into building apps that harness the power of LLMs through private data is limitless!

The source code for this article can be found in [my GitHub repo](https://github.com/wenqiglantz/DevSecOpsKB-LlamaIndex-LangChain-OpenAI/tree/main/DevSecOpsKB).

Happy coding!