LLM/DocsGPT
1
0
Fork 0
mirror of https://github.com/arc53/DocsGPT.git synced 2025-11-29 08:33:20 +00:00
Code Issues Packages Projects Releases Wiki Activity

models+guide-upd+extentions

Browse Source
This commit is contained in:
Pavel
2025-02-12 14:38:21 +03:00
parent 28cdbe407c
commit b44b9d8016
31 changed files with 955 additions and 311 deletions
Download Patch File Download Diff File Expand all files Collapse all files

79
docs/pages/Models/cloud-providers.md
View File

@@ -1,79 +0,0 @@
---
title: Connecting DocsGPT to LLM Providers
description: Explore the different Large Language Model (LLM) providers you can connect to DocsGPT, from cloud APIs to local inference engines.
---
# Connecting DocsGPT to LLM Providers
DocsGPT is designed to be flexible and work with a variety of Large Language Model (LLM) providers. Whether you prefer the simplicity of a public API, the power of cloud-based models, or the control of local inference engines, DocsGPT can be configured to meet your needs.
This guide will introduce you to the LLM providers that DocsGPT natively supports and explain how to connect to them.
## Supported LLM Providers
DocsGPT offers built-in support for the following LLM providers, selectable during the `setup.sh` script execution:
**Cloud API Providers:**
* **DocsGPT Public API**
* **OpenAI**
* **Google (Vertex AI, Gemini)**
* **Anthropic (Claude)**
* **Groq**
* **HuggingFace Inference API**
* **Azure OpenAI**
## Configuration via `.env` file
Connecting DocsGPT to an LLM provider is primarily configured through environment variables set in the `.env` file located in the root directory of your DocsGPT project.
**Basic Configuration Parameters:**
* **`LLM_NAME`**: This setting is crucial and specifies the provider you want to use. The values correspond to the provider names listed above (e.g., `docsgpt`, `openai`, `google`, `ollama`, etc.).
* **`MODEL_NAME`**: Determines the specific model to be used from the chosen provider (e.g., `gpt-4o`, `gemini-2.0-flash`, `llama3.2:1b`). Refer to the provider's documentation for available model names.
* **`API_KEY`**: Required for most cloud API providers. Obtain this key from your provider's platform and set it in the `.env` file.
* **`OPENAI_BASE_URL`**: Specifically used when connecting to a local inference engine that is OpenAI API compatible. This setting points DocsGPT to the address of your local server.
## Configuration Examples
Here are examples of `.env` configurations for different LLM providers.
**Example for OpenAI:**
To use OpenAI's `gpt-4o` model, your `.env` file would look like this:
```
LLM_NAME=openai
API_KEY=YOUR_OPENAI_API_KEY # Replace with your actual OpenAI API key
MODEL_NAME=gpt-4o
```
**Example for Local Ollama:**
To connect to a local Ollama instance running `llama3.2:1b`, configure your `.env` as follows:
```
LLM_NAME=openai # Using OpenAI compatible API format for local models
API_KEY=None # API Key is not needed for local Ollama
MODEL_NAME=llama3.2:1b
OPENAI_BASE_URL=http://host.docker.internal:11434/v1 # Default Ollama API URL within Docker
```
**Example for OpenAI-Compatible API (DeepSeek):**
Many LLM providers offer APIs that are compatible with the OpenAI API format. DeepSeek is one such example. To connect to DeepSeek, you would still use `LLM_NAME=openai` and point `OPENAI_BASE_URL` to the DeepSeek API endpoint.
```
LLM_NAME=openai
API_KEY=YOUR_DEEPSEEK_API_KEY # Your DeepSeek API key
MODEL_NAME=deepseek-chat # Or your desired DeepSeek model name
OPENAI_BASE_URL=https://api.deepseek.com/v1 # DeepSeek API base URL
```
**Important Note:** When using OpenAI-compatible APIs, you might need to adjust other settings as well, depending on the specific API's requirements. Always consult the provider's API documentation and the [DocsGPT Settings Guide](/Deploying/DocsGPT-Settings) for detailed configuration options.
## Exploring More Providers and Advanced Settings
The providers listed above are those with direct support in `setup.sh`. However, DocsGPT's flexible design allows you to connect to virtually any LLM provider that offers an API, especially those compatible with the OpenAI API standard.
For a comprehensive list of all configurable settings, including advanced options for each provider and details on how to connect to other LLMs, please refer to the [DocsGPT Settings Guide](/Deploying/DocsGPT-Settings). This guide provides in-depth information on customizing your DocsGPT setup to work with a wide range of LLM providers and tailor the application to your specific needs.

55
docs/pages/Models/cloud-providers.mdx Normal file
View File

@@ -0,0 +1,55 @@
---
title: Connecting DocsGPT to Cloud LLM Providers
description: Connect DocsGPT to various Cloud Large Language Model (LLM) providers to power your document Q&A.
---
# Connecting DocsGPT to Cloud LLM Providers
DocsGPT is designed to seamlessly integrate with a variety of Cloud Large Language Model (LLM) providers, giving you access to state-of-the-art AI models for document question answering.
## Configuration via `.env` file
The primary method for configuring your LLM provider in DocsGPT is through the `.env` file. For a comprehensive understanding of all available settings, please refer to the detailed [DocsGPT Settings Guide](/Deploying/DocsGPT-Settings).
To connect to a cloud LLM provider, you will typically need to configure the following basic settings in your `.env` file:
* **`LLM_NAME`**: This setting is essential and identifies the specific cloud provider you wish to use (e.g., `openai`, `google`, `anthropic`).
* **`MODEL_NAME`**: Specifies the exact model you want to utilize from your chosen provider (e.g., `gpt-4o`, `gemini-2.0-flash`, `claude-3-5-sonnet-latest`). Refer to your provider's documentation for a list of available models.
* **`API_KEY`**: Almost all cloud LLM providers require an API key for authentication. Obtain your API key from your chosen provider's platform and securely store it in your `.env` file.
## Explicitly Supported Cloud Providers
DocsGPT offers direct, streamlined support for the following cloud LLM providers, making configuration straightforward. The table below outlines the `LLM_NAME` and example `MODEL_NAME` values to use for each provider in your `.env` file.
| Provider | `LLM_NAME` | Example `MODEL_NAME` |
| :--------------------------- | :------------- | :-------------------------- |
| DocsGPT Public API | `docsgpt` | `None` |
| OpenAI | `openai` | `gpt-4o` |
| Google (Vertex AI, Gemini) | `google` | `gemini-2.0-flash` |
| Anthropic (Claude) | `anthropic` | `claude-3-5-sonnet-latest` |
| Groq | `groq` | `llama-3.1-8b-instant` |
| HuggingFace Inference API | `huggingface` | `meta-llama/Llama-3.1-8B-Instruct` |
| Azure OpenAI | `azure_openai` | `gpt-4o` |
## Connecting to OpenAI-Compatible Cloud APIs
DocsGPT's flexible architecture allows you to connect to any cloud provider that offers an API compatible with the OpenAI API standard. This opens up a vast ecosystem of LLM services.
To connect to an OpenAI-compatible cloud provider, you will still use `LLM_NAME=openai` in your `.env` file. However, you will also need to specify the API endpoint of your chosen provider using the `OPENAI_BASE_URL` setting. You will also likely need to provide an `API_KEY` and `MODEL_NAME` as required by that provider.
**Example for DeepSeek (OpenAI-Compatible API):**
To connect to DeepSeek, which offers an OpenAI-compatible API, your `.env` file could be configured as follows:
```
LLM_NAME=openai
API_KEY=YOUR_API_KEY # Your DeepSeek API key
MODEL_NAME=deepseek-chat # Or your desired DeepSeek model name
OPENAI_BASE_URL=https://api.deepseek.com/v1 # DeepSeek's OpenAI API URL
```
Remember to consult the documentation of your chosen OpenAI-compatible cloud provider for their specific API endpoint, required model names, and authentication methods.
## Adding Support for Other Cloud Providers
If you wish to connect to a cloud provider that is not explicitly listed above or doesn't offer OpenAI API compatibility, you can extend DocsGPT to support it. Within the DocsGPT repository, navigate to the `application/llm` directory. Here, you will find Python files defining the existing LLM integrations. You can use these files as examples to create a new module for your desired cloud provider. After creating your new LLM module, you will need to register it within the `llm_creator.py` file. This process involves some coding, but it allows for virtually unlimited extensibility to connect to any cloud-based LLM service with an accessible API.

72
docs/pages/Models/embeddings.md
View File

@@ -0,0 +1,72 @@
---
title: Understanding and Configuring Embedding Models in DocsGPT
description: Learn about embedding models, their importance in DocsGPT, and how to configure them for optimal performance.
---
# Understanding and Configuring Embedding Models in DocsGPT
Embedding models are a crucial component of DocsGPT, enabling its powerful document understanding and question-answering capabilities. This guide will explain what embedding models are, why they are essential for DocsGPT, and how to configure them.
## What are Embedding Models?
In simple terms, an embedding model is a type of language model that converts text into numerical vectors. These vectors, known as embeddings, capture the semantic meaning of the text. Think of it as translating words and sentences into a language that computers can understand mathematically, where similar meanings are represented by vectors that are close to each other in vector space.
**Why are embedding models important for DocsGPT?**
DocsGPT uses embedding models for several key tasks:
* **Semantic Search:** When you upload documents to DocsGPT, the application uses an embedding model to generate embeddings for each document chunk. These embeddings are stored in a vector store. When you ask a question, your query is also converted into an embedding. DocsGPT then performs a semantic search in the vector store, finding document chunks whose embeddings are most similar to your query embedding. This allows DocsGPT to retrieve relevant information based on the *meaning* of your question and documents, not just keyword matching.
* **Document Understanding:** Embeddings help DocsGPT understand the underlying meaning of your documents, enabling it to answer questions accurately and contextually, even if the exact keywords from your question are not present in the retrieved document chunks.
In essence, embedding models are the bridge that allows DocsGPT to understand the nuances of human language and connect your questions to the relevant information within your documents.
## Out-of-the-Box Embedding Model Support in DocsGPT
DocsGPT is designed to be flexible and supports a wide range of embedding models right out of the box. Currently, DocsGPT provides native support for models from two major sources:
* **Sentence Transformers:** DocsGPT supports all models available through the [Sentence Transformers library](https://www.sbert.net/). This library offers a vast selection of pre-trained embedding models, known for their quality and efficiency in various semantic tasks.
* **OpenAI Embeddings:** DocsGPT also supports using embedding models from OpenAI, specifically the `text-embedding-ada-002` model, which is a powerful and widely used embedding model from OpenAI's API.
## Configuring Sentence Transformer Models
To utilize Sentence Transformer models within DocsGPT, you need to follow these steps:
1. **Download the Model:** Sentence Transformer models are typically hosted on Hugging Face Model Hub. You need to download your chosen model and place it in the `model/` folder in the root directory of your DocsGPT project.
For example, to use the `all-mpnet-base-v2` model, you would set `EMBEDDINGS_NAME` as described below, and ensure that the model files are available locally (DocsGPT will attempt to download it if it's not found, but local download is recommended for development and offline use).
2. **Set `EMBEDDINGS_NAME` in `.env` (or `settings.py`):** You need to configure the `EMBEDDINGS_NAME` setting in your `.env` file (or `settings.py`) to point to the desired Sentence Transformer model.
* **Using a pre-downloaded model from `model/` folder:** You can specify a path to the downloaded model within the `model/` directory. For instance, if you downloaded `all-mpnet-base-v2` and it's in `model/all-mpnet-base-v2`, you could potentially use a relative path like (though direct path to the model name is usually sufficient):
```
EMBEDDINGS_NAME=huggingface_sentence-transformers/all-mpnet-base-v2
```
or simply use the model identifier:
```
EMBEDDINGS_NAME=sentence-transformers/all-mpnet-base-v2
```
* **Using a model directly from Hugging Face Model Hub:** You can directly specify the model identifier from Hugging Face Model Hub:
```
EMBEDDINGS_NAME=huggingface_sentence-transformers/all-mpnet-base-v2
```
## Using OpenAI Embeddings
To use OpenAI's `text-embedding-ada-002` embedding model, you need to set `EMBEDDINGS_NAME` to `openai_text-embedding-ada-002` and ensure you have your OpenAI API key configured correctly via `API_KEY` in your `.env` file (if you are not using Azure OpenAI).
**Example `.env` configuration for OpenAI Embeddings:**
```
LLM_NAME=openai
API_KEY=YOUR_OPENAI_API_KEY # Your OpenAI API Key
EMBEDDINGS_NAME=openai_text-embedding-ada-002
```
## Adding Support for Other Embedding Models
If you wish to use an embedding model that is not supported out-of-the-box, a good starting point for adding custom embedding model support is to examine the `base.py` file located in the `application/vectorstore` directory.
Specifically, pay attention to the `EmbeddingsWrapper` and `EmbeddingsSingleton` classes. `EmbeddingsWrapper` provides a way to wrap different embedding model libraries into a consistent interface for DocsGPT. `EmbeddingsSingleton` manages the instantiation and retrieval of embedding model instances. By understanding these classes and the existing embedding model implementations, you can create your own custom integration for virtually any embedding model library you desire.

0
docs/pages/Models/local-inference.md
View File

44
docs/pages/Models/local-inference.mdx Normal file
View File

@@ -0,0 +1,44 @@
---
title: Connecting DocsGPT to Local Inference Engines
description: Connect DocsGPT to local inference engines for running LLMs directly on your hardware.
---
# Connecting DocsGPT to Local Inference Engines
DocsGPT can be configured to leverage local inference engines, allowing you to run Large Language Models directly on your own infrastructure. This approach offers enhanced privacy and control over your LLM processing.
Currently, DocsGPT primarily supports local inference engines that are compatible with the OpenAI API format. This means you can connect DocsGPT to various local LLM servers that mimic the OpenAI API structure.
## Configuration via `.env` file
Setting up a local inference engine with DocsGPT is configured through environment variables in the `.env` file. For a detailed explanation of all settings, please consult the [DocsGPT Settings Guide](/Deploying/DocsGPT-Settings).
To connect to a local inference engine, you will generally need to configure these settings in your `.env` file:
* **`LLM_NAME`**: Crucially set this to `openai`. This tells DocsGPT to use the OpenAI-compatible API format for communication, even though the LLM is local.
* **`MODEL_NAME`**: Specify the model name as recognized by your local inference engine. This might be a model identifier or left as `None` if the engine doesn't require explicit model naming in the API request.
* **`OPENAI_BASE_URL`**: This is essential. Set this to the base URL of your local inference engine's API endpoint. This tells DocsGPT where to find your local LLM server.
* **`API_KEY`**: Generally, for local inference engines, you can set `API_KEY=None` as authentication is usually not required in local setups.
## Supported Local Inference Engines (OpenAI API Compatible)
DocsGPT is readily configurable to work with the following local inference engines, all communicating via the OpenAI API format. Here are example `OPENAI_BASE_URL` values for each, based on default setups:
| Inference Engine | `LLM_NAME` | `OPENAI_BASE_URL` |
| :---------------------------- | :--------- | :------------------------- |
| LLaMa.cpp | `openai` | `http://localhost:8000/v1` |
| Ollama | `openai` | `http://localhost:11434/v1` |
| Text Generation Inference (TGI)| `openai` | `http://localhost:8080/v1` |
| SGLang | `openai` | `http://localhost:30000/v1` |
| vLLM | `openai` | `http://localhost:8000/v1` |
| Aphrodite | `openai` | `http://localhost:2242/v1` |
| FriendliAI | `openai` | `http://localhost:8997/v1` |
| LMDeploy | `openai` | `http://localhost:23333/v1` |
**Important Note on `localhost` vs `host.docker.internal`:**
The `OPENAI_BASE_URL` examples above use `http://localhost`. If you are running DocsGPT within Docker and your local inference engine is running on your host machine (outside of Docker), you will likely need to replace `localhost` with `http://host.docker.internal` to ensure Docker can correctly access your host's services. For example, `http://host.docker.internal:11434/v1` for Ollama.
## Adding Support for Other Local Engines
While DocsGPT currently focuses on OpenAI API compatible local engines, you can extend its capabilities to support other local inference solutions. To do this, navigate to the `application/llm` directory in the DocsGPT repository. Examine the existing Python files for examples of LLM integrations. You can create a new module for your desired local engine, and then register it in the `llm_creator.py` file within the same directory. This allows for custom integration with a wide range of local LLM servers beyond those listed above.