diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md index 512b6c23..ba590c87 100644 --- a/CODE_OF_CONDUCT.md +++ b/CODE_OF_CONDUCT.md @@ -2,58 +2,58 @@ ## Our Pledge -We as members, contributors, and leaders, pledge to make participation in our -community a harassment-free experience for everyone, regardless of age, body +We as members, contributors and leaders pledge to make participation in our +community, a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, -nationality, personal appearance, race, religion, or sexual identity +nationality, personal appearance, race, religion or sexual identity and orientation. We pledge to act and interact in ways that contribute to an open, welcoming, -diverse, inclusive, and a healthy community. +diverse, inclusive and a healthy community. ## Our Standards Examples of behavior that contribute to a positive environment for our community include: -* Demonstrating empathy and kindness towards other people -* Being respectful and open to differing opinions, viewpoints, and experiences -* Giving and gracefully accepting constructive feedback -* Taking accountability and offering apologies to those who have been impacted by our errors, +## Demonstrating empathy and kindness towards other people +1. Being respectful and open to differing opinions, viewpoints, and experiences +2. Giving and gracefully accepting constructive feedback +3. Taking accountability and offering apologies to those who have been impacted by our errors, while also gaining insights from the situation -* Focusing on what is best not just for us as individuals, but for the +4. Focusing on what is best not just for us as individuals but for the community as a whole Examples of unacceptable behavior include: -* The use of sexualized language or imagery, and sexual attention or +1. The use of sexualized language or imagery, and sexual attention or advances of any kind -* Trolling, insulting or derogatory comments, and personal or political attacks -* Public or private harassment -* Publishing other's private information, such as a physical or email +2. Trolling, insulting or derogatory comments, and personal or political attacks +3. Public or private harassment +4. Publishing other's private information, such as a physical or email address, without their explicit permission -* Other conduct which could reasonably be considered inappropriate in a +5. Other conduct which could reasonably be considered inappropriate in a professional setting ## Enforcement Responsibilities Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in -response to any behavior that they deem inappropriate, threatening, offensive, +response to any behavior that they deem inappropriate, threatening, offensive or harmful. Community leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are -not aligned to this Code of Conduct, and will communicate reasons for moderation +not aligned to this Code of Conduct and will communicate reasons for moderation decisions when appropriate. ## Scope -This Code of Conduct applies within all community spaces, and also applies when +This Code of Conduct applies within all community spaces and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official e-mail address, -posting via an official social media account, or acting as an appointed +posting via an official social media account or acting as an appointed representative at an online or offline event. ## Enforcement @@ -63,29 +63,27 @@ reported to the community leaders responsible for enforcement at contact@arc53.com. All complaints will be reviewed and investigated promptly and fairly. -All community leaders are obligated to respect the privacy and security of the +All community leaders are obligated to be respectful towards the privacy and security of the reporter of any incident. ## Enforcement Guidelines Community leaders will follow these Community Impact Guidelines in determining -the consequences for any action they deem in violation of this Code of Conduct: +the consequences for any action that they deem in violation of this Code of Conduct: ### 1. Correction - -**Community Impact**: Use of inappropriate language or other behavior deemed +* **Community Impact**: Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community space. -**Consequence**: A private, written warning from community leaders, providing +* **Consequence**: A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested. ### 2. Warning - -**Community Impact**: A violation through a single incident or series +* **Community Impact**: A violation through a single incident or series of actions. -**Consequence**: A warning with consequences for continued behavior. No +* **Consequence**: A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels @@ -93,23 +91,21 @@ like social media. Violating these terms may lead to a temporary or permanent ban. ### 3. Temporary Ban - -**Community Impact**: A serious violation of community standards, including +* **Community Impact**: A serious violation of community standards, including sustained inappropriate behavior. -**Consequence**: A temporary ban from any sort of interaction or public +* **Consequence**: A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban. ### 4. Permanent Ban +* **Community Impact**: Demonstrating a pattern of violation of community +standards, including sustained inappropriate behavior,harassment of an +individual or aggression towards or disparagement of classes of individuals. -**Community Impact**: Demonstrating a pattern of violation of community -standards, including sustained inappropriate behavior, harassment of an -individual, or aggression towards or disparagement of classes of individuals. - -**Consequence**: A permanent ban from any sort of public interaction within +* **Consequence**: A permanent ban from any sort of public interaction within the community. ## Attribution diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 3407fdc9..5cfd1206 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -17,12 +17,18 @@ Thank you for choosing to contribute to DocsGPT! We are all very grateful! ## 🐞 Issues and Pull requests -We value contributions in the form of discussions or suggestions. We recommend taking a look at existing issues and our [roadmap](https://github.com/orgs/arc53/projects/2). +- We value contributions in the form of discussions or suggestions. We recommend taking a look at existing issues and our [roadmap](https://github.com/orgs/arc53/projects/2). + +- If you're interested in contributing code, here are some important things to know: + +- We have a frontend built on React (Vite) and a backend in Python. +======= Before creating issues, please check out how the latest version of our app looks and works by launching it via [Quickstart](https://github.com/arc53/DocsGPT#quickstart) the version on our live demo is slightly modified with login. Your issues should relate to the version that you can launch via [Quickstart](https://github.com/arc53/DocsGPT#quickstart). ### πŸ‘¨β€πŸ’» If you're interested in contributing code, here are some important things to know: + Tech Stack Overview: - 🌐 Frontend: Built with React (Vite) βš›οΈ, @@ -57,42 +63,55 @@ Here's a step-by-step guide on how to contribute to DocsGPT: 1. **Fork the Repository:** - Click the "Fork" button at the top-right of this repository to create your fork. -2. **Create and Switch to a New Branch:** +2. **Clone the Forked Repository:** + - Clone the repository using: + ``` shell + git clone https://github.com//DocsGPT.git + ``` + +3. **Keep your Fork in Sync:** + - Before you make any changes, make sure that your fork is in sync to avoid merge conflicts using: + ```shell + git remote add upstream https://github.com/arc53/DocsGPT.git + git pull upstream main + ``` + +4. **Create and Switch to a New Branch:** - Create a new branch for your contribution using: ```shell git checkout -b your-branch-name ``` -3. **Make Changes:** +5. **Make Changes:** - Make the required changes in your branch. -4. **Add Changes to the Staging Area:** +6. **Add Changes to the Staging Area:** - Add your changes to the staging area using: ```shell git add . ``` -5. **Commit Your Changes:** +7. **Commit Your Changes:** - Commit your changes with a descriptive commit message using: ```shell git commit -m "Your descriptive commit message" ``` -6. **Push Your Changes to the Remote Repository:** +8. **Push Your Changes to the Remote Repository:** - Push your branch with changes to your fork on GitHub using: ```shell git push origin your-branch-name ``` -7. **Submit a Pull Request (PR):** +9. **Submit a Pull Request (PR):** - Create a Pull Request from your branch to the main repository. Make sure to include a detailed description of your changes and reference any related issues. -8. **Collaborate:** +10. **Collaborate:** - Be responsive to comments and feedback on your PR. - Make necessary updates as suggested. - Once your PR is approved, it will be merged into the main repository. -9. **Testing:** +11. **Testing:** - Before submitting a Pull Request, ensure your code passes all unit tests. - To run unit tests from the root of the repository, execute: ```shell @@ -101,7 +120,7 @@ Here's a step-by-step guide on how to contribute to DocsGPT: *Note: You should run the unit test only after making the changes to the backend code.* -10. **Questions and Collaboration:** +12. **Questions and Collaboration:** - Feel free to join our Discord. We're very friendly and welcoming to new contributors, so don't hesitate to reach out. Thank you for considering contributing to DocsGPT! πŸ™ diff --git a/README.md b/README.md index 8105e67e..4bc9d199 100644 --- a/README.md +++ b/README.md @@ -14,10 +14,12 @@ Say goodbye to time-consuming manual searches, and let - ![example1](https://img.shields.io/github/stars/arc53/docsgpt?style=social) - ![example2](https://img.shields.io/github/forks/arc53/docsgpt?style=social) - ![example3](https://img.shields.io/github/license/arc53/docsgpt) - ![example3](https://img.shields.io/discord/1070046503302877216) + ![link to main GitHub showing Stars number](https://img.shields.io/github/stars/arc53/docsgpt?style=social) + ![link to main GitHub showing Forks number](https://img.shields.io/github/forks/arc53/docsgpt?style=social) + ![link to license file](https://img.shields.io/github/license/arc53/docsgpt) + ![link to discord](https://img.shields.io/discord/1070046503302877216) + ![X (formerly Twitter) URL](https://img.shields.io/twitter/follow/ATushynski) + @@ -25,10 +27,10 @@ Say goodbye to time-consuming manual searches, and let Vite and React. ## QuickStart -Note: Make sure you have Docker installed +Note: Make sure you have [Docker](https://docs.docker.com/engine/install/) installed On Mac OS or Linux, write: @@ -87,7 +89,7 @@ It will install all the dependencies and allow you to download the local model o Otherwise, refer to this Guide: 1. Download and open this repository with `git clone https://github.com/arc53/DocsGPT.git` -2. Create a `.env` file in your root directory and set the env variable `OPENAI_API_KEY` with your [OpenAI API key](https://platform.openai.com/account/api-keys) and `VITE_API_STREAMING` to true or false, depending on whether you want streaming answers or not. +2. Create a `.env` file in your root directory and set the env variable `API_KEY` with your [OpenAI API key](https://platform.openai.com/account/api-keys) and `VITE_API_STREAMING` to true or false, depending on whether you want streaming answers or not. It should look like this inside: ``` @@ -148,7 +150,7 @@ python -m venv venv pip install -r requirements.txt ``` -4. Run the app using `flask run --host=0.0.0.0 --port=7091`. +4. Run the app using `flask --app application/app.py run --host=0.0.0.0 --port=7091`. 5. Start worker with `celery -A application.app.celery worker -l INFO`. ### Start frontend @@ -174,9 +176,9 @@ Please refer to the [CONTRIBUTING.md](CONTRIBUTING.md) file for information abou We as members, contributors, and leaders, pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation. Please refer to the [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md) file for more information about contributing. -## Many Thanks To Our Contributors +## Many Thanks To Our Contributors⚑ - + Contributors @@ -184,4 +186,4 @@ We as members, contributors, and leaders, pledge to make participation in our co The source code license is [MIT](https://opensource.org/license/mit/), as described in the [LICENSE](LICENSE) file. -Built with [πŸ¦œοΈπŸ”— LangChain](https://github.com/hwchase17/langchain) +Built with [:bird: :link: LangChain](https://github.com/hwchase17/langchain) diff --git a/application/api/answer/routes.py b/application/api/answer/routes.py index b694c4aa..a9327255 100644 --- a/application/api/answer/routes.py +++ b/application/api/answer/routes.py @@ -29,6 +29,8 @@ answer = Blueprint('answer', __name__) if settings.LLM_NAME == "gpt4": gpt_model = 'gpt-4' +elif settings.LLM_NAME == "anthropic": + gpt_model = 'claude-2' else: gpt_model = 'gpt-3.5-turbo' diff --git a/application/core/settings.py b/application/core/settings.py index 116735a6..25a3295d 100644 --- a/application/core/settings.py +++ b/application/core/settings.py @@ -19,7 +19,7 @@ class Settings(BaseSettings): API_URL: str = "http://localhost:7091" # backend url for celery worker API_KEY: str = None # LLM api key - EMBEDDINGS_KEY: str = None # api key for embeddings (if using openai, just copy API_KEY + EMBEDDINGS_KEY: str = None # api key for embeddings (if using openai, just copy API_KEY) OPENAI_API_BASE: str = None # azure openai api base url OPENAI_API_VERSION: str = None # azure openai api version AZURE_DEPLOYMENT_NAME: str = None # azure deployment name for answering diff --git a/application/llm/anthropic.py b/application/llm/anthropic.py new file mode 100644 index 00000000..a64d71e9 --- /dev/null +++ b/application/llm/anthropic.py @@ -0,0 +1,40 @@ +from application.llm.base import BaseLLM +from application.core.settings import settings + +class AnthropicLLM(BaseLLM): + + def __init__(self, api_key=None): + from anthropic import Anthropic, HUMAN_PROMPT, AI_PROMPT + self.api_key = api_key or settings.ANTHROPIC_API_KEY # If not provided, use a default from settings + self.anthropic = Anthropic(api_key=self.api_key) + self.HUMAN_PROMPT = HUMAN_PROMPT + self.AI_PROMPT = AI_PROMPT + + def gen(self, model, messages, engine=None, max_tokens=300, stream=False, **kwargs): + context = messages[0]['content'] + user_question = messages[-1]['content'] + prompt = f"### Context \n {context} \n ### Question \n {user_question}" + if stream: + return self.gen_stream(model, prompt, max_tokens, **kwargs) + + completion = self.anthropic.completions.create( + model=model, + max_tokens_to_sample=max_tokens, + stream=stream, + prompt=f"{self.HUMAN_PROMPT} {prompt}{self.AI_PROMPT}", + ) + return completion.completion + + def gen_stream(self, model, messages, engine=None, max_tokens=300, **kwargs): + context = messages[0]['content'] + user_question = messages[-1]['content'] + prompt = f"### Context \n {context} \n ### Question \n {user_question}" + stream_response = self.anthropic.completions.create( + model=model, + prompt=f"{self.HUMAN_PROMPT} {prompt}{self.AI_PROMPT}", + max_tokens_to_sample=max_tokens, + stream=True, + ) + + for completion in stream_response: + yield completion.completion \ No newline at end of file diff --git a/application/llm/llm_creator.py b/application/llm/llm_creator.py index 6a60f1b6..bbc2c792 100644 --- a/application/llm/llm_creator.py +++ b/application/llm/llm_creator.py @@ -2,6 +2,7 @@ from application.llm.openai import OpenAILLM, AzureOpenAILLM from application.llm.sagemaker import SagemakerAPILLM from application.llm.huggingface import HuggingFaceLLM from application.llm.llama_cpp import LlamaCpp +from application.llm.anthropic import AnthropicLLM @@ -11,7 +12,8 @@ class LLMCreator: 'azure_openai': AzureOpenAILLM, 'sagemaker': SagemakerAPILLM, 'huggingface': HuggingFaceLLM, - 'llama.cpp': LlamaCpp + 'llama.cpp': LlamaCpp, + 'anthropic': AnthropicLLM } @classmethod diff --git a/application/parser/file/openapi3_parser.py b/application/parser/file/openapi3_parser.py new file mode 100644 index 00000000..3c5082fa --- /dev/null +++ b/application/parser/file/openapi3_parser.py @@ -0,0 +1,51 @@ +from urllib.parse import urlparse + +from openapi_parser import parse + +try: + from application.parser.file.base_parser import BaseParser +except ModuleNotFoundError: + from base_parser import BaseParser + + +class OpenAPI3Parser(BaseParser): + def init_parser(self) -> None: + return super().init_parser() + + def get_base_urls(self, urls): + base_urls = [] + for i in urls: + parsed_url = urlparse(i) + base_url = parsed_url.scheme + "://" + parsed_url.netloc + if base_url not in base_urls: + base_urls.append(base_url) + return base_urls + + def get_info_from_paths(self, path): + info = "" + if path.operations: + for operation in path.operations: + info += ( + f"\n{operation.method.value}=" + f"{operation.responses[0].description}" + ) + return info + + def parse_file(self, file_path): + data = parse(file_path) + results = "" + base_urls = self.get_base_urls(link.url for link in data.servers) + base_urls = ",".join([base_url for base_url in base_urls]) + results += f"Base URL:{base_urls}\n" + i = 1 + for path in data.paths: + info = self.get_info_from_paths(path) + results += ( + f"Path{i}: {path.url}\n" + f"description: {path.description}\n" + f"parameters: {path.parameters}\nmethods: {info}\n" + ) + i += 1 + with open("results.txt", "w") as f: + f.write(results) + return results diff --git a/application/requirements.txt b/application/requirements.txt index 9c60e421..b6f5f8d0 100644 --- a/application/requirements.txt +++ b/application/requirements.txt @@ -4,6 +4,7 @@ aiohttp-retry==2.8.3 aiosignal==1.3.1 aleph-alpha-client==2.16.1 amqp==5.1.1 +anthropic==0.5.0 async-timeout==4.0.2 attrs==22.2.0 billiard==3.6.4.0 @@ -57,6 +58,7 @@ nltk==3.8.1 numcodecs==0.11.0 numpy==1.24.2 openai==0.27.8 +openapi3-parser==1.1.14 packaging==23.0 pathos==0.3.0 Pillow==10.0.1 @@ -100,7 +102,7 @@ transformers==4.30.0 typer==0.7.0 typing-inspect==0.8.0 typing_extensions==4.5.0 -urllib3==1.26.17 +urllib3==1.26.18 vine==5.0.0 wcwidth==0.2.6 yarl==1.8.2 diff --git a/application/worker.py b/application/worker.py index 71fcd615..ae8f240c 100644 --- a/application/worker.py +++ b/application/worker.py @@ -20,17 +20,34 @@ except FileExistsError: pass +# Define a function to extract metadata from a given filename. def metadata_from_filename(title): store = '/'.join(title.split('/')[1:3]) return {'title': title, 'store': store} +# Define a function to generate a random string of a given length. def generate_random_string(length): return ''.join([string.ascii_letters[i % 52] for i in range(length)]) current_dir = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +# Define the main function for ingesting and processing documents. def ingest_worker(self, directory, formats, name_job, filename, user): + """ + Ingest and process documents. + + Args: + self: Reference to the instance of the task. + directory (str): Specifies the directory for ingesting ('inputs' or 'temp'). + formats (list of str): List of file extensions to consider for ingestion (e.g., [".rst", ".md"]). + name_job (str): Name of the job for this ingestion task. + filename (str): Name of the file to be ingested. + user (str): Identifier for the user initiating the ingestion. + + Returns: + dict: Information about the completed ingestion task, including input parameters and a "limited" flag. + """ # directory = 'inputs' or 'temp' # formats = [".rst", ".md"] input_files = None diff --git a/codecov.yml b/codecov.yml index 9fa3a4e6..3b859168 100644 --- a/codecov.yml +++ b/codecov.yml @@ -1,2 +1,2 @@ ignore: - - "*/tests/*” \ No newline at end of file + - "*/tests/*" \ No newline at end of file diff --git a/docker-compose.yaml b/docker-compose.yaml index 84cc5681..7008b53d 100644 --- a/docker-compose.yaml +++ b/docker-compose.yaml @@ -16,6 +16,7 @@ services: environment: - API_KEY=$API_KEY - EMBEDDINGS_KEY=$API_KEY + - LLM_NAME=$LLM_NAME - CELERY_BROKER_URL=redis://redis:6379/0 - CELERY_RESULT_BACKEND=redis://redis:6379/1 - MONGO_URI=mongodb://mongo:27017/docsgpt @@ -35,6 +36,7 @@ services: environment: - API_KEY=$API_KEY - EMBEDDINGS_KEY=$API_KEY + - LLM_NAME=$LLM_NAME - CELERY_BROKER_URL=redis://redis:6379/0 - CELERY_RESULT_BACKEND=redis://redis:6379/1 - MONGO_URI=mongodb://mongo:27017/docsgpt diff --git a/docs/README.md b/docs/README.md index 4e41a0de..4b90b598 100644 --- a/docs/README.md +++ b/docs/README.md @@ -4,16 +4,15 @@ ### 1. Clone the DocsGPT repository: -``` +```bash git clone https://github.com/arc53/DocsGPT.git - ``` ### 2. Navigate to the docs folder: -``` +```bash cd DocsGPT/docs - ``` + The docs folder contains the markdown files that make up the documentation. The majority of the files are in the pages directory. Some notable files in this folder include: `index.mdx`: The main documentation file. @@ -22,30 +21,29 @@ The docs folder contains the markdown files that make up the documentation. The ### 3. Verify that you have Node.js and npm installed in your system. You can check by running: -``` +```bash node --version npm --version - ``` + ### 4. If not installed, download Node.js and npm from the respective official websites. ### 5. Once you have Node.js and npm running, proceed to install yarn - another package manager that helps to manage project dependencies: -``` +```bash npm install --global yarn - ``` + ### 6. Install the project dependencies using yarn: -``` +```bash yarn install - ``` + ### 7. After the successful installation of the project dependencies, start the local server: -``` +```bash yarn dev - ``` - Now, you should be able to view the docs on your local environment by visiting `http://localhost:5000`. You can explore the different markdown files and make changes as you see fit. diff --git a/docs/pages/Deploying/Hosting-the-app.md b/docs/pages/Deploying/Hosting-the-app.md index 31c3f55a..9550e810 100644 --- a/docs/pages/Deploying/Hosting-the-app.md +++ b/docs/pages/Deploying/Hosting-the-app.md @@ -1,6 +1,6 @@ # Self-hosting DocsGPT on Amazon Lightsail -Here's a step-by-step guide on how to setup an Amazon Lightsail instance to host DocsGPT. +Here's a step-by-step guide on how to set up an Amazon Lightsail instance to host DocsGPT. ## Configuring your instance diff --git a/docs/pages/Deploying/Quickstart.md b/docs/pages/Deploying/Quickstart.md index 411e3234..cb5fcf38 100644 --- a/docs/pages/Deploying/Quickstart.md +++ b/docs/pages/Deploying/Quickstart.md @@ -2,22 +2,28 @@ **Note**: Make sure you have Docker installed **On macOS or Linux:** -Just run the following command:: +Just run the following command: -`./setup.sh` +```bash +./setup.sh +``` This command will install all the necessary dependencies and provide you with an option to download the local model or use OpenAI. If you prefer to follow manual steps, refer to this guide: 1. Open and download this repository with -`git clone https://github.com/arc53/DocsGPT.git`. + ```bash + git clone https://github.com/arc53/DocsGPT.git + ``` 2. Create a `.env` file in your root directory and set your `API_KEY` with your [OpenAI API key](https://platform.openai.com/account/api-keys). 3. Run the following commands: -`docker-compose build && docker-compose up`. -4. Navigate to `http://localhost:5173/`. + ```bash + docker-compose build && docker-compose up + ``` +4. Navigate to http://localhost:5173/. -To stop, simply press Ctrl + C. +To stop, simply press **Ctrl + C**. **For WINDOWS:** @@ -28,32 +34,36 @@ To run the setup on Windows, you have two options: using the Windows Subsystem f 1. Install WSL if you haven't already. You can follow the official Microsoft documentation for installation: (https://learn.microsoft.com/en-us/windows/wsl/install). 2. After setting up WSL, open the WSL terminal. 3. Clone the repository and create the `.env` file: - ``` + ```bash git clone https://github.com/arc53/DocsGPT.git cd DocsGPT echo "API_KEY=Yourkey" > .env echo "VITE_API_STREAMING=true" >> .env ``` 4. Run the following command to start the setup with Docker Compose: - `./run-with-docker-compose.sh` -5. Open your web browser and navigate to (http://localhost:5173/). -6. To stop the setup, just press `Ctrl + C` in the WSL terminal + ```bash + ./run-with-docker-compose.sh + ``` +6. Open your web browser and navigate to http://localhost:5173/. +7. To stop the setup, just press **Ctrl + C** in the WSL terminal **Option 2: Using Git Bash or Command Prompt (CMD):** 1. Install Git for Windows if you haven't already. Download it from the official website: (https://gitforwindows.org/). 2. Open Git Bash or Command Prompt. 3. Clone the repository and create the `.env` file: - ``` + ```bash git clone https://github.com/arc53/DocsGPT.git cd DocsGPT echo "API_KEY=Yourkey" > .env echo "VITE_API_STREAMING=true" >> .env ``` -4.Run the following command to start the setup with Docker Compose: - `./run-with-docker-compose.sh` -5.Open your web browser and navigate to (http://localhost:5173/). -6.To stop the setup, just press Ctrl + C in the Git Bash or Command Prompt terminal. +4. Run the following command to start the setup with Docker Compose: + ```bash + ./run-with-docker-compose.sh + ``` +5. Open your web browser and navigate to http://localhost:5173/. +6. To stop the setup, just press **Ctrl + C** in the Git Bash or Command Prompt terminal. These steps should help you set up and run the project on Windows using either WSL or Git Bash/Command Prompt. **Important:** Ensure that Docker is installed and properly configured on your Windows system for these steps to work. @@ -68,32 +78,36 @@ Option 1: Using Windows Subsystem for Linux (WSL): 1. Install WSL if you haven't already. You can follow the official Microsoft documentation for installation: (https://learn.microsoft.com/en-us/windows/wsl/install). 2. After setting up WSL, open the WSL terminal. 3. Clone the repository and create the `.env` file: - ``` + ```bash git clone https://github.com/arc53/DocsGPT.git cd DocsGPT echo "API_KEY=Yourkey" > .env echo "VITE_API_STREAMING=true" >> .env ``` 4. Run the following command to start the setup with Docker Compose: - `./run-with-docker-compose.sh` -5. Open your web browser and navigate to (http://localhost:5173/). -6. To stop the setup, just press `Ctrl + C` in the WSL terminal + ```bash + ./run-with-docker-compose.sh + ``` +5. Open your web browser and navigate to http://localhost:5173/. +6. To stop the setup, just press **Ctrl + C** in the WSL terminal. Option 2: Using Git Bash or Command Prompt (CMD): 1. Install Git for Windows if you haven't already. You can download it from the official website: (https://gitforwindows.org/). 2. Open Git Bash or Command Prompt. 3. Clone the repository and create the `.env` file: - ``` + ```bash git clone https://github.com/arc53/DocsGPT.git cd DocsGPT echo "API_KEY=Yourkey" > .env echo "VITE_API_STREAMING=true" >> .env ``` -4.Run the following command to start the setup with Docker Compose: - `./run-with-docker-compose.sh` -5.Open your web browser and navigate to (http://localhost:5173/). -6.To stop the setup, just press Ctrl + C in the Git Bash or Command Prompt terminal. +4. Run the following command to start the setup with Docker Compose: + ```bash + ./run-with-docker-compose.sh + ``` +5. Open your web browser and navigate to http://localhost:5173/. +6. To stop the setup, just press **Ctrl + C** in the Git Bash or Command Prompt terminal. These steps should help you set up and run the project on Windows using either WSL or Git Bash/Command Prompt. Make sure you have Docker installed and properly configured on your Windows system for this to work. @@ -103,12 +117,12 @@ These steps should help you set up and run the project on Windows using either W #### Installing the Chrome extension: To enhance your DocsGPT experience, you can install the DocsGPT Chrome extension. Here's how: -1. In the DocsGPT GitHub repository, click on the "Code" button and select "Download ZIP". +1. In the DocsGPT GitHub repository, click on the **Code** button and select **Download ZIP**. 2. Unzip the downloaded file to a location you can easily access. 3. Open the Google Chrome browser and click on the three dots menu (upper right corner). -4. Select "More Tools" and then "Extensions". -5. Turn on the "Developer mode" switch in the top right corner of the Extensions page. -6. Click on the "Load unpacked" button. -7. Select the "Chrome" folder where the DocsGPT files have been unzipped (docsgpt-main > extensions > chrome). +4. Select **More Tools** and then **Extensions**. +5. Turn on the **Developer mode** switch in the top right corner of the **Extensions page**. +6. Click on the **Load unpacked** button. +7. Select the **Chrome** folder where the DocsGPT files have been unzipped (docsgpt-main > extensions > chrome). 8. The extension should now be added to Google Chrome and can be managed on the Extensions page. -9. To disable or remove the extension, simply turn off the toggle switch on the extension card or click the "Remove" button. +9. To disable or remove the extension, simply turn off the toggle switch on the extension card or click the **Remove** button. diff --git a/docs/pages/Deploying/Railway-Deploying.md b/docs/pages/Deploying/Railway-Deploying.md new file mode 100644 index 00000000..34937b3d --- /dev/null +++ b/docs/pages/Deploying/Railway-Deploying.md @@ -0,0 +1,254 @@ + +# Self-hosting DocsGPT on Railway + + + +Here's a step-by-step guide on how to host DocsGPT on Railway App. + + + +At first Clone and setup the project locally to run , test and Modify. + + + +### 1. Clone and GitHub SetUp + +a. Open Terminal (Windows Shell or Git bash(recommended)). + + + +b. Type `git clone https://github.com/arc53/DocsGPT.git` + + + +#### Download the package information + + + +Once it has finished cloning the repository, it is time to download the package information from all sources. To do so, simply enter the following command: + + + +`sudo apt update` + + + +#### Install Docker and Docker Compose + + + +DocsGPT backend and worker use Python, Frontend is written on React and the whole application is containerized using Docker. To install Docker and Docker Compose, enter the following commands: + + + +`sudo apt install docker.io` + + + +And now install docker-compose: + + + +`sudo apt install docker-compose` + + + +#### Access the DocsGPT Folder + + + +Enter the following command to access the folder in which the DocsGPT docker-compose file is present. + + + +`cd DocsGPT/` + + + +#### Prepare the Environment + + + +Inside the DocsGPT folder create a `.env` file and copy the contents of `.env_sample` into it. + + + +`nano .env` + + + +Make sure your `.env` file looks like this: + + + +``` + +OPENAI_API_KEY=(Your OpenAI API key) + +VITE_API_STREAMING=true + +SELF_HOSTED_MODEL=false + +``` + + + +To save the file, press CTRL+X, then Y, and then ENTER. + + + +Next, set the correct IP for the Backend by opening the docker-compose.yml file: + + + +`nano docker-compose.yml` + + + +And Change line 7 to: `VITE_API_HOST=http://localhost:7091` + +to this `VITE_API_HOST=http://:7091` + + + +This will allow the frontend to connect to the backend. + + + +#### Running the Application + + + +You're almost there! Now that all the necessary bits and pieces have been installed, it is time to run the application. To do so, use the following command: + + + +`sudo docker-compose up -d` + + + +Launching it for the first time will take a few minutes to download all the necessary dependencies and build. + + + +Once this is done you can go ahead and close the terminal window. + + + +### 2. Pushing it to your own Repository + + + +a. Create a Repository on your GitHub. + + + +b. Open Terminal in the same directory of the Cloned project. + + + +c. Type `git init` + + + +d. `git add .` + + + +e. `git commit -m "first-commit"` + + + +f. `git remote add origin ` + + + +g. `git push git push --set-upstream origin master` + +Your local files will now be pushed to your GitHub Account. :) + + +### 3. Create a Railway Account: + + + +If you haven't already, create or log in to your railway account do it by visiting [Railway](https://railway.app/) + + + +Signup via **GitHub** [Recommended]. + + + +### 4. Start New Project: + + + +a. Open Railway app and Click on "Start New Project." + + + +b. Choose any from the list of options available (Recommended "**Deploy from GitHub Repo**") + + + +c. Choose the required Repository from your GitHub. + + + +d. Configure and allow access to modify your GitHub content from the pop-up window. + + + +e. Agree to all the terms and conditions. + + + +PS: It may take a few minutes for the account setup to complete. + + + +#### You will get A free trial of $5 (use it for trial and then purchase if satisfied and needed) + + + +### 5. Connecting to Your newly Railway app with GitHub + + + +a. Choose DocsGPT repo from the list of your GitHub repository that you want to deploy now. + + + +b. Click on Deploy now. + + + +![Three Tabs will be there](/Railway-selection.png) + + + +c. Select Variables Tab. + + + +d. Upload the env file here that you used for local setup. + + + +e. Go to Settings Tab now. + + + +f. Go to "Networking" and click on Generate Domain Name, to get the URL of your hosted project. + + + +g. You can update the Root directory, build command, installation command as per need. + +*[However recommended not the disturb these options and leave them as default if not that needed.]* + + + + +Your own DocsGPT is now available at the Generated domain URl. :) diff --git a/docs/pages/Deploying/_meta.json b/docs/pages/Deploying/_meta.json index 27090f4b..0e91f35e 100644 --- a/docs/pages/Deploying/_meta.json +++ b/docs/pages/Deploying/_meta.json @@ -6,5 +6,9 @@ "Quickstart": { "title": "⚑️Quickstart", "href": "/Deploying/Quickstart" + }, + "Railway-Deploying": { + "title": "πŸš‚Deploying on Rainway", + "href": "/Deploying/Railway-Deploying" } -} \ No newline at end of file +} diff --git a/docs/pages/Developing/API-docs.md b/docs/pages/Developing/API-docs.md index 09e4f875..5601ebc8 100644 --- a/docs/pages/Developing/API-docs.md +++ b/docs/pages/Developing/API-docs.md @@ -10,14 +10,16 @@ This endpoint is used to request answers to user-provided questions. **Request:** -Method: POST -Headers: Content-Type should be set to "application/json; charset=utf-8" -Request Body: JSON object with the following fields: -* **question:** The user's question -* **history:** (Optional) Previous conversation history -* **api_key:** Your API key -* **embeddings_key:** Your embeddings key -* **active_docs:** The location of active documentation +**Method**: `POST` + +**Headers**: Content-Type should be set to `application/json; charset=utf-8` + +**Request Body**: JSON object with the following fields: +* `question` β€” The user's question. +* `history` β€” (Optional) Previous conversation history. +* `api_key`β€” Your API key. +* `embeddings_key` β€” Your embeddings key. +* `active_docs` β€” The location of active documentation. Here is a JavaScript Fetch Request example: ```js @@ -36,12 +38,12 @@ fetch("http://127.0.0.1:5000/api/answer", { **Response** -In response, you will get a JSON document containing the answer,query and the result: +In response, you will get a JSON document containing the `answer`, `query` and `result`: ```json { - "answer": " Hi there! How can I help you?\n", + "answer": "Hi there! How can I help you?\n", "query": "Hi", - "result": " Hi there! How can I help you?\nSOURCES:" + "result": "Hi there! How can I help you?\nSOURCES:" } ``` @@ -53,9 +55,12 @@ This endpoint will make sure documentation is loaded on the server (just run it **Request:** -Headers: Content-Type should be set to "application/json; charset=utf-8" -Request Body: JSON object with the field: -* **docs:** The location of the documentation +**Method**: `POST` + +**Headers**: Content-Type should be set to `application/json; charset=utf-8` + +**Request Body**: JSON object with the field: +* `docs` β€” The location of the documentation: ```js // answer (POST http://127.0.0.1:5000/api/docs_check) fetch("http://127.0.0.1:5000/api/docs_check", { @@ -71,7 +76,7 @@ fetch("http://127.0.0.1:5000/api/docs_check", { **Response:** -In response, you will get a JSON document like this one indicating whether the documentation exists or not.: +In response, you will get a JSON document like this one indicating whether the documentation exists or not: ```json { "status": "exists" @@ -86,19 +91,25 @@ This endpoint provides information about available vectors and their locations w **Request:** -Method: GET +**Method**: `GET` **Response:** Response will include: -`date`, `description`, `docLink`, `fullName`, `language`, `location` (local or docshub), `model`, `name`, `version`. - +* `date` +* `description` +* `docLink` +* `fullName` +* `language` +* `location` (local or docshub) +* `model` +* `name` +* `version` Example of JSON in Docshub and local: image - ### 4. /api/upload **Description:** @@ -106,8 +117,9 @@ This endpoint is used to upload a file that needs to be trained, response is JSO **Request:** -Method: POST -Request Body: A multipart/form-data form with file upload and additional fields, including "user" and "name." +**Method**: `POST` + +**Request Body**: A multipart/form-data form with file upload and additional fields, including `user` and `name`. HTML example: @@ -134,8 +146,10 @@ JSON response with a status and a task ID that can be used to check the task's p This endpoint is used to get the status of a task (`task_id`) from `/api/upload` **Request:** -Method: GET -Query Parameter: task_id (task ID to check) + +**Method**: `GET` + +**Query Parameter**: `task_id` (task ID to check) **Sample JavaScript Fetch Request:** ```js @@ -155,33 +169,32 @@ fetch("http://localhost:5001/api/task_status?task_id=YOUR_TASK_ID", { There are two types of responses: 1. While the task is still running, the 'current' value will show progress from 0 to 100. - -```json -{ - "result": { - "current": 1 - }, - "status": "PROGRESS" -} -``` + ```json + { + "result": { + "current": 1 + }, + "status": "PROGRESS" + } + ``` 2. When task is completed: -```json -{ - "result": { - "directory": "temp", - "filename": "install.rst", - "formats": [ - ".rst", - ".md", - ".pdf" - ], - "name_job": "somename", - "user": "local" - }, - "status": "SUCCESS" -} -``` + ```json + { + "result": { + "directory": "temp", + "filename": "install.rst", + "formats": [ + ".rst", + ".md", + ".pdf" + ], + "name_job": "somename", + "user": "local" + }, + "status": "SUCCESS" + } + ``` ### 6. /api/delete_old **Description:** @@ -190,7 +203,8 @@ This endpoint is used to delete old Vector Stores. **Request:** -Method: GET +**Method**: `GET` + ```js // Task status (GET http://127.0.0.1:5000/api/docs_check) fetch("http://localhost:5001/api/task_status?task_id=b2d2a0f4-387c-44fd-a443-e4fe2e7454d1", { @@ -205,7 +219,8 @@ fetch("http://localhost:5001/api/task_status?task_id=b2d2a0f4-387c-44fd-a443-e4f ``` **Response:** -JSON response indicating the status of the operation. +JSON response indicating the status of the operation: + ```json { "status": "ok" } ``` diff --git a/docs/pages/Extensions/react-widget.md b/docs/pages/Extensions/react-widget.md index 63461608..de585581 100644 --- a/docs/pages/Extensions/react-widget.md +++ b/docs/pages/Extensions/react-widget.md @@ -1,22 +1,31 @@ -### How to set up react docsGPT widget on your website: +### Setting up the DocsGPT Widget in Your React Project + +### Introduction: +The DocsGPT Widget is a powerful tool that allows you to integrate AI-powered documentation assistance into your web applications. This guide will walk you through the installation and usage of the DocsGPT Widget in your React project. Whether you're building a web app or a knowledge base, this widget can enhance your user experience. ### Installation -Go to your project and install a new dependency: `npm install docsgpt`. +First, make sure you have Node.js and npm installed in your project. Then go to your project and install a new dependency: `npm install docsgpt`. ### Usage -Go to your project and in the file where you want to use the widget, import it: +In the file where you want to use the widget, import it and include the CSS file: ```js import { DocsGPTWidget } from "docsgpt"; import "docsgpt/dist/style.css"; ``` -Then you can use it like this: `` - -DocsGPTWidget takes 3 props: -1. `apiHost` β€” URL of your DocsGPT API. -2. `selectDocs` β€” documentation that you want to use for your widget (e.g. `default` or `local/docs1.zip`). -3. `apiKey` β€” usually it's empty. +Now, you can use the widget in your component like this : +```jsx + +``` +DocsGPTWidget takes 3 **props**: +1. `apiHost` β€” The URL of your DocsGPT API. +2. `selectDocs` β€” The documentation source that you want to use for your widget (e.g. `default` or `local/docs1.zip`). +3. `apiKey` β€” Usually, it's empty. ### How to use DocsGPTWidget with [Nextra](https://nextra.site/) (Next.js + MDX) Install your widget as described above and then go to your `pages/` folder and create a new file `_app.js` with the following content: @@ -32,6 +41,7 @@ export default function MyApp({ Component, pageProps }) { ) } -``` +``` +For more information about React, refer to this [link here](https://react.dev/learn) diff --git a/docs/pages/Guides/Customising-prompts.md b/docs/pages/Guides/Customising-prompts.md index 6cfbbff7..9d95aa8f 100644 --- a/docs/pages/Guides/Customising-prompts.md +++ b/docs/pages/Guides/Customising-prompts.md @@ -1,10 +1,10 @@ # Customizing the Main Prompt -To customize the main prompt for DocsGPT, follow these steps: +Customizing the main prompt for DocsGPT gives you the ability to tailor the AI's responses to your specific requirements. By modifying the prompt text, you can achieve more accurate and relevant answers. Here's how you can do it: -1. Navigate to `/application/prompt/combine_prompt.txt`. +1. Navigate to `/application/prompts/combine_prompt.txt`. -2. Edit the `combine_prompt.txt` file to modify the prompt text. You can experiment with different phrasings and structures to see how the model responds. +2. Open the `combine_prompt.txt` file and modify the prompt text to suit your needs. You can experiment with different phrasings and structures to observe how the model responds. The main prompt serves as guidance to the AI model on how to generate responses. ## Example Prompt Modification @@ -19,7 +19,7 @@ When using code examples, use the following format: {summaries} ``` - +Feel free to customize the prompt to align it with your specific use case or the kind of responses you want from the AI. For example, you can focus on specific document types, industries, or topics to get more targeted results. ## Conclusion diff --git a/docs/pages/Guides/How-to-train-on-other-documentation.md b/docs/pages/Guides/How-to-train-on-other-documentation.md index aa1ff41d..8ddfa564 100644 --- a/docs/pages/Guides/How-to-train-on-other-documentation.md +++ b/docs/pages/Guides/How-to-train-on-other-documentation.md @@ -1,43 +1,47 @@ ## How to train on other documentation -This AI can use any documentation, but first it needs to be prepared for similarity search. +This AI can utilize any documentation, but it requires preparation for similarity search. Follow these steps to get your documentation ready: + +**Step 1: Prepare Your Documentation** ![video-example-of-how-to-do-it](https://d3dg1063dc54p9.cloudfront.net/videos/how-to-vectorise.gif) Start by going to `/scripts/` folder. If you open this file, you will see that it uses RST files from the folder to create a `index.faiss` and `index.pkl`. -It currently uses OPEN_AI to create the vector store, so make sure your documentation is not too big. Pandas cost me around $3-$4. +It currently uses OPENAI to create the vector store, so make sure your documentation is not too large. Using Pandas cost me around $3-$4. -You can usually find documentation on Github in `docs/` folder for most open-source projects. +You can typically find documentation on GitHub in the `docs/` folder for most open-source projects. -### 1. Find documentation in .rst/.md and create a folder with it in your scripts directory +### 1. Find documentation in .rst/.md format and create a folder with it in your scripts directory. - Name it `inputs/`. - Put all your .rst/.md files in there. - The search is recursive, so you don't need to flatten them. -If there are no .rst/.md files just convert whatever you find to .txt file and feed it. (don't forget to change the extension in script) +If there are no .rst/.md files, convert whatever you find to a .txt file and feed it. (Don't forget to change the extension in the script). -### 2. Create .env file in `scripts/` folder -And write your OpenAI API key inside -`OPENAI_API_KEY=`. +### Step 2: Configure Your OpenAI API Key +1. Create a .env file in the scripts/ folder. + - Add your OpenAI API key inside: OPENAI_API_KEY=. -### 3. Run scripts/ingest.py +### Step 3: Run the Ingestion Script `python ingest.py ingest` -It will tell you how much it will cost. +It will provide you with the estimated cost. -### 4. Move `index.faiss` and `index.pkl` generated in `scripts/output` to `application/` folder. +### Step 4: Move `index.faiss` and `index.pkl` generated in `scripts/output` to `application/` folder. -### 5. Run web app -Once you run it will use new context that is relevant to your documentation. -Make sure you select default in the dropdown in the UI. +### Step 5: Run the Web App +Once you run it, it will use new context relevant to your documentation.Make sure you select default in the dropdown in the UI. ## Customization You can learn more about options while running ingest.py by running: + - Make sure you select 'default' from the dropdown in the UI. +## Customization +You can learn more about options while running ingest.py by executing: `python ingest.py --help` | Options | | |:--------------------------------:|:------------------------------------------------------------------------------------------------------------------------------:| diff --git a/docs/pages/Guides/How-to-use-different-LLM.md b/docs/pages/Guides/How-to-use-different-LLM.md index 8d7cccce..6c7e08a9 100644 --- a/docs/pages/Guides/How-to-use-different-LLM.md +++ b/docs/pages/Guides/How-to-use-different-LLM.md @@ -1,36 +1,42 @@ -Fortunately, there are many providers for LLMs, and some of them can even be run locally. +# Setting Up Local Language Models for Your App -There are two models used in the app: -1. Embeddings. -2. Text generation. +Your app relies on two essential models: Embeddings and Text Generation. While OpenAI's default models work seamlessly, you have the flexibility to switch providers or even run the models locally. -By default, we use OpenAI's models, but if you want to change it or even run it locally, it's very simple! +## Step 1: Configure Environment Variables -### Go to .env file or set environment variables: +Navigate to the `.env` file or set the following environment variables: -`LLM_NAME=` +```env +LLM_NAME= +API_KEY= +EMBEDDINGS_NAME= +EMBEDDINGS_KEY= +VITE_API_STREAMING= +``` -`API_KEY=` +You can omit the keys if users provide their own. Ensure you set `LLM_NAME` and `EMBEDDINGS_NAME`. -`EMBEDDINGS_NAME=` +## Step 2: Choose Your Models -`EMBEDDINGS_KEY=` +**Options for `LLM_NAME`:** +- OpenAI ([More details](https://platform.openai.com/docs/models)) +- manifest ([More details](https://python.langchain.com/docs/integrations/llms/manifest)) +- cohere ([More details](https://docs.cohere.com/docs/llmu)) +- Arc53/docsgpt-14b ([More details](https://huggingface.co/Arc53/docsgpt-14b)) +- Arc53/docsgpt-7b-falcon ([More details](https://huggingface.co/Arc53/docsgpt-7b-falcon)) +- llama.cpp ([More details](https://python.langchain.com/docs/integrations/llms/llamacpp)) -`VITE_API_STREAMING=` +**Options for `EMBEDDINGS_NAME`:** +- openai_text-embedding-ada-002 +- huggingface_sentence-transformers/all-mpnet-base-v2 +- huggingface_hkunlp/instructor-large +- cohere_medium -You don't need to provide keys if you are happy with users providing theirs, so make sure you set `LLM_NAME` and `EMBEDDINGS_NAME`. +If using Llama, set `EMBEDDINGS_NAME` to `huggingface_sentence-transformers/all-mpnet-base-v2`. Download the required model and place it in the `models/` folder. -Options: -LLM_NAME (openai, manifest, cohere, Arc53/docsgpt-14b, Arc53/docsgpt-7b-falcon, llama.cpp) -EMBEDDINGS_NAME (openai_text-embedding-ada-002, huggingface_sentence-transformers/all-mpnet-base-v2, huggingface_hkunlp/instructor-large, cohere_medium) +Alternatively, for local Llama setup, run `setup.sh` and choose option 1. The script handles the DocsGPT model addition. -If using Llama, set the `EMBEDDINGS_NAME` to `huggingface_sentence-transformers/all-mpnet-base-v2` and be sure to download [this model](https://d3dg1063dc54p9.cloudfront.net/models/docsgpt-7b-f16.gguf) into the `models/` folder: `https://d3dg1063dc54p9.cloudfront.net/models/docsgpt-7b-f16.gguf`. +## Step 3: Local Hosting for Privacy -Alternatively, if you wish to run Llama locally, you can run `setup.sh` and choose option 1 when prompted. You do not need to manually add the DocsGPT model mentioned above to your `models/` folder if you use `setup.sh`, as the script will manage that step for you. - -That's it! - -### Hosting everything locally and privately (for using our optimised open-source models) -If you are working with critical data and don't want anything to leave your premises. - -Make sure you set `SELF_HOSTED_MODEL` as true in your `.env` variable, and for your `LLM_NAME`, you can use anything that is on Hugging Face. +If working with sensitive data, host everything locally by setting `SELF_HOSTED_MODEL` to true in your `.env`. For `LLM_NAME`, use any model available on Hugging Face. +That's it! Your app is now configured for local and private hosting, ensuring optimal security for critical data. diff --git a/docs/pages/Guides/My-AI-answers-questions-using-external-knowledge.md b/docs/pages/Guides/My-AI-answers-questions-using-external-knowledge.md index a546116e..99e3c757 100644 --- a/docs/pages/Guides/My-AI-answers-questions-using-external-knowledge.md +++ b/docs/pages/Guides/My-AI-answers-questions-using-external-knowledge.md @@ -1,3 +1,5 @@ +# Avoiding hallucinations + If your AI uses external knowledge and is not explicit enough, it is ok, because we try to make DocsGPT friendly. But if you want to adjust it, here is a simple way:- diff --git a/docs/public/Railway-selection.png b/docs/public/Railway-selection.png new file mode 100644 index 00000000..2100e5b4 Binary files /dev/null and b/docs/public/Railway-selection.png differ diff --git a/frontend/src/About.tsx b/frontend/src/About.tsx index 53a87bc8..eaffafdc 100644 --- a/frontend/src/About.tsx +++ b/frontend/src/About.tsx @@ -1,5 +1,6 @@ //TODO - Add hyperlinks to text //TODO - Styling +import DocsGPT3 from './assets/cute_docsgpt3.svg'; export default function About() { return ( @@ -7,7 +8,7 @@ export default function About() {

About DocsGPT

-

πŸ¦–

+ DocsGPT

Find the information in your documentation through AI-powered diff --git a/frontend/src/Avatar.tsx b/frontend/src/Avatar.tsx deleted file mode 100644 index 04819e7e..00000000 --- a/frontend/src/Avatar.tsx +++ /dev/null @@ -1,18 +0,0 @@ -export default function Avatar({ - avatar, - size, - className, -}: { - avatar: string; - size?: 'SMALL' | 'MEDIUM' | 'LARGE'; - className: string; -}) { - const styles = { - transform: 'scale(-1, 1)', - }; - return ( -

- {avatar} -
- ); -} diff --git a/frontend/src/Hero.tsx b/frontend/src/Hero.tsx index 34347c09..b62823aa 100644 --- a/frontend/src/Hero.tsx +++ b/frontend/src/Hero.tsx @@ -1,9 +1,11 @@ +import DocsGPT3 from './assets/cute_docsgpt3.svg'; + export default function Hero({ className = '' }: { className?: string }) { return (

DocsGPT

-

πŸ¦–

+ DocsGPT

Welcome to DocsGPT, your technical documentation assistant! @@ -17,9 +19,9 @@ export default function Hero({ className = '' }: { className?: string }) { Start by entering your query in the input field below and we will do the rest!

-
-
-
+
+
+
lock
-
-
+
+
lock

Secure Data Storage

@@ -47,8 +49,8 @@ export default function Hero({ className = '' }: { className?: string }) {

-
-
+
+
lock {!navOpen && ( )} @@ -191,21 +195,27 @@ export default function Navigation({ navOpen, setNavOpen }: NavigationProps) { ref={navRef} className={`${ !navOpen && '-ml-96 md:-ml-[18rem]' - } duration-20 fixed top-0 z-20 flex h-full w-72 flex-col border-r-2 bg-gray-50 transition-all`} + } duration-20 fixed top-0 z-20 flex h-full w-72 flex-col border-r-2 bg-white transition-all`} > -
+
+
+ +

DocsGPT

+
@@ -221,30 +231,38 @@ export default function Navigation({ navOpen, setNavOpen }: NavigationProps) { }} className={({ isActive }) => `${ - isActive && conversationId === null ? 'bg-gray-3000' : '' - } my-auto mx-4 mt-4 flex h-9 cursor-pointer gap-4 rounded-3xl hover:bg-gray-100` + isActive ? 'bg-gray-3000' : '' + } group my-auto mx-4 mt-4 flex cursor-pointer gap-2.5 rounded-3xl border border-silver p-3 hover:border-rainy-gray hover:bg-gray-3000` } > - -

New Chat

+ new +

+ New Chat +

-
- {conversations?.map((conversation) => ( - handleConversationClick(id)} - onDeleteConversation={(id) => handleDeleteConversation(id)} - onSave={(conversation) => updateConversationName(conversation)} - /> - ))} -
+ {conversations && ( +
+

Chats

+ {conversations?.map((conversation) => ( + handleConversationClick(id)} + onDeleteConversation={(id) => handleDeleteConversation(id)} + onSave={(conversation) => updateConversationName(conversation)} + /> + ))} +
+ )} -
setIsDocsListOpen(!isDocsListOpen)} > {selectedDocs && ( @@ -306,7 +324,7 @@ export default function Navigation({ navOpen, setNavOpen }: NavigationProps) {
)}
-

Source Docs

+

Source Docs

key -

Reset Key

+

Reset Key

@@ -341,7 +359,7 @@ export default function Navigation({ navOpen, setNavOpen }: NavigationProps) { } > info -

About

+

About

documentation -

Documentation

+

Documentation

 link -

Visit our Discord

+

+ Visit our Discord +

link -

Visit our Github

+

Visit our Github
diff --git a/frontend/src/assets/add.svg b/frontend/src/assets/add.svg new file mode 100644 index 00000000..d38c815e --- /dev/null +++ b/frontend/src/assets/add.svg @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/frontend/src/assets/checkMark.svg b/frontend/src/assets/checkMark2.svg similarity index 100% rename from frontend/src/assets/checkMark.svg rename to frontend/src/assets/checkMark2.svg diff --git a/frontend/src/assets/checkmark.svg b/frontend/src/assets/checkmark.svg index 9ed02cbd..499000e8 100644 --- a/frontend/src/assets/checkmark.svg +++ b/frontend/src/assets/checkmark.svg @@ -1,3 +1 @@ - - - \ No newline at end of file + \ No newline at end of file diff --git a/frontend/src/assets/cute_docsgpt3.svg b/frontend/src/assets/cute_docsgpt3.svg new file mode 100644 index 00000000..53f7cac1 --- /dev/null +++ b/frontend/src/assets/cute_docsgpt3.svg @@ -0,0 +1,9 @@ + + + + + + + + + diff --git a/frontend/src/assets/documentation.svg b/frontend/src/assets/documentation.svg index a1620ff3..344fb6cc 100644 --- a/frontend/src/assets/documentation.svg +++ b/frontend/src/assets/documentation.svg @@ -1,3 +1,3 @@ - - \ No newline at end of file + + diff --git a/frontend/src/assets/expand.svg b/frontend/src/assets/expand.svg new file mode 100644 index 00000000..a3260d98 --- /dev/null +++ b/frontend/src/assets/expand.svg @@ -0,0 +1,4 @@ + + + + diff --git a/frontend/src/components/Avatar.tsx b/frontend/src/components/Avatar.tsx new file mode 100644 index 00000000..202be207 --- /dev/null +++ b/frontend/src/components/Avatar.tsx @@ -0,0 +1,13 @@ +import { ReactNode } from 'react'; + +export default function Avatar({ + avatar, + size, + className, +}: { + avatar: string | ReactNode; + size?: 'SMALL' | 'MEDIUM' | 'LARGE'; + className: string; +}) { + return
{avatar}
; +} diff --git a/frontend/src/conversation/Conversation.tsx b/frontend/src/conversation/Conversation.tsx index 9e852f49..34b146a7 100644 --- a/frontend/src/conversation/Conversation.tsx +++ b/frontend/src/conversation/Conversation.tsx @@ -192,7 +192,7 @@ export default function Conversation() {
)}
-

+

This is a chatbot that uses the GPT-3, Faiss and LangChain to answer questions.

diff --git a/frontend/src/conversation/ConversationBubble.tsx b/frontend/src/conversation/ConversationBubble.tsx index dbf00ad3..8dca54af 100644 --- a/frontend/src/conversation/ConversationBubble.tsx +++ b/frontend/src/conversation/ConversationBubble.tsx @@ -1,16 +1,17 @@ import { forwardRef, useState } from 'react'; -import Avatar from '../Avatar'; +import Avatar from '../components/Avatar'; import { FEEDBACK, MESSAGE_TYPE } from './conversationModels'; import classes from './ConversationBubble.module.css'; import Alert from './../assets/alert.svg'; import { ReactComponent as Like } from './../assets/like.svg'; import { ReactComponent as Dislike } from './../assets/dislike.svg'; import { ReactComponent as Copy } from './../assets/copy.svg'; -import { ReactComponent as Checkmark } from './../assets/checkmark.svg'; +import { ReactComponent as CheckMark } from './../assets/checkmark.svg'; import ReactMarkdown from 'react-markdown'; import copy from 'copy-to-clipboard'; import { Prism as SyntaxHighlighter } from 'react-syntax-highlighter'; import { vscDarkPlus } from 'react-syntax-highlighter/dist/cjs/styles/prism'; +import DocsGPT3 from '../assets/cute_docsgpt3.svg'; const DisableSourceFE = import.meta.env.VITE_DISABLE_SOURCE_FE || false; @@ -37,8 +38,13 @@ const ConversationBubble = forwardRef< // Reset copied to false after a few seconds setTimeout(() => { setCopied(false); - }, 2000); + }, 3000); }; + const [isCopyHovered, setIsCopyHovered] = useState(false); + const [isLikeHovered, setIsLikeHovered] = useState(false); + const [isDislikeHovered, setIsDislikeHovered] = useState(false); + const [isLikeClicked, setIsLikeClicked] = useState(false); + const [isDislikeClicked, setIsDislikeClicked] = useState(false); let bubble; @@ -55,13 +61,26 @@ const ConversationBubble = forwardRef< ); } else { bubble = ( -
+
- + + } + /> +
@@ -146,52 +165,113 @@ const ConversationBubble = forwardRef< )}
- {copied ? ( - - ) : ( - { - handleCopyClick(message); +
+
- )} + > + {copied ? ( + setIsCopyHovered(true)} + onMouseLeave={() => setIsCopyHovered(false)} + /> + ) : ( + { + handleCopyClick(message); + }} + onMouseEnter={() => setIsCopyHovered(true)} + onMouseLeave={() => setIsCopyHovered(false)} + > + )} +
+
- handleFeedback?.('LIKE')} - > +
+
+ { + handleFeedback?.('LIKE'); + setIsLikeClicked(true); + setIsDislikeClicked(false); + }} + onMouseEnter={() => setIsLikeHovered(true)} + onMouseLeave={() => setIsLikeHovered(false)} + > +
+
- handleFeedback?.('DISLIKE')} - > +
+
+ { + handleFeedback?.('DISLIKE'); + setIsDislikeClicked(true); + setIsLikeClicked(false); + }} + onMouseEnter={() => setIsDislikeHovered(true)} + onMouseLeave={() => setIsDislikeHovered(false)} + > +
+
diff --git a/frontend/src/conversation/ConversationTile.tsx b/frontend/src/conversation/ConversationTile.tsx index 1b5dc42c..8fd6b261 100644 --- a/frontend/src/conversation/ConversationTile.tsx +++ b/frontend/src/conversation/ConversationTile.tsx @@ -3,11 +3,10 @@ import { useSelector } from 'react-redux'; import Edit from '../assets/edit.svg'; import Exit from '../assets/exit.svg'; import Message from '../assets/message.svg'; -import CheckMark from '../assets/checkmark.svg'; +import CheckMark2 from '../assets/checkMark2.svg'; import Trash from '../assets/trash.svg'; import { selectConversationId } from '../preferences/preferenceSlice'; -import { useOutsideAlerter } from '../hooks'; interface ConversationProps { name: string; @@ -31,15 +30,15 @@ export default function ConversationTile({ const [isEdit, setIsEdit] = useState(false); const [conversationName, setConversationsName] = useState(''); - useOutsideAlerter( - tileRef, - () => - handleSaveConversation({ - id: conversationId || conversation.id, - name: conversationName, - }), - [conversationName], - ); + // useOutsideAlerter( + // tileRef, + // () => + // handleSaveConversation({ + // id: conversationId || conversation.id, + // name: conversationName, + // }), + // [conversationName], + // ); useEffect(() => { setConversationsName(conversation.name); @@ -95,7 +94,7 @@ export default function ConversationTile({ {conversationId === conversation.id && (
Edit None: + return super().init_parser() + + def get_base_urls(self, urls): + base_urls = [] + for i in urls: + parsed_url = urlparse(i) + base_url = parsed_url.scheme + "://" + parsed_url.netloc + if base_url not in base_urls: + base_urls.append(base_url) + return base_urls + + def get_info_from_paths(self, path): + info = "" + if path.operations: + for operation in path.operations: + info += ( + f"\n{operation.method.value}=" + f"{operation.responses[0].description}" + ) + return info + + def parse_file(self, file_path): + data = parse(file_path) + results = "" + base_urls = self.get_base_urls(link.url for link in data.servers) + base_urls = ",".join([base_url for base_url in base_urls]) + results += f"Base URL:{base_urls}\n" + i = 1 + for path in data.paths: + info = self.get_info_from_paths(path) + results += ( + f"Path{i}: {path.url}\n" + f"description: {path.description}\n" + f"parameters: {path.parameters}\nmethods: {info}\n" + ) + i += 1 + with open("results.txt", "w") as f: + f.write(results) + return results diff --git a/scripts/requirements.txt b/scripts/requirements.txt index 5ead1d07..dc0f7e96 100644 --- a/scripts/requirements.txt +++ b/scripts/requirements.txt @@ -61,6 +61,7 @@ nltk==3.8.1 numcodecs==0.11.0 numpy==1.25.2 openai==0.27.8 +openapi3-parser==1.1.14 openpyxl==3.1.2 packaging==23.1 pandas==2.0.3 diff --git a/tests/llm/test_anthropic.py b/tests/llm/test_anthropic.py new file mode 100644 index 00000000..ee4ba15f --- /dev/null +++ b/tests/llm/test_anthropic.py @@ -0,0 +1,57 @@ +import unittest +from unittest.mock import patch, Mock +from application.llm.anthropic import AnthropicLLM + +class TestAnthropicLLM(unittest.TestCase): + + def setUp(self): + self.api_key = "TEST_API_KEY" + self.llm = AnthropicLLM(api_key=self.api_key) + + @patch("application.llm.anthropic.settings") + def test_init_default_api_key(self, mock_settings): + mock_settings.ANTHROPIC_API_KEY = "DEFAULT_API_KEY" + llm = AnthropicLLM() + self.assertEqual(llm.api_key, "DEFAULT_API_KEY") + + def test_gen(self): + messages = [ + {"content": "context"}, + {"content": "question"} + ] + mock_response = Mock() + mock_response.completion = "test completion" + + with patch.object(self.llm.anthropic.completions, "create", return_value=mock_response) as mock_create: + response = self.llm.gen("test_model", messages) + self.assertEqual(response, "test completion") + + prompt_expected = "### Context \n context \n ### Question \n question" + mock_create.assert_called_with( + model="test_model", + max_tokens_to_sample=300, + stream=False, + prompt=f"{self.llm.HUMAN_PROMPT} {prompt_expected}{self.llm.AI_PROMPT}" + ) + + def test_gen_stream(self): + messages = [ + {"content": "context"}, + {"content": "question"} + ] + mock_responses = [Mock(completion="response_1"), Mock(completion="response_2")] + + with patch.object(self.llm.anthropic.completions, "create", return_value=iter(mock_responses)) as mock_create: + responses = list(self.llm.gen_stream("test_model", messages)) + self.assertListEqual(responses, ["response_1", "response_2"]) + + prompt_expected = "### Context \n context \n ### Question \n question" + mock_create.assert_called_with( + model="test_model", + prompt=f"{self.llm.HUMAN_PROMPT} {prompt_expected}{self.llm.AI_PROMPT}", + max_tokens_to_sample=300, + stream=True + ) + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_openapi3.yaml b/tests/test_openapi3.yaml new file mode 100644 index 00000000..a015e14b --- /dev/null +++ b/tests/test_openapi3.yaml @@ -0,0 +1,116 @@ +openapi: "3.0.0" +info: + version: 1.0.0 + title: Swagger Petstore + license: + name: MIT +servers: + - url: http://petstore.swagger.io/v1 + - url: https://api.example.com/v1/resource + - url: https://api.example.com/v1/another/resource + - url: https://api.example.com/v1/some/endpoint +paths: + /pets: + get: + summary: List all pets + operationId: listPets + tags: + - pets + parameters: + - name: limit + in: query + description: How many items to return at one time (max 100) + required: false + schema: + type: integer + maximum: 100 + format: int32 + responses: + '200': + description: A paged array of pets + headers: + x-next: + description: A link to the next page of responses + schema: + type: string + content: + application/json: + schema: + $ref: "#/components/schemas/Pets" + default: + description: unexpected error + content: + application/json: + schema: + $ref: "#/components/schemas/Error" + post: + summary: Create a pet + operationId: createPets + tags: + - pets + responses: + '201': + description: Null response + default: + description: unexpected error + content: + application/json: + schema: + $ref: "#/components/schemas/Error" + /pets/{petId}: + get: + summary: Info for a specific pet + operationId: showPetById + tags: + - pets + parameters: + - name: petId + in: path + required: true + description: The id of the pet to retrieve + schema: + type: string + responses: + '200': + description: Expected response to a valid request + content: + application/json: + schema: + $ref: "#/components/schemas/Pet" + default: + description: unexpected error + content: + application/json: + schema: + $ref: "#/components/schemas/Error" +components: + schemas: + Pet: + type: object + required: + - id + - name + properties: + id: + type: integer + format: int64 + name: + type: string + tag: + type: string + Pets: + type: array + maxItems: 100 + items: + $ref: "#/components/schemas/Pet" + Error: + type: object + required: + - code + - message + properties: + code: + type: integer + format: int32 + message: + type: string \ No newline at end of file diff --git a/tests/test_openapi3parser.py b/tests/test_openapi3parser.py new file mode 100644 index 00000000..1ce01ef5 --- /dev/null +++ b/tests/test_openapi3parser.py @@ -0,0 +1,50 @@ +import pytest +from openapi_parser import parse +from application.parser.file.openapi3_parser import OpenAPI3Parser + + +@pytest.mark.parametrize( + "urls, expected_base_urls", + [ + ( + [ + "http://petstore.swagger.io/v1", + "https://api.example.com/v1/resource", + "https://api.example.com/v1/another/resource", + "https://api.example.com/v1/some/endpoint", + ], + ["http://petstore.swagger.io", "https://api.example.com"], + ), + ], +) +def test_get_base_urls(urls, expected_base_urls): + assert OpenAPI3Parser().get_base_urls(urls) == expected_base_urls + + +def test_get_info_from_paths(): + file_path = "tests/test_openapi3.yaml" + data = parse(file_path) + path = data.paths[1] + assert ( + OpenAPI3Parser().get_info_from_paths(path) + == "\nget=Expected response to a valid request" + ) + + +def test_parse_file(): + file_path = "tests/test_openapi3.yaml" + results_expected = ( + "Base URL:http://petstore.swagger.io,https://api.example.com\nPath1: " + + "/pets\ndescription: None\nparameters: []\nmethods: \n" + + "get=A paged array of pets\npost=Null " + + "response\nPath2: /pets/{petId}\ndescription: None\n" + + "parameters: []\nmethods: " + + "\nget=Expected response to a valid request\n" + ) + openapi_parser_test = OpenAPI3Parser() + results = openapi_parser_test.parse_file(file_path) + assert results == results_expected + + +if __name__ == "__main__": + pytest.main()