LLM/DocsGPT
1
0
Fork 0
mirror of https://github.com/arc53/DocsGPT.git synced 2025-11-29 08:33:20 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'arc53:main' into contribution-guideline

Browse Source
This commit is contained in:
Ankur Singh
2023-10-15 09:32:43 +05:30
committed by GitHub
parent 80e2d0651b f724f10a35
commit 21b6ce204d
11 changed files with 116 additions and 85 deletions
Download Patch File Download Diff File Expand all files Collapse all files

16
CONTRIBUTING.md
View File

@@ -19,19 +19,25 @@ Thank you for choosing to contribute to DocsGPT! We are all very grateful!
We value contributions in the form of discussions or suggestions. We recommend taking a look at existing issues and our [roadmap](https://github.com/orgs/arc53/projects/2).
If you're interested in contributing code, here are some important things to know:
Before creating issues, please check out how the latest version of our app looks and works by launching it via [Quickstart](https://github.com/arc53/DocsGPT#quickstart) the version on our live demo is slightly modified with login. Your issues should relate to the version that you can launch via [Quickstart](https://github.com/arc53/DocsGPT#quickstart).
We have a frontend built with React (Vite) and a backend in Python.
### 👨‍💻 If you're interested in contributing code, here are some important things to know:
### If you are looking to contribute to frontend (⚛️React, Vite):
Tech Stack Overview:
- 🌐 Frontend: Built with React (Vite) ⚛️,
- 🖥 Backend: Developed in Python 🐍
### 🌐 If you are looking to contribute to frontend (⚛️React, Vite):
- The current frontend is being migrated from [`/application`](https://github.com/arc53/DocsGPT/tree/main/application) to [`/frontend`](https://github.com/arc53/DocsGPT/tree/main/frontend) with a new design, so please contribute to the new one.
- Check out this [milestone](https://github.com/arc53/DocsGPT/milestone/1) and its issues.
- The Figma design can be found [here](https://www.figma.com/file/OXLtrl1EAy885to6S69554/DocsGPT?node-id=0%3A1&t=hjWVuxRg9yi5YkJ9-1).
- The updated Figma design can be found [here](https://www.figma.com/file/OXLtrl1EAy885to6S69554/DocsGPT?node-id=0%3A1&t=hjWVuxRg9yi5YkJ9-1).
Please try to follow the guidelines.
### If you are looking to contribute to Backend (🐍 Python):
### 🖥 If you are looking to contribute to Backend (🐍 Python):
- Review our issues and contribute to [`/application`](https://github.com/arc53/DocsGPT/tree/main/application) or [`/scripts`](https://github.com/arc53/DocsGPT/tree/main/scripts) (please disregard old [`ingest_rst.py`](https://github.com/arc53/DocsGPT/blob/main/scripts/old/ingest_rst.py) [`ingest_rst_sphinx.py`](https://github.com/arc53/DocsGPT/blob/main/scripts/old/ingest_rst_sphinx.py) files; they will be deprecated soon).
- All new code should be covered with unit tests ([pytest](https://github.com/pytest-dev/pytest)). Please find tests under [`/tests`](https://github.com/arc53/DocsGPT/tree/main/tests) folder.

8
README.md
View File

@@ -75,7 +75,7 @@ If you don't have enough resources to run it, you can use bitsnbytes to quantize
- Extensions - Chrome extension.
- Scripts - Script that creates similarity search index and stores for other libraries.
- Scripts - Script that creates similarity search index for other libraries.
- Frontend - Frontend uses Vite and React.
@@ -105,6 +105,10 @@ Otherwise, refer to this Guide:
To stop, just run `Ctrl + C`.
## Development environments
### Spin up mongo and redis
@@ -127,7 +131,7 @@ Make sure you have Python 3.10 or 3.11 installed.
(check out [`application/core/settings.py`](application/core/settings.py) if you want to see more config options.)
2. (optional) Create a Python virtual environment:
You can follow the [Python official documentation](https://docs.python.org/3/tutorial/venv.html) for virtual environments .
You can follow the [Python official documentation](https://docs.python.org/3/tutorial/venv.html) for virtual environments.
a) On Mac OS and Linux
```commandline

2
application/requirements.txt
View File

@@ -41,7 +41,7 @@ Jinja2==3.1.2
jmespath==1.0.1
joblib==1.2.0
kombu==5.2.4
langchain==0.0.308
langchain==0.0.312
loguru==0.6.0
lxml==4.9.2
MarkupSafe==2.1.2

58
docs/pages/Deploying/Hosting-the-app.md
View File

@@ -6,37 +6,30 @@ Here's a step-by-step guide on how to setup an Amazon Lightsail instance to host
(If you know how to create a Lightsail instance, you can skip to the recommended configuration part by clicking here).
### 1. Create an account or login to https://lightsail.aws.amazon.com
### 1. Create an AWS Account:
If you haven't already, create or log in to your AWS account at https://lightsail.aws.amazon.com.
### 2. Click on "Create instance"
### 2. Create an Instance:
### 3. Create your instance
a. Click "Create Instance."
The first step is to select the "Instance location". In most cases, there's no need to switch locations as the default one will work well.
b. Select the "Instance location." In most cases, the default location works fine.
After that, it is time to pick your Instance Image. We recommend using "Linux/Unix" as the image and "Ubuntu 20.04 LTS" as the Operating System.
c. Choose "Linux/Unix" as the image and "Ubuntu 20.04 LTS" as the Operating System.
As for instance plan, it'll vary depending on your unique demands, but a "1 GB, 1vCPU, 40GB SSD and 2TB transfer" setup should cover most scenarios.
d. Configure the instance plan based on your requirements. A "1 GB, 1vCPU, 40GB SSD, and 2TB transfer" setup is recommended for most scenarios.
Lastly, identify your instance by giving it a unique name and then hit "Create instance".
e. Give your instance a unique name and click "Create Instance."
PS: Once you create your instance, it'll likely take a few minutes for the setup to be completed.
PS: It may take a few minutes for the instance setup to complete.
#### The recommended configuration is as follows:
### Connecting to Your newly created Instance
- Ubuntu 20.04 LTS
- 1GB RAM
- 1vCPU
- 40GB SSD Hard Drive
- 2TB transfer
Your instance will be ready a few minutes after creation. To access it, open the instance and click "Connect using SSH."
### Connecting to your newly created instance
#### Clone the DocsGPT Repository
Your instance will be ready for use a few minutes after being created. To access it, just open it up and click on "Connect using SSH".
#### Clone the repository
A terminal window will pop up, and the first step will be to clone the DocsGPT git repository:
A terminal window will pop up, and the first step will be to clone the DocsGPT Git repository:
`git clone https://github.com/arc53/DocsGPT.git`
@@ -56,15 +49,15 @@ And now install docker-compose:
`sudo apt install docker-compose`
#### Access the DocsGPT folder
#### Access the DocsGPT Folder
Enter the following command to access the folder in which DocsGPT docker-compose file is present.
`cd DocsGPT/`
#### Prepare the environment
#### Prepare the Environment
Inside the DocsGPT folder, create a `.env` file and copy the contents of `.env_sample` into it.
Inside the DocsGPT folder create a `.env` file and copy the contents of `.env_sample` into it.
`nano .env`
@@ -78,16 +71,16 @@ SELF_HOSTED_MODEL=false
To save the file, press CTRL+X, then Y, and then ENTER.
Next, we need to set a correct IP for our Backend. To do so, open the docker-compose.yml file:
Next, set the correct IP for the Backend by opening the docker-compose.yml file:
`nano docker-compose.yml`
And change this line 7 `VITE_API_HOST=http://localhost:7091`
And Change line 7 to: `VITE_API_HOST=http://localhost:7091`
to this `VITE_API_HOST=http://<your instance public IP>:7091`
This will allow the frontend to connect to the backend.
#### Running the app
#### Running the Application
You're almost there! Now that all the necessary bits and pieces have been installed, it is time to run the application. To do so, use the following command:
@@ -95,18 +88,17 @@ You're almost there! Now that all the necessary bits and pieces have been instal
Launching it for the first time will take a few minutes to download all the necessary dependencies and build.
Once this is done, you can go ahead and close the terminal window.
Once this is done you can go ahead and close the terminal window.
#### Enabling ports
#### Enabling Ports
Before you are able to access your live instance, you must first enable the port that it is using.
a. Before you are able to access your live instance, you must first enable the port that it is using.
Open your Lightsail instance and head to "Networking".
b. Open your Lightsail instance and head to "Networking".
Then click on "Add rule" under "IPv4 Firewall", enter `5173` as your port, and hit "Create".
c. Then click on "Add rule" under "IPv4 Firewall", enter `5173` as your port, and hit "Create".
Repeat the process for port `7091`.
#### Access your instance
Your instance will now be available under your Public IP Address and port `5173`. Enjoy!
Your instance is now available at your Public IP Address on port 5173. Enjoy using DocsGPT!

39
docs/pages/Deploying/Quickstart.md
View File

@@ -16,6 +16,45 @@ Otherwise, refer to this Guide:
To stop, just run `Ctrl + C`.
For WINDOWS:
To run the given setup on Windows, you can use the Windows Subsystem for Linux (WSL) or a Git Bash terminal to execute similar commands. Here are the steps adapted for Windows:
Option 1: Using Windows Subsystem for Linux (WSL):
1. Install WSL if you haven't already. You can follow the official Microsoft documentation for installation: (https://learn.microsoft.com/en-us/windows/wsl/install).
2. After setting up WSL, open the WSL terminal.
3. Clone the repository and create the `.env` file:
```
git clone https://github.com/arc53/DocsGPT.git
cd DocsGPT
echo "API_KEY=Yourkey" > .env
echo "VITE_API_STREAMING=true" >> .env
```
4. Run the following command to start the setup with Docker Compose:
`./run-with-docker-compose.sh`
5. Open your web browser and navigate to (http://localhost:5173/).
6. To stop the setup, just press `Ctrl + C` in the WSL terminal
Option 2: Using Git Bash or Command Prompt (CMD):
1. Install Git for Windows if you haven't already. You can download it from the official website: (https://gitforwindows.org/).
2. Open Git Bash or Command Prompt.
3. Clone the repository and create the `.env` file:
```
git clone https://github.com/arc53/DocsGPT.git
cd DocsGPT
echo "API_KEY=Yourkey" > .env
echo "VITE_API_STREAMING=true" >> .env
```
4.Run the following command to start the setup with Docker Compose:
`./run-with-docker-compose.sh`
5.Open your web browser and navigate to (http://localhost:5173/).
6.To stop the setup, just press Ctrl + C in the Git Bash or Command Prompt terminal.
These steps should help you set up and run the project on Windows using either WSL or Git Bash/Command Prompt. Make sure you have Docker installed and properly configured on your Windows system for this to work.
### Chrome Extension
To install the Chrome extension:

8
docs/pages/Developing/API-docs.md
View File

@@ -104,7 +104,9 @@ fetch("http://localhost:5001/api/task_status?task_id=b2d2a0f4-387c-44fd-a443-e4f
Responses:
There are two types of responses:
1. While task is still running, where "current" will show progress from 0 to 100:
1. While the task is still running, the 'current' value will show progress from 0 to 100.
```json
{
"result": {
@@ -133,7 +135,8 @@ There are two types of responses:
```
### /api/delete_old
Deletes old Vector stores:
Deletes old Vector Stores:
```js
// Task status (GET http://127.0.0.1:5000/api/docs_check)
fetch("http://localhost:5001/api/task_status?task_id=b2d2a0f4-387c-44fd-a443-e4fe2e7454d1", {
@@ -144,7 +147,6 @@ fetch("http://localhost:5001/api/task_status?task_id=b2d2a0f4-387c-44fd-a443-e4f
})
.then((res) => res.text())
.then(console.log.bind(console))
```
Response:

36
frontend/src/Navigation.tsx
View File

@@ -165,11 +165,7 @@ export default function Navigation({ navOpen, setNavOpen }: NavigationProps) {
*/
useEffect(() => {
if (isMobile) {
setNavOpen(false);
return;
}
setNavOpen(true);
setNavOpen(!isMobile);
}, [isMobile]);
return (
@@ -194,7 +190,7 @@ export default function Navigation({ navOpen, setNavOpen }: NavigationProps) {
ref={navRef}
className={`${
!navOpen && '-ml-96 md:-ml-[18rem]'
} duration-20 fixed z-20 flex h-full w-72 flex-col border-r-2 bg-gray-50 transition-all`}
} duration-20 fixed top-0 z-20 flex h-full w-72 flex-col border-r-2 bg-gray-50 transition-all`}
>
<div className={'visible h-16 w-full border-b-2 md:h-12'}>
<button
@@ -229,22 +225,18 @@ export default function Navigation({ navOpen, setNavOpen }: NavigationProps) {
}
>
<img src={Message} className="ml-4 w-5"></img>
<p className="my-auto text-eerie-black">New Chat</p>
<p className="my-auto text-sm text-eerie-black">New Chat</p>
</NavLink>
<div className="conversations-container max-h-[25rem] overflow-y-auto">
{conversations
? conversations.map((conversation) => (
<ConversationTile
key={conversation.id}
conversation={conversation}
selectConversation={(id) => handleConversationClick(id)}
onDeleteConversation={(id) => handleDeleteConversation(id)}
onSave={(conversation) =>
updateConversationName(conversation)
}
/>
))
: null}
{conversations?.map((conversation) => (
<ConversationTile
key={conversation.id}
conversation={conversation}
selectConversation={(id) => handleConversationClick(id)}
onDeleteConversation={(id) => handleDeleteConversation(id)}
onSave={(conversation) => updateConversationName(conversation)}
/>
))}
</div>
<div className="flex-grow border-b-2 border-gray-100"></div>
@@ -289,7 +281,7 @@ export default function Navigation({ navOpen, setNavOpen }: NavigationProps) {
<p className="ml-5 flex-1 overflow-hidden overflow-ellipsis whitespace-nowrap py-3">
{doc.name} {doc.version}
</p>
{doc.location === 'local' ? (
{doc.location === 'local' && (
<img
src={Exit}
alt="Exit"
@@ -300,7 +292,7 @@ export default function Navigation({ navOpen, setNavOpen }: NavigationProps) {
handleDeleteClick(index, doc);
}}
/>
) : null}
)}
</div>
);
}

10
frontend/src/conversation/Conversation.tsx
View File

@@ -39,11 +39,7 @@ export default function Conversation() {
useEffect(() => {
const observerCallback: IntersectionObserverCallback = (entries) => {
entries.forEach((entry) => {
if (entry.isIntersecting) {
setHasScrolledToLast(true);
} else {
setHasScrolledToLast(false);
}
setHasScrolledToLast(entry.isIntersecting);
});
};
@@ -121,7 +117,7 @@ export default function Conversation() {
return (
<div className="flex flex-col justify-center p-4 md:flex-row">
{queries.length > 0 && !hasScrolledToLast ? (
{queries.length > 0 && !hasScrolledToLast && (
<button
onClick={scrollIntoView}
aria-label="scroll to bottom"
@@ -133,7 +129,7 @@ export default function Conversation() {
className="h4- w-4 opacity-50 md:h-5 md:w-5"
/>
</button>
) : null}
)}
{queries.length > 0 && (
<div className="mt-20 flex flex-col transition-all md:w-3/4">

8
frontend/src/conversation/ConversationTile.tsx
View File

@@ -68,7 +68,7 @@ export default function ConversationTile({
onClick={() => {
selectConversation(conversation.id);
}}
className={`my-auto mx-4 mt-4 flex h-12 cursor-pointer items-center justify-between gap-4 rounded-3xl hover:bg-gray-100 ${
className={`my-auto mx-4 mt-4 flex h-9 cursor-pointer items-center justify-between gap-4 rounded-3xl hover:bg-gray-100 ${
conversationId === conversation.id ? 'bg-gray-100' : ''
}`}
>
@@ -77,7 +77,7 @@ export default function ConversationTile({
conversationId === conversation.id ? 'w-[75%]' : 'w-[95%]'
} gap-4`}
>
<img src={Message} className="ml-2 w-5"></img>
<img src={Message} className="ml-4 w-5"></img>
{isEdit ? (
<input
autoFocus
@@ -92,7 +92,7 @@ export default function ConversationTile({
</p>
)}
</div>
{conversationId === conversation.id ? (
{conversationId === conversation.id && (
<div className="flex">
<img
src={isEdit ? CheckMark : Edit}
@@ -122,7 +122,7 @@ export default function ConversationTile({
}}
/>
</div>
) : null}
)}
</div>
);
}

14
frontend/src/upload/Upload.tsx
View File

@@ -55,9 +55,8 @@ export default function Upload({
setProgress(undefined);
setModalState('INACTIVE');
}}
className={`rounded-3xl bg-purple-30 px-4 py-2 text-sm font-medium text-white ${
isCancellable ? '' : 'hidden'
}`}
className={`rounded-3xl bg-purple-30 px-4 py-2 text-sm font-medium text-white ${isCancellable ? '' : 'hidden'
}`}
>
Finish
</button>
@@ -206,7 +205,9 @@ export default function Upload({
<div className="flex flex-row-reverse">
<button
onClick={uploadFile}
className="ml-6 rounded-3xl bg-purple-30 py-2 px-6 text-white"
className={`ml-6 rounded-3xl bg-purple-30 text-white ${files.length > 0 ? '' : 'bg-opacity-75 text-opacity-80'
} py-2 px-6`}
disabled={files.length === 0} // Disable the button if no file is selected
>
Train
</button>
@@ -227,9 +228,8 @@ export default function Upload({
return (
<article
className={`${
modalState === 'ACTIVE' ? 'visible' : 'hidden'
} absolute z-30 h-screen w-screen bg-gray-alpha`}
className={`${modalState === 'ACTIVE' ? 'visible' : 'hidden'
} absolute z-30 h-screen w-screen bg-gray-alpha`}
>
<article className="mx-auto mt-24 flex w-[90vw] max-w-lg flex-col gap-4 rounded-lg bg-white p-6 shadow-lg">
{view}

2
scripts/requirements.txt
View File

@@ -47,7 +47,7 @@ javalang==0.13.0
Jinja2==3.1.2
jmespath==1.0.1
joblib==1.3.1
langchain==0.0.308
langchain==0.0.312
lxml==4.9.3
manifest-ml==0.1.8
MarkupSafe==2.1.3