LLM/DocsGPT
1
0
Fork 0
mirror of https://github.com/arc53/DocsGPT.git synced 2025-11-29 16:43:16 +00:00
Code Issues Packages Projects Releases Wiki Activity

Ingest rst with sphinx

Browse Source 
Transforms all rst files in provided folder to txt format first (utilising sphinx library). In my tests size of raw sample decreased 2-3 times.
This commit is contained in:
Pavel
2023-02-06 23:43:23 +04:00
parent 5e18a3a7c3
commit 1c734727a1
2 changed files with 73 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
scripts/ingest_rst.py
View File

@@ -5,13 +5,12 @@ from langchain.vectorstores import FAISS
from langchain.embeddings import OpenAIEmbeddings
import pickle
import dotenv
import os
dotenv.load_dotenv()
# Here we load in the data in the format that Notion exports it in.
ps = list(Path("pandasdocs/").glob("**/*.rst"))
ps = list(Path("scikit-learn").glob("**/*.rst"))
# parse all child directories
data = []
@@ -37,4 +36,4 @@ store = FAISS.from_texts(docs, OpenAIEmbeddings(), metadatas=metadatas)
faiss.write_index(store.index, "docs.index")
store.index = None
with open("faiss_store.pkl", "wb") as f:
pickle.dump(store, f)
pickle.dump(store, f)