LLM/DocsGPT
1
0
Fork 0
mirror of https://github.com/arc53/DocsGPT.git synced 2026-02-24 21:31:05 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update application files and fix LLM models, create new retriever class

Browse Source
This commit is contained in:
Alex
2024-04-09 14:02:33 +01:00
parent e07df29ab9
commit 391f686173
13 changed files with 202 additions and 185 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
application/utils.py Normal file
View File

@@ -0,0 +1,6 @@
from transformers import GPT2TokenizerFast
def count_tokens(string):
tokenizer = GPT2TokenizerFast.from_pretrained('gpt2')
return len(tokenizer(string)['input_ids'])