56 Commits

Author	SHA1	Message	Date
ManishMadan2882	5ad34e2216	(fix:indexes) look for the right path	2025-04-22 17:34:25 +05:30
ManishMadan2882	64c42f0ddf	(feat:storage) file, indexes uploads	2025-04-22 05:18:07 +05:30
ManishMadan2882	0d3e6157cd	(feat:attachmentUpload) parse content before upload	2025-04-17 16:23:01 +05:30
ManishMadan2882	0a0e16547e	(feat:fs_abstract) attachment uploads	2025-04-17 02:35:45 +05:30
ManishMadan2882	0c1138179b	(feat:attch) store file mime type	2025-04-07 20:16:03 +05:30
ManishMadan2882	a37bd76950	(feat:storeAttach) store in inputs, raise errors from worker	2025-04-06 16:01:57 +05:30
ManishMadan2882	19d68252cd	(fix/attach): inputs are created in application	2025-04-02 16:36:58 +05:30
ManishMadan2882	f235a94986	(feat:attach) pass attachments for generation	2025-04-02 15:14:56 +05:30
ManishMadan2882	55050a9f58	(feat:attachment) upload single file	2025-03-27 03:28:03 +05:30
ManishMadan2882	502dc9ec52	(feat:attachments) store and ingest files shared	2025-03-26 18:01:31 +05:30
Pavel	13fcbe3e74	scraper with markdownify	2025-01-15 01:08:09 +03:00
Alex	41b4c28430	fix: linting	2024-12-23 17:41:44 +00:00
Pavel	b41a989051	test version	2024-12-23 16:59:27 +00:00
kom-senapati	3b76b3ddce	fix: Settings import	2024-10-22 18:03:46 +05:30
kom-senapati	83f2fb1e62	refactor: Use MongoDB singleton for connection management	2024-10-18 09:18:15 +05:30
Alex	355fd2b5d7	fix: file check	2024-10-08 15:56:48 +01:00
Dhruv Kadam	6698a000e6	Update worker.py ... 1. Error Handling: Added try-except blocks for file operations and network requests. 2. Logging Enhancements: Improved logging to capture more detailed information. 3. Code Refactoring: Created download_file and upload_index functions to avoid code repetition. 4. Configuration: Used constants for MIN_TOKENS, MAX_TOKENS, and RECURSION_DEPTH.	2024-09-29 14:49:40 +05:30
Siddhant Rai	3d292aa485	feat: sync remote sources through celery periodic tasks	2024-09-25 15:20:11 +05:30
Alex	44d225e6ca	Merge branch 'main' into 1059-migrating-database-to-new-model	2024-09-09 23:55:25 +01:00
Alex	2f9c72c1cf	feat: migrate store to source_id	2024-09-09 15:46:18 +01:00
Alex	1bb81614a5	fix: metadata things	2024-09-09 13:37:11 +01:00
Alex	8166642ff9	fix: write id instead of old path on remote db's	2024-09-09 12:00:59 +01:00
Alex	a353e69648	feat: new vectors structure	2024-09-08 16:59:51 +01:00
Alex	d9309ebc6e	feat: better token counter	2024-08-31 17:07:40 +01:00
Alex	4f88b6dc71	feat: logging	2024-08-31 12:30:03 +01:00
ManishMadan2882	1eb168be55	vector indexes to be named after mongo _id	2024-08-11 19:33:31 +05:30
ManishMadan2882	9000838aab	(feat:vectors): calc, add token in db	2024-05-24 21:10:50 +05:30
Alex	5c8133a810	chore: Update Docker build platforms for application and frontend and optimised embedding import	2024-05-18 12:10:24 +01:00
Siddhant Rai	aa670efe3a	fix: connection aborted in WebBaseLoader	2024-05-03 18:25:01 +05:30
Pavel	7a02df5588	Multiple uploads	2024-04-09 19:56:07 +04:00
Siddhant Rai	eed1bfbe50	feat: fields to handle reddit loader + minor changes	2024-03-26 16:07:44 +05:30
Siddhant Rai	60cfea1126	feat: added reddit loader	2024-03-16 20:22:05 +05:30
Pavel	54d187a0ad	Fixing ingestion metadata grouping	2024-02-28 19:52:58 +03:00
Alex	0cb3d12d94	Refactor loader classes to accept inputs directly	2024-02-14 15:17:56 +00:00
Pavel	030c2a740f	upload_remote class	2024-02-13 23:41:36 +03:00
Alex	2e14dec12d	Merge pull request #849 from arc53/main ... Sync	2024-02-09 14:05:39 +00:00
Arnav Kohli	4c6b8b4173	Update worker.py ... Added comments in difficult to understand areas	2023-10-31 20:00:07 +05:30
Pavel	381a2740ee	change input	2023-10-13 21:52:56 +04:00
Alex	8b3b16bce4	inputs	2023-10-13 08:46:35 +01:00
Pavel	658867cb46	No crawler, no sitemap	2023-10-12 01:03:40 +04:00
Robbie Walmsley	57fb29b600	Update worker.py	2023-10-09 10:55:34 +01:00
Alex	d85eb83ea2	elastic search fixes	2023-09-30 15:25:31 +01:00
Alex	347cfe253f	elastic2	2023-09-29 17:17:48 +01:00
Alex	8fa9657ba6	working full	2023-09-27 16:25:57 +01:00
Anton Larin	98a97f34f5	fix packaging and imports and introduce tests with pytest. ... still issues with celery worker.	2023-08-14 18:20:25 +02:00
Anton Larin	968849e52b	code readability, formatting, minor version bump	2023-06-17 12:42:05 +02:00
Alex	e49dd0cc6a	metadata on ingestion	2023-05-17 21:41:24 +01:00
Anton Larin	962becb9a5	Linting ... * validate python formatting on every build with Ruff * fix lint warnings	2023-05-13 10:36:17 +02:00
Serj	8742cdae0a	Refactored url join	2023-04-30 10:46:52 +01:00
Serj	2d92e95c8a	Added settings usage to the worker	2023-04-29 15:56:32 +01:00