feat: template-based prompt rendering with dynamic namespace injection (#2091)

* feat: template-based prompt rendering with dynamic namespace injection

* refactor: improve template engine initialization with clearer formatting

* refactor: streamline ReActAgent methods and improve content extraction logic

feat: enhance error handling in NamespaceManager and TemplateEngine

fix: update NewAgent component to ensure consistent form data submission

test: modify tests for ReActAgent and prompt renderer to reflect method changes and improve coverage

* feat: tools namespace + three-tier token budget

* refactor: remove unused variable assignment in message building tests

* Enhance prompt customization and tool pre-fetching functionality

* ruff lint fix

* refactor: cleaner error handling and reduce code clutter

---------

Co-authored-by: Alex <a@tushynski.me>

application/api/answer/routes/stream.py

@@ -60,6 +60,10 @@ class StreamResource(Resource, BaseAnswerResource):
             "attachments": fields.List(
                 fields.String, required=False, description="List of attachment IDs"
             ),
+            "passthrough": fields.Raw(
+                required=False,
+                description="Dynamic parameters to inject into prompt template",
+            ),
         },
     )
 
@@ -73,17 +77,20 @@ class StreamResource(Resource, BaseAnswerResource):
         processor = StreamProcessor(data, decoded_token)
         try:
             processor.initialize()
-            agent = processor.create_agent()
-            retriever = processor.create_retriever()
+
+            docs_together, docs_list = processor.pre_fetch_docs(data["question"])
+            tools_data = processor.pre_fetch_tools()
+
+            agent = processor.create_agent(
+                docs_together=docs_together, docs=docs_list, tools_data=tools_data
+            )
 
             if error := self.check_usage(processor.agent_config):
                 return error
-
             return Response(
                 self.complete_stream(
                     question=data["question"],
                     agent=agent,
-                    retriever=retriever,
                     conversation_id=processor.conversation_id,
                     user_api_key=processor.agent_config.get("user_api_key"),
                     decoded_token=processor.decoded_token,