diff --git a/.claude/commands/add-new-service.md b/.claude/commands/add-new-service.md new file mode 100644 index 0000000..5ee4ede --- /dev/null +++ b/.claude/commands/add-new-service.md @@ -0,0 +1,678 @@ +# Add New Service: $ARGUMENTS + +Add a new optional service called **$ARGUMENTS** to the n8n-install project. + +## Naming Conventions + +Derive these from `$ARGUMENTS`: +- `SERVICE_SLUG` = lowercase with hyphens, e.g., `my-service` +- `SERVICE_SLUG_UNDERSCORE` = lowercase with underscores, e.g., `my_service` +- `SERVICE_NAME_UPPER` = UPPERCASE with underscores, e.g., `MY_SERVICE` +- `SERVICE_NAME_TITLE` = Title Case, e.g., `MyService` + +--- + +## STEP 1: docker-compose.yml + +**File:** `docker-compose.yml` + +### 1.1 Basic Service Definition + +Add service block in `services:` section (maintain alphabetical order): + +```yaml + $ARGUMENTS: + image: /: + container_name: $ARGUMENTS + profiles: ["$ARGUMENTS"] + restart: unless-stopped + logging: + driver: "json-file" + options: + max-size: "1m" + max-file: "1" + environment: + # Service-specific env vars + SOME_VAR: ${SOME_VAR:-default} + healthcheck: + test: ["CMD-SHELL", "wget -qO- http://localhost:/health || exit 1"] + interval: 30s + timeout: 10s + retries: 5 + # volumes: + # - ${SERVICE_SLUG_UNDERSCORE}_data:/data +``` + +### 1.2 Caddy Environment Passthrough + +Add to `caddy` service `environment:` section (if externally accessible): + +```yaml + caddy: + environment: + # ... existing vars ... + - ${SERVICE_NAME_UPPER}_HOSTNAME=${${SERVICE_NAME_UPPER}_HOSTNAME} + # If using basic auth: + - ${SERVICE_NAME_UPPER}_USERNAME=${${SERVICE_NAME_UPPER}_USERNAME} + - ${SERVICE_NAME_UPPER}_PASSWORD_HASH=${${SERVICE_NAME_UPPER}_PASSWORD_HASH} +``` + +### 1.3 Named Volume (if persistent storage needed) + +Add to top-level `volumes:` section: + +```yaml +volumes: + # ... existing ... + ${SERVICE_SLUG_UNDERSCORE}_data: +``` + +### 1.4 Service Dependencies + +If service requires database/cache, add `depends_on`: + +```yaml + $ARGUMENTS: + depends_on: + postgres: + condition: service_healthy + redis: + condition: service_healthy +``` + +Common dependencies: +- `postgres` - PostgreSQL database +- `redis` - Redis cache/queue +- `ollama` - Local LLM inference +- `minio` - S3-compatible object storage +- `clickhouse` - Analytics database (for Langfuse) + +### 1.5 Proxy Configuration (for outbound AI API calls) + +If service makes HTTP requests to external AI APIs (OpenAI, Anthropic, Google, etc.), add proxy support: + +```yaml + $ARGUMENTS: + environment: + <<: *proxy-env # Inherits HTTP_PROXY, HTTPS_PROXY, NO_PROXY + # ... other env vars +``` + +The `x-proxy-env` anchor (defined at top of docker-compose.yml) provides: +- `HTTP_PROXY`, `HTTPS_PROXY`, `http_proxy`, `https_proxy` → `${GOST_PROXY_URL:-}` +- `NO_PROXY`, `no_proxy` → `${GOST_NO_PROXY:-}` + +### 1.6 Healthcheck Proxy Bypass + +**CRITICAL:** If using `<<: *proxy-env`, healthcheck MUST bypass proxy: + +```yaml +healthcheck: + test: ["CMD-SHELL", "http_proxy= https_proxy= HTTP_PROXY= HTTPS_PROXY= wget -qO- http://localhost:/health || exit 1"] + interval: 30s + timeout: 10s + retries: 5 +``` + +The `http_proxy= https_proxy= HTTP_PROXY= HTTPS_PROXY=` prefix clears proxy vars for healthcheck only. + +### 1.7 Multi-service Profiles + +For services with multiple containers, use same profile for all: + +```yaml + $ARGUMENTS-worker: + image: /-worker: + container_name: $ARGUMENTS-worker + profiles: ["$ARGUMENTS"] # Same profile + # ... + + $ARGUMENTS-web: + image: /-web: + container_name: $ARGUMENTS-web + profiles: ["$ARGUMENTS"] # Same profile + # ... +``` + +For shared config, use YAML anchors: + +```yaml +# At top of docker-compose.yml (x- prefix = extension, ignored by Docker) +x-$ARGUMENTS-common: &$ARGUMENTS-common + depends_on: + postgres: + condition: service_healthy + redis: + condition: service_healthy + +# In service definitions: + $ARGUMENTS-worker: + <<: *$ARGUMENTS-common + # ... rest of config + + $ARGUMENTS-web: + <<: *$ARGUMENTS-common + # ... rest of config +``` + +Examples in project: +- `langfuse` → langfuse-worker + langfuse-web + clickhouse + minio +- `ragflow` → ragflow + ragflow-mysql + ragflow-redis + ragflow-minio + ragflow-elasticsearch +- `monitoring` → prometheus + grafana + cadvisor + node-exporter + +### 1.8 Hardware/GPU Profiles + +For services with CPU/GPU variants, use mutually exclusive profiles: + +```yaml + # CPU variant + $ARGUMENTS-cpu: + <<: *service-$ARGUMENTS + profiles: ["cpu"] + + # NVIDIA GPU variant + $ARGUMENTS-gpu: + <<: *service-$ARGUMENTS + profiles: ["gpu-nvidia"] + deploy: + resources: + reservations: + devices: + - driver: nvidia + count: 1 + capabilities: [gpu] + + # AMD GPU variant + $ARGUMENTS-gpu-amd: + <<: *service-$ARGUMENTS + image: /:rocm + profiles: ["gpu-amd"] + devices: + - "/dev/kfd" + - "/dev/dri" +``` + +Example: Ollama (ollama-cpu, ollama-gpu, ollama-gpu-amd) + +--- + +## STEP 2: Caddyfile + +**File:** `Caddyfile` + +### 2.1 Basic Reverse Proxy + +```caddyfile +# $ARGUMENTS +{$${SERVICE_NAME_UPPER}_HOSTNAME} { + reverse_proxy $ARGUMENTS: +} +``` + +### 2.2 With Basic Auth + +```caddyfile +{$${SERVICE_NAME_UPPER}_HOSTNAME} { + basic_auth { + {$${SERVICE_NAME_UPPER}_USERNAME} {$${SERVICE_NAME_UPPER}_PASSWORD_HASH} + } + reverse_proxy $ARGUMENTS: +} +``` + +### 2.3 Conditional Basic Auth (allow internal networks without auth) + +```caddyfile +{$${SERVICE_NAME_UPPER}_HOSTNAME} { + @protected not remote_ip 127.0.0.0/8 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 100.64.0.0/10 + + basic_auth @protected { + {$${SERVICE_NAME_UPPER}_USERNAME} {$${SERVICE_NAME_UPPER}_PASSWORD_HASH} + } + reverse_proxy $ARGUMENTS: +} +``` + +### 2.4 Special Protocols (non-standard port) + +```caddyfile +# Example: Neo4j Bolt Protocol on port 7687 +https://{$${SERVICE_NAME_UPPER}_HOSTNAME}:7687 { + reverse_proxy $ARGUMENTS:7687 +} +``` +Note: Non-standard ports must be exposed in Caddy's `ports:` section in docker-compose.yml. + +### 2.5 Static File Serving + +```caddyfile +{$${SERVICE_NAME_UPPER}_HOSTNAME} { + basic_auth { + {$${SERVICE_NAME_UPPER}_USERNAME} {$${SERVICE_NAME_UPPER}_PASSWORD_HASH} + } + root * /srv/$ARGUMENTS + file_server + try_files {path} /index.html # SPA fallback +} +``` + +--- + +## STEP 3: .env.example + +**File:** `.env.example` + +### 3.1 Hostname (in Caddy section) + +```dotenv +${SERVICE_NAME_UPPER}_HOSTNAME=$ARGUMENTS.yourdomain.com +``` + +### 3.2 Credentials (if basic auth) + +```dotenv +############ +# [required] +# ${SERVICE_NAME_TITLE} credentials (for Caddy basic auth) +############ +${SERVICE_NAME_UPPER}_USERNAME= +${SERVICE_NAME_UPPER}_PASSWORD= +${SERVICE_NAME_UPPER}_PASSWORD_HASH= +``` + +### 3.3 GOST_NO_PROXY (if using proxy-env) + +Add service to comma-separated list: + +```dotenv +GOST_NO_PROXY=localhost,127.0.0.1,...existing...,$ARGUMENTS +``` + +--- + +## STEP 4: scripts/03_generate_secrets.sh + +**File:** `scripts/03_generate_secrets.sh` + +### 4.1 VARS_TO_GENERATE Map (~line 75) + +Add password/secret generation: + +```bash +["${SERVICE_NAME_UPPER}_PASSWORD"]="password:32" +``` + +Available types: +- `password:32` - 32-char alphanumeric password +- `api_key:32` - Prefixed API key (sk_...) +- `base64:64` - Base64 encoded secret +- `jwt` - JWT secret +- `hex:32` - Hex string + +### 4.2 EMAIL_VARS Array (~line 42) + +If username should default to installer's email: + +```bash +EMAIL_VARS=( + # ... existing ... + "${SERVICE_NAME_UPPER}_USERNAME" +) +``` + +### 4.3 SERVICES_NEEDING_HASH Array (~line 520) + +**CRITICAL for Basic Auth:** Add service to generate bcrypt hash: + +```bash +SERVICES_NEEDING_HASH=("PROMETHEUS" "SEARXNG" ... "${SERVICE_NAME_UPPER}") +``` + +This automatically: +1. Reads `${SERVICE_NAME_UPPER}_PASSWORD` from `.env` +2. Generates bcrypt hash via `docker exec caddy caddy hash-password` +3. Writes hash to `${SERVICE_NAME_UPPER}_PASSWORD_HASH` in `.env` + +--- + +## STEP 5: scripts/04_wizard.sh + +**File:** `scripts/04_wizard.sh` + +### 5.1 Basic Addition (~line 40) + +Add to `base_services_data` array: + +```bash +"$ARGUMENTS" "${SERVICE_NAME_TITLE} ()" +``` + +### 5.2 Services with Additional Prompts + +For services requiring extra input (like GOST asking for upstream proxy): + +```bash +# After profile selection, add custom dialog +if [[ " ${selected_profiles[*]} " =~ " $ARGUMENTS " ]]; then + CUSTOM_VALUE=$(whiptail --inputbox "Enter value for $ARGUMENTS:" 10 60 "" 3>&1 1>&2 2>&3) + # Save to .env or use later +fi +``` + +See GOST implementation (~lines 206-242) as example. + +### 5.3 Mutually Exclusive Services + +For services that conflict (like Dify and Supabase): + +```bash +# Remove conflicting service from selection +if [[ " ${selected_profiles[*]} " =~ " $ARGUMENTS " ]]; then + selected_profiles=("${selected_profiles[@]/conflicting-service}") +fi +``` + +See Dify/Supabase exclusion (~lines 139-156) as example. + +--- + +## STEP 5.5: scripts/05_configure_services.sh + +**File:** `scripts/05_configure_services.sh` + +For services needing runtime configuration beyond wizard: + +```bash +# Configure $ARGUMENTS if active +if is_profile_active "$ARGUMENTS"; then + log_info "Configuring $ARGUMENTS..." + + # Example: Generate config from template + envsubst < ./$ARGUMENTS/config.template.yml > ./$ARGUMENTS/config.yml + + # Example: Handle mutual exclusion + if is_profile_active "conflicting-service"; then + log_warn "$ARGUMENTS and conflicting-service cannot run together" + # Remove conflicting profile from COMPOSE_PROFILES + fi +fi +``` + +Use cases: +- Token-based authentication (Cloudflare Tunnel) +- Mutual exclusion logic (Supabase vs Dify) +- Dynamic config file generation + +--- + +## STEP 6: scripts/generate_welcome_page.sh + +**File:** `scripts/generate_welcome_page.sh` + +### 6.1 SERVICES_ARRAY Entry + +Add conditional JSON block: + +```bash +# ${SERVICE_NAME_TITLE} +if is_profile_active "$ARGUMENTS"; then + SERVICES_ARRAY+=(" \"$ARGUMENTS\": { + \"hostname\": \"$(json_escape "$${SERVICE_NAME_UPPER}_HOSTNAME")\", + \"credentials\": { + \"username\": \"$(json_escape "$${SERVICE_NAME_UPPER}_USERNAME")\", + \"password\": \"$(json_escape "$${SERVICE_NAME_UPPER}_PASSWORD")\" + }, + \"extra\": { + \"internal_api\": \"http://$ARGUMENTS:\", + \"docs\": \"\" + } + }") +fi +``` + +### 6.2 JSON Structure Variants + +**External service (with hostname):** +```bash +\"hostname\": \"$(json_escape "$${SERVICE_NAME_UPPER}_HOSTNAME")\", +``` + +**Internal-only service (no external access):** +```bash +\"hostname\": null, +``` + +**Credentials - username/password:** +```bash +\"credentials\": { + \"username\": \"$(json_escape "$${SERVICE_NAME_UPPER}_USERNAME")\", + \"password\": \"$(json_escape "$${SERVICE_NAME_UPPER}_PASSWORD")\" +} +``` + +**Credentials - API key only:** +```bash +\"credentials\": { + \"api_key\": \"$(json_escape "$${SERVICE_NAME_UPPER}_API_KEY")\" +} +``` + +**Credentials - none (informational):** +```bash +\"credentials\": { + \"note\": \"No authentication required\" +} +``` + +**Extra field examples:** +```bash +\"extra\": { + \"internal_api\": \"http://$ARGUMENTS:\", + \"docs\": \"https://docs.example.com\", + \"dashboard\": \"http://$ARGUMENTS:9000\", + \"note\": \"Some helpful information\" +} +``` + +**Always use `json_escape "$VAR"`** to prevent JSON breaking from special characters. + +### 6.3 QUICK_START_ARRAY (for first-run setup) + +If service requires initial configuration: + +```bash +if is_profile_active "$ARGUMENTS"; then + QUICK_START_ARRAY+=(" \"$ARGUMENTS\": { + \"title\": \"Configure ${SERVICE_NAME_TITLE}\", + \"description\": \"Complete initial setup\", + \"action\": \"Visit https://\${${SERVICE_NAME_UPPER}_HOSTNAME} to configure\" + }") +fi +``` + +--- + +## STEP 6.5: scripts/07_final_report.sh + +**File:** `scripts/07_final_report.sh` + +For services with post-installation instructions: + +```bash +if is_profile_active "$ARGUMENTS"; then + echo -e " ${GREEN}*${NC} ${WHITE}${SERVICE_NAME_TITLE}${NC}: Visit dashboard to complete initial setup" +fi +``` + +Use for: +- First-time configuration steps +- Important security notices +- Links to documentation + +--- + +## STEP 7: welcome/app.js + +**File:** `welcome/app.js` + +Add to `SERVICE_METADATA` object (~line 145): + +```javascript +'$ARGUMENTS': { + name: '${SERVICE_NAME_TITLE}', + description: '', + icon: '<2-3 letter abbrev>', + color: 'bg-[#]', + category: '', + docsUrl: '' +}, +``` + +### Categories + +| Category | Description | Examples | +|----------|-------------|----------| +| `ai` | AI/ML services | Flowise, LangChain, Ollama, LightRAG | +| `database` | Data storage | PostgreSQL, Qdrant, Weaviate, Neo4j | +| `monitoring` | Observability | Prometheus, Grafana, Langfuse | +| `tools` | Utilities | Gotenberg, Docling, LibreTranslate, PaddleOCR | +| `infra` | Infrastructure | Caddy, Redis, Gost, Portainer | +| `automation` | Workflow automation | n8n, Postiz | + +### Color Examples + +```javascript +color: 'bg-blue-500' // Tailwind named color +color: 'bg-[#FF6B35]' // Custom hex color +color: 'bg-lime-500' // Another named color +``` + +--- + +## STEP 8: README.md + +**File:** `README.md` + +### 8.1 What's Included Section + +Add one-line description: + +```markdown +✅ [**${SERVICE_NAME_TITLE}**]() - +``` + +### 8.2 Quick Start and Usage Section + +Add service URL (alphabetical order): + +```markdown +- **${SERVICE_NAME_TITLE}:** `$ARGUMENTS.yourdomain.com` () +``` + +--- + +## STEP 8.5: CHANGELOG.md + +**File:** `CHANGELOG.md` + +Add under `## [Unreleased]` → `### Added`: + +```markdown +- **${SERVICE_NAME_TITLE}** - +``` + +--- + +## STEP 10.5: scripts/update_preview.sh (optional) + +**File:** `scripts/update_preview.sh` + +If service image should be tracked for updates: + +```bash +if is_profile_active "$ARGUMENTS"; then + check_image_update "/" "${SERVICE_NAME_UPPER}" +fi +``` + +--- + +## STEP 10.6: scripts/apply_update.sh (for complex services) + +**File:** `scripts/apply_update.sh` + +For services with external docker-compose files: + +```bash +if is_profile_active "$ARGUMENTS"; then + log_info "Updating $ARGUMENTS..." + docker compose -f docker-compose.$ARGUMENTS.yml pull +fi +``` + +Most services don't need this. + +--- + +## VALIDATION + +After all changes, validate: + +```bash +# Docker Compose syntax +docker compose -p localai config --quiet + +# Bash script syntax +bash -n scripts/03_generate_secrets.sh +bash -n scripts/04_wizard.sh +bash -n scripts/generate_welcome_page.sh +bash -n scripts/05_configure_services.sh +bash -n scripts/07_final_report.sh +``` + +--- + +## FINAL CHECKLIST + +### Core (REQUIRED) +- [ ] `docker-compose.yml`: service with `profiles`, `container_name`, `logging` +- [ ] `docker-compose.yml`: caddy environment vars (if external) +- [ ] `Caddyfile`: reverse proxy block (if external) +- [ ] `.env.example`: hostname added +- [ ] `scripts/03_generate_secrets.sh`: password in `VARS_TO_GENERATE` +- [ ] `scripts/04_wizard.sh`: service in `base_services_data` +- [ ] `scripts/generate_welcome_page.sh`: `SERVICES_ARRAY` entry +- [ ] `welcome/app.js`: `SERVICE_METADATA` entry +- [ ] `README.md`: description added +- [ ] `CHANGELOG.md`: entry added + +### If Basic Auth +- [ ] `.env.example`: USERNAME, PASSWORD, PASSWORD_HASH vars +- [ ] `scripts/03_generate_secrets.sh`: username in `EMAIL_VARS` +- [ ] `scripts/03_generate_secrets.sh`: service in `SERVICES_NEEDING_HASH` +- [ ] `Caddyfile`: `basic_auth` block +- [ ] `docker-compose.yml`: USERNAME and PASSWORD_HASH passed to caddy + +### If Outbound Proxy (AI API calls) +- [ ] `docker-compose.yml`: `<<: *proxy-env` in environment +- [ ] `.env.example`: service added to `GOST_NO_PROXY` +- [ ] `docker-compose.yml`: healthcheck bypasses proxy + +### If Database Required +- [ ] `docker-compose.yml`: `depends_on` with `condition: service_healthy` + +### If First-Run Setup Needed +- [ ] `scripts/generate_welcome_page.sh`: `QUICK_START_ARRAY` entry +- [ ] `scripts/07_final_report.sh`: post-install instructions + +### If Special Configuration +- [ ] `scripts/04_wizard.sh`: custom prompts +- [ ] `scripts/05_configure_services.sh`: configuration logic + +### If Multi-Container Service +- [ ] All containers use same profile +- [ ] YAML anchors for shared config + +### If Hardware Variants (CPU/GPU) +- [ ] Mutually exclusive profiles (cpu, gpu-nvidia, gpu-amd) +- [ ] GPU resource reservations diff --git a/.cursor/rules/add-new-service.mdc b/.cursor/rules/add-new-service.mdc deleted file mode 100644 index 056bdc9..0000000 --- a/.cursor/rules/add-new-service.mdc +++ /dev/null @@ -1,602 +0,0 @@ ---- -alwaysApply: false ---- -# Guide: Adding a New Service to n8n-install - -This document shows how to add a new optional service (behind Docker Compose profiles) and wire it into the installer, Caddy, and Welcome Page. - -Use a short lowercase slug for your service, e.g., `myservice`. - -## 1) docker-compose.yml -- Add a service block under `services:` with a Compose profile: - - `profiles: ["myservice"]` - - `restart: unless-stopped` - - image/build/command/healthcheck as needed -- IMPORTANT: do not publish ports and do not expose ports. Let Caddy do external HTTPS. - - Avoid `ports:` and avoid `expose:` entries unless strictly required for internal discovery. -- If you intend to proxy it via Caddy, ensure you define a hostname env in `.env.example` (e.g., `MYSERVICE_HOSTNAME`) and pass it to the `caddy` container via the `environment:` section if needed for the Caddyfile. - -Minimal example: -```yaml - myservice: - image: yourorg/myservice:latest - container_name: myservice # Required - used in scripts and Caddyfile - profiles: ["myservice"] - restart: unless-stopped - logging: # Recommended - prevents log bloat - driver: "json-file" - options: - max-size: "1m" - max-file: "1" - # command: ... - # healthcheck: { test: ["CMD-SHELL", "curl -fsS http://localhost:8080/health || exit 1"], interval: 30s, timeout: 10s, retries: 5 } -``` - -If adding Caddy env passthrough (only if used in Caddyfile): -```yaml - caddy: - # ... - environment: - - MYSERVICE_HOSTNAME=${MYSERVICE_HOSTNAME} - # If using basic auth: - - MYSERVICE_USERNAME=${MYSERVICE_USERNAME} - - MYSERVICE_PASSWORD_HASH=${MYSERVICE_PASSWORD_HASH} -``` - -## 1.5) Proxy Configuration (if service needs outbound AI API access) - -If your service makes outbound HTTP requests to external AI APIs (OpenAI, Anthropic, Google, etc.), it should support optional proxy routing via GOST: - -1. Add proxy environment anchor to your service: -```yaml - myservice: - environment: - <<: *proxy-env # Inherits HTTP_PROXY, HTTPS_PROXY, NO_PROXY - # ... other env vars -``` - -2. Add your service name to `GOST_NO_PROXY` in `.env.example`: - - Find the `GOST_NO_PROXY=` line (contains comma-separated list of services) - - Add your service name to prevent internal Docker traffic from routing through proxy - - Example: `...,waha,libretranslate,myservice` - -The `x-proxy-env` anchor is defined at the top of docker-compose.yml and provides: -- `HTTP_PROXY`, `HTTPS_PROXY`, `http_proxy`, `https_proxy` - set to `${GOST_PROXY_URL:-}` (empty if gost not active) -- `NO_PROXY`, `no_proxy` - set to `${GOST_NO_PROXY:-}` (internal services bypass list) - -## 1.6) Healthcheck Proxy Bypass (if using *proxy-env) - -**CRITICAL:** If your service uses `<<: *proxy-env`, the healthcheck command MUST explicitly bypass the proxy. Otherwise, the healthcheck will route through the external proxy and fail. - -```yaml -healthcheck: - test: ["CMD-SHELL", "http_proxy= https_proxy= HTTP_PROXY= HTTPS_PROXY= wget -qO- http://localhost:8080/health || exit 1"] - interval: 30s - timeout: 10s - retries: 5 -``` - -Note: The `http_proxy= https_proxy= HTTP_PROXY= HTTPS_PROXY=` prefix clears proxy env vars for the healthcheck command only. - -## 1.7) Service Dependencies - -If your service requires other services (database, cache, etc.) to be healthy before starting: - -```yaml - myservice: - depends_on: - postgres: - condition: service_healthy - redis: - condition: service_healthy -``` - -Common dependencies: -- `postgres` - PostgreSQL database -- `redis` - Redis cache/queue -- `ollama` - Local LLM inference -- `minio` - S3-compatible object storage -- `clickhouse` - Analytics database (for Langfuse) - -## 1.8) Multi-service Profiles - -Some services consist of multiple containers that should be enabled together. Use the same profile for all related containers: - -```yaml - myservice-worker: - image: yourorg/myservice-worker:latest - container_name: myservice-worker - profiles: ["myservice"] # Same profile as main service - # ... - - myservice-web: - image: yourorg/myservice-web:latest - container_name: myservice-web - profiles: ["myservice"] # Same profile as main service - # ... -``` - -For shared configuration, use YAML anchors: -```yaml -# At the top of docker-compose.yml (x- prefix = extension, ignored by Docker) -x-myservice-common: &myservice-common - depends_on: - postgres: - condition: service_healthy - redis: - condition: service_healthy - -# In service definitions: - myservice-worker: - <<: *myservice-common - # ... rest of config - - myservice-web: - <<: *myservice-common - # ... rest of config -``` - -Examples in this project: -- `langfuse` → langfuse-worker + langfuse-web + clickhouse + minio -- `ragflow` → ragflow + ragflow-mysql + ragflow-redis + ragflow-minio + ragflow-elasticsearch -- `monitoring` → prometheus + grafana + cadvisor + node-exporter - -## 1.9) Hardware/GPU Profiles - -For services with multiple hardware variants (CPU, NVIDIA GPU, AMD GPU), use mutually exclusive profiles: - -```yaml - # CPU variant - myservice-cpu: - <<: *service-myservice - profiles: ["cpu"] - - # NVIDIA GPU variant - myservice-gpu: - <<: *service-myservice - profiles: ["gpu-nvidia"] - deploy: - resources: - reservations: - devices: - - driver: nvidia - count: 1 - capabilities: [gpu] - - # AMD GPU variant - myservice-gpu-amd: - <<: *service-myservice - image: yourorg/myservice:rocm - profiles: ["gpu-amd"] - devices: - - "/dev/kfd" - - "/dev/dri" -``` - -The wizard (`scripts/04_wizard.sh`) handles mutual exclusion - only one hardware profile can be active. - -Example in this project: Ollama (ollama-cpu, ollama-gpu, ollama-gpu-amd) - -## 2) Caddyfile -- Add a site block for the service hostname if it should be reachable externally: -- Ask users whether the service needs Basic Auth via Caddy; if yes, add `basic_auth` with env-based credentials. - -### Basic Example -```caddyfile -{$MYSERVICE_HOSTNAME} { - # Optional. Ask the user if we should protect this endpoint via Basic Auth - basic_auth { - {$MYSERVICE_USERNAME} {$MYSERVICE_PASSWORD_HASH} - } - reverse_proxy myservice:8080 -} -``` - -### Conditional Basic Auth (allow internal networks without auth) -```caddyfile -{$MYSERVICE_HOSTNAME} { - @protected not remote_ip 127.0.0.0/8 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 100.64.0.0/10 - - basic_auth @protected { - {$MYSERVICE_USERNAME} {$MYSERVICE_PASSWORD_HASH} - } - reverse_proxy myservice:8080 -} -``` - -### Special Protocols (e.g., Neo4j Bolt on non-standard port) -```caddyfile -# Neo4j Bolt Protocol on port 7687 -https://{$NEO4J_HOSTNAME}:7687 { - reverse_proxy neo4j:7687 -} -``` -Note: Non-standard ports must be exposed in Caddy's `ports:` section in docker-compose.yml. - -### Static File Serving (e.g., Welcome Page) -```caddyfile -{$WELCOME_HOSTNAME} { - basic_auth { - {$WELCOME_USERNAME} {$WELCOME_PASSWORD_HASH} - } - root * /srv/welcome - file_server - try_files {path} /index.html # SPA fallback -} -``` - -Notes: -- Keep using env placeholders (e.g., `{$MYSERVICE_HOSTNAME}`), supplied by the `caddy` service environment in `docker-compose.yml`. - -## 3) .env.example -- Add the service hostname under the Caddy config section: -```dotenv -MYSERVICE_HOSTNAME=myservice.yourdomain.com -``` -- If Basic Auth is desired, add credentials (username, password, and password hash): -```dotenv -############ -# [required] -# MyService credentials (for Caddy basic auth) -############ -MYSERVICE_USERNAME= -MYSERVICE_PASSWORD= -MYSERVICE_PASSWORD_HASH= -``` - -## 4) scripts/03_generate_secrets.sh - -The script uses several arrays to manage variable generation: - -### Key Arrays - -1. **`VARS_TO_GENERATE`** - Map of variables to auto-generate with type:length format: -```bash -["MYSERVICE_PASSWORD"]="password:32" # 32-char alphanumeric password -["MYSERVICE_API_KEY"]="api_key:32" # Prefixed API key (sk_...) -["MYSERVICE_SECRET"]="base64:64" # Base64 secret -``` - -2. **`EMAIL_VARS`** - Variables that get the installer's email as default value: -```bash -EMAIL_VARS=( - # ... existing vars ... - "MYSERVICE_USERNAME" # Add here if username should be email -) -``` - -3. **`USER_INPUT_VARS`** - Variables from user input (includes EMAIL_VARS): -```bash -USER_INPUT_VARS=( - "${EMAIL_VARS[@]}" - # ... other vars ... -) -``` - -### Steps to add a new service: - -1. Add password to `VARS_TO_GENERATE` map (~line 75): -```bash -["MYSERVICE_PASSWORD"]="password:32" -``` - -2. If username should default to email, add to `EMAIL_VARS` array (~line 42): -```bash -EMAIL_VARS=( - # ... existing ... - "MYSERVICE_USERNAME" -) -``` - -### SERVICES_NEEDING_HASH (Critical for Basic Auth) - -**⚠️ IMPORTANT:** If your service uses Caddy basic auth, you MUST add it to this array. Missing this step will cause authentication to fail! - -Add to `SERVICES_NEEDING_HASH` array (~line 520): -```bash -SERVICES_NEEDING_HASH=("PROMETHEUS" "SEARXNG" ... "MYSERVICE") -``` - -This automatically generates bcrypt hash using: `docker exec caddy caddy hash-password` - -The script will: -1. Read `MYSERVICE_PASSWORD` from `.env` -2. Generate bcrypt hash via Caddy container -3. Write hash to `MYSERVICE_PASSWORD_HASH` in `.env` - -**Note:** Hash generation requires the Caddy container to be running. - -## 5) scripts/04_wizard.sh - -Add the service to the selectable profiles list so users can opt-in during installation: -```bash -# In base_services_data array: -"myservice" "MyService (Short description)" -``` - -### Special Wizard Patterns - -**Services requiring additional prompts** (like GOST asking for upstream proxy URL): -- Add custom whiptail dialog in the wizard -- See GOST implementation (~lines 206-242) as example - -**Mutually exclusive services** (like Dify and Supabase): -- Add logic to remove conflicting service from selection -- See Dify/Supabase exclusion (~lines 139-156) as example - -For complex service-specific configuration, see section 5.5. - -## 5.5) scripts/05_configure_services.sh - -For services that need configuration beyond what the wizard can handle, add logic to this script. - -Use cases: -- **Token-based authentication** (e.g., Cloudflare Tunnel token) -- **Mutual exclusion logic** (e.g., Supabase vs Dify - both use same ports) -- **Dynamic configuration files** (e.g., generating config from template) - -Example pattern: -```bash -# Configure MyService if active -if is_profile_active "myservice"; then - log_info "Configuring MyService..." - - # Example: Generate config file from template - envsubst < ./myservice/config.template.yml > ./myservice/config.yml - - # Example: Handle mutual exclusion - if is_profile_active "conflicting-service"; then - log_warn "MyService and ConflictingService cannot run together" - # Remove conflicting profile from COMPOSE_PROFILES - fi -fi -``` - -See existing implementations: -- Cloudflare Tunnel token handling (~lines 132-153) -- Supabase/Dify mutual exclusion (~lines 159-173) - -## 6) scripts/generate_welcome_page.sh - -This script is called automatically from `scripts/07_final_report.sh` during installation. - -Add a conditional block to generate JSON for the Welcome Page: - -```bash -# MyService -if is_profile_active "myservice"; then - SERVICES_ARRAY+=(" \"myservice\": { - \"hostname\": \"$(json_escape "$MYSERVICE_HOSTNAME")\", - \"credentials\": { - \"username\": \"$(json_escape "$MYSERVICE_USERNAME")\", - \"password\": \"$(json_escape "$MYSERVICE_PASSWORD")\" - }, - \"extra\": { - \"internal_api\": \"http://myservice:8080\", - \"docs\": \"https://example.com/docs\" - } - }") -fi -``` - -### JSON Structure Options - -**External service (with hostname):** -```bash -\"hostname\": \"$(json_escape "$MYSERVICE_HOSTNAME")\", -``` - -**Internal-only service (no external access):** -```bash -\"hostname\": null, -``` - -**Credentials variants:** -```bash -# With username/password: -\"credentials\": { - \"username\": \"...\", - \"password\": \"...\" -} - -# With API key only: -\"credentials\": { - \"api_key\": \"...\" -} - -# Without credentials (informational): -\"credentials\": { - \"note\": \"No authentication required\" -} -``` - -**Extra field examples:** -```bash -\"extra\": { - \"internal_api\": \"http://myservice:8080\", # Docker internal URL - \"docs\": \"https://docs.example.com\", # Documentation link - \"dashboard\": \"http://myservice:9000\", # Alternative UI port - \"note\": \"Some helpful information\" -} -``` - -**Important:** Always use `json_escape "$VAR"` to prevent JSON breaking from special characters. - -### Quick Start Steps (QUICK_START_ARRAY) - -If your service requires first-run setup (e.g., initial configuration, API key setup), add a Quick Start step: - -```bash -# MyService Quick Start -if is_profile_active "myservice"; then - QUICK_START_ARRAY+=(" \"myservice\": { - \"title\": \"Configure MyService\", - \"description\": \"Complete initial setup\", - \"action\": \"Visit https://\${MYSERVICE_HOSTNAME} to configure\" - }") -fi -``` - -This appears in the Welcome Page's "Quick Start" section to guide users through post-installation steps. - -## 6.5) scripts/07_final_report.sh - -For services with first-run instructions that should appear in the terminal after installation: - -```bash -if is_profile_active "myservice"; then - echo -e " ${GREEN}*${NC} ${WHITE}MyService${NC}: Visit dashboard to complete initial setup" -fi -``` - -This provides immediate guidance in the terminal after installation completes. Use for: -- First-time configuration steps -- Important security notices -- Links to documentation - -## 7) welcome/app.js - -Add service metadata to the `SERVICE_METADATA` object (located around line 145): - -```javascript -'myservice': { - name: 'MyService', // Display name - description: 'Short description', // One-line description (~50 chars max) - icon: 'MS', // 2-letter abbreviation - color: 'bg-[#ABC123]', // Tailwind color (hex or named) - category: 'tools', // See categories below - docsUrl: 'https://docs.example.com' // Official documentation URL -}, -``` - -### Categories - -| Category | Description | Examples | -|----------|-------------|----------| -| `ai` | AI/ML services | Flowise, LangChain, Ollama, LightRAG | -| `database` | Data storage | PostgreSQL, Qdrant, Weaviate, Neo4j | -| `monitoring` | Observability | Prometheus, Grafana, Langfuse | -| `tools` | Utilities | Gotenberg, Docling, LibreTranslate, PaddleOCR | -| `infra` | Infrastructure | Caddy, Redis, Gost, Portainer | -| `automation` | Workflow automation | n8n, Postiz | - -### Color Examples - -```javascript -color: 'bg-blue-500' // Tailwind named color -color: 'bg-[#FF6B35]' // Custom hex color -color: 'bg-lime-500' // Another named color -``` - -## 8) README.md -- Add a short, one-line description under "What's Included", linking to your service docs/homepage. -```md -✅ [**MyService**](https://example.com) - One-line description of what it provides. -``` -- Also add the service URL to the "Quick Start and Usage" section in alphabetical order: -```md - - **MyService:** `myservice.yourdomain.com` (Brief description) -``` - -## 8.5) CHANGELOG.md - -Add an entry when introducing a new service: - -```markdown -## [Unreleased] - -### Added -- **MyService** - Brief description of what the service provides -``` - -Follow [Keep a Changelog](https://keepachangelog.com/) format: -- `Added` - for new features -- `Changed` - for changes in existing functionality -- `Fixed` - for bug fixes -- `Removed` - for removed features - -## 9) Ask about Basic Auth (important) -When adding any new public-facing service, explicitly ask the user whether they want to protect the service with Basic Auth via Caddy. If yes, add: -- Credentials section to `.env.example` -- Secret generation in `scripts/03_generate_secrets.sh` -- `basic_auth` in `Caddyfile` -- Pass the username/hash through `docker-compose.yml` `caddy.environment` - -## 10) Verify and apply -- Regenerate secrets to populate new variables: -```bash -bash scripts/03_generate_secrets.sh -``` -- Start (or recreate) only the affected services: -```bash -docker compose -p localai up -d --no-deps --force-recreate caddy -# If your service was added/changed -docker compose -p localai up -d --no-deps --force-recreate myservice -``` -- Check logs: -```bash -docker compose -p localai logs -f --tail=200 myservice | cat -docker compose -p localai logs -f --tail=200 caddy | cat -``` - -## 10.5) scripts/update_preview.sh (optional) - -If your service uses a Docker image that should be tracked for updates, add version checking logic: - -```bash -# Check MyService version -if is_profile_active "myservice"; then - check_image_update "yourorg/myservice" "MYSERVICE" -fi -``` - -This allows `make update` to show available updates for your service. - -## 10.6) scripts/apply_update.sh (for complex services) - -For services with their own docker-compose files (like Supabase, Dify), add handling logic: - -```bash -# Update MyService -if is_profile_active "myservice"; then - log_info "Updating MyService..." - # Pull latest compose file or run service-specific update - docker compose -f docker-compose.myservice.yml pull -fi -``` - -Most services don't need this - only use for services with external compose files or complex update procedures. - -## 11) Quick checklist - -### Core setup -- [ ] Service added to `docker-compose.yml` with profile (no external ports) -- [ ] `container_name: servicename` specified (required) -- [ ] `logging:` driver json-file configured (recommended) -- [ ] Hostname and credentials added to `.env.example` -- [ ] Password added to `VARS_TO_GENERATE` in `scripts/03_generate_secrets.sh` -- [ ] If basic auth: added to `SERVICES_NEEDING_HASH` array -- [ ] Exposed via `Caddyfile` with `reverse_proxy` (+ `basic_auth` if desired) -- [ ] Service selectable in `scripts/04_wizard.sh` -- [ ] Service data added to `scripts/generate_welcome_page.sh` -- [ ] Service metadata added to `welcome/app.js` (`SERVICE_METADATA`) -- [ ] One-line description added to `README.md` -- [ ] Entry added to `CHANGELOG.md` - -### If service needs outbound proxy (AI API calls) -- [ ] Added `<<: *proxy-env` to service environment in `docker-compose.yml` -- [ ] Added service name to `GOST_NO_PROXY` list in `.env.example` -- [ ] Healthcheck bypasses proxy: `http_proxy= https_proxy= ... wget ...` - -### If service has first-run setup -- [ ] Added to `QUICK_START_ARRAY` in `scripts/generate_welcome_page.sh` -- [ ] Added instructions in `scripts/07_final_report.sh` - -### If service needs special configuration -- [ ] Added prompts in `scripts/04_wizard.sh` -- [ ] Added logic in `scripts/05_configure_services.sh` - -### For complex multi-container services -- [ ] All related containers use same profile -- [ ] Consider YAML anchors for shared dependencies (see section 1.8) -