Monitoring Guide

Monitor your Wealth trading bot's performance with built-in dashboards and AI-powered queries.

Table of Contents

• Overview
• Quick Health Check
• AI-Powered Monitoring
• Grafana Dashboards
• Key Metrics
• Alerts
• Log Viewing
• Kubernetes Metrics Flow
• Docker Compose Monitoring Stack

Overview

flowchart TB
    subgraph Bot["Wealth Bot"]
        STRAT[Strategy Engine]
        METRICS[Metrics Exporter]
        LOGS[Structured Logs]
        HEALTH[Health Endpoint]
    end
    
    subgraph Collection["Data Collection"]
        OTEL[OpenTelemetry Collector]
        PROM_EXP[Prometheus Exporter<br/>:8889/metrics]
    end
    
    subgraph Storage["Storage Backends"]
        PROM[Prometheus]
        LOKI[Loki]
        TEMPO[Tempo]
    end
    
    subgraph Viz["Visualization"]
        GRAF[Grafana Dashboards]
        AI[AI Queries via MCP]
        ALERTS[Alert Manager]
    end
    
    STRAT --> METRICS
    STRAT --> LOGS
    STRAT --> HEALTH
    
    METRICS -->|OTLP| OTEL
    LOGS -->|OTLP| OTEL
    HEALTH --> GRAF
    
    OTEL --> PROM_EXP
    PROM -->|scrape| PROM_EXP
    OTEL --> LOKI
    OTEL --> TEMPO
    
    PROM --> GRAF
    LOKI --> GRAF
    TEMPO --> GRAF
    GRAF --> AI
    GRAF --> ALERTS
    
    style Bot fill:#e3f2fd
    style Collection fill:#fff3e0
    style Storage fill:#e8f5e9
    style Viz fill:#f3e5f5

The bot provides multiple monitoring options:

• 🤖 AI Queries - Ask questions in natural language via Grafana MCP
• 📊 Dashboards - Visual monitoring in Grafana
• 📈 Metrics - Real-time performance data via OpenTelemetry
• 🔔 Alerts - Automated notifications for important events
• ❤️ Health Endpoint - Quick status checks

Monitoring Options

flowchart LR
    subgraph Options["Monitoring Options"]
        CLI[CLI Commands]
        HEALTH[Health Endpoint]
        DASH[Grafana Dashboards]
        AI[AI Queries]
        ALERTS[Automated Alerts]
    end
    
    CLI --> |"wealth positions"| STATUS[Quick Status]
    HEALTH --> |":9090/health"| STATUS
    DASH --> |"Visual"| DEEP[Deep Analysis]
    AI --> |"Natural Language"| DEEP
    ALERTS --> |"Notifications"| AUTO[Automated Response]
    
    style CLI fill:#e3f2fd
    style HEALTH fill:#e3f2fd
    style DASH fill:#c8e6c9
    style AI fill:#f3e5f5
    style ALERTS fill:#fff3e0

Quick Health Check

CLI Commands

# Overall health status
curl http://localhost:9090/health | jq

# Check positions
wealth positions

# Check balances
wealth balance

# View configuration
wealth config

Health Endpoint Response

{
  "status": "healthy",
  "timestamp": "2024-01-15T10:30:00Z",
  "uptime_seconds": 3600,
  "components": {
    "websockets": {
      "binance": { "connected": true, "last_update": "2024-01-15T10:29:58Z", "error_count": 0 },
      "bybit": { "connected": true, "last_update": "2024-01-15T10:29:59Z", "error_count": 0 },
      "aster": { "connected": true, "last_update": "2024-01-15T10:29:57Z", "error_count": 0 }
    },
    "exchanges": {
      "binance_api": { "reachable": true, "consecutive_failures": 0 },
      "bybit_api": { "reachable": true, "consecutive_failures": 0 },
      "aster_api": { "reachable": true, "consecutive_failures": 0 }
    },
    "strategy": {
      "active_positions": 2,
      "fresh_funding_rates": 15
    }
  }
}

AI-Powered Monitoring

Ask your AI assistant (GitHub Copilot, Claude, etc.) natural language questions:

Examples:

• "What's my current P&L?"
• "Show funding rates for BTCUSDT"
• "Are there any firing alerts?"
• "Which exchange has the best execution performance?"
• "What's the average position holding time?"

No need to write queries or navigate dashboards - AI queries metrics automatically.

Setup (5 minutes)

1. Start services:

   docker compose up -d
   

2. Generate Grafana token at http://localhost:3000

3. Add to .env:

   GRAFANA_SERVICE_ACCOUNT_TOKEN=glsa_...
   

4. Restart MCP:

   docker compose restart grafana-mcp
   

Grafana Dashboards

Accessing Dashboards

1. Open Grafana: http://localhost:3000
2. Default credentials: admin / admin
3. Navigate to Dashboards → Wealth Trading Bot

Key Dashboard Panels

Setting Up Grafana Cloud

For cloud monitoring (recommended for production):

1. Create free Grafana Cloud account
2. Configure OTLP export
3. Import dashboard templates

See Grafana Cloud Setup for full instructions.

Key Metrics

Trading Performance

System Health

Error Types

Order errors are classified by type for easier diagnosis:

Rejection Reasons

Opportunities can be rejected for various reasons:

View in Terminal

# Metrics info endpoint
curl http://localhost:9090/metrics | jq

Alerts

Built-in Alert Types

Configuring Alerts in Grafana

1. Go to Alerting → Alert rules
2. Create new alert rule
3. Set condition (e.g., wealth_balance_total < 1000)
4. Add notification channel (email, Slack, Discord)

Example Alert Rule

# Low Balance Alert
condition: wealth_balance_total{exchange="binance"} < 1000
for: 5m
severity: critical

Log Viewing

Real-time Logs

# Follow bot logs
wealth run 2>&1 | tee -a bot.log

# Or with Docker
docker compose logs -f wealth

Log Levels

Searching Logs in Grafana (Loki)

# All errors
{service="wealth-bot"} |= "ERROR"

# Order execution logs
{service="wealth-bot"} |= "arbitrage"

# WebSocket issues
{service="wealth-bot"} |= "WebSocket"

Kubernetes Metrics Flow

When deploying to Kubernetes with the Helm chart, metrics flow through the OpenTelemetry Collector sidecar:

App → OTLP (gRPC :4317) → OTEL Collector → Prometheus Exporter (:8889) → Prometheus scrape

Configuration

The Helm chart enables the Prometheus exporter by default:

otelCollector:
  enabled: true
  prometheus:
    enabled: true   # Exposes :8889/metrics for Prometheus scraping
    port: 8889

How It Works

1. App exports OTLP - The wealth bot sends metrics via OTLP to the collector sidecar
2. Collector processes - Batching, resource enrichment, and format conversion
3. Prometheus scrapes - ServiceMonitor scrapes /metrics from the collector (port 8889)
4. Grafana queries - Dashboards query Prometheus for visualization

Note: The app's /metrics endpoint returns JSON info about OTLP config, not Prometheus format. For Prometheus-compatible metrics, enable otelCollector.prometheus.enabled=true.

Docker Compose Monitoring Stack

The included compose.yml provides:

• Grafana - Dashboards and visualization
• OpenTelemetry Collector - Metrics aggregation
• Prometheus - Metrics storage
• Loki - Log aggregation
• Tempo - Distributed tracing

Starting the Stack

# Start all services
docker compose up -d

# Check service status
docker compose ps

# View logs
docker compose logs -f

Accessing Services

Recommended Monitoring Setup

Development / Paper Trading

• Local Grafana (Docker Compose)
• Terminal logs
• Health endpoint checks

Production / Live Trading

• Grafana Cloud (free tier available)
• Alert notifications (email/Slack)
• AI-powered monitoring (MCP)
• 24/7 uptime monitoring

Quick Reference

# Health check
curl http://localhost:9090/health | jq

# View positions
wealth positions

# View balances  
wealth balance

# Start monitoring stack
docker compose up -d

# View Grafana
open http://localhost:3000

# Follow logs
docker compose logs -f wealth

Related Documentation