Wealth Trading System - User Guide

Disclaimer: This software is provided for educational purposes. Cryptocurrency trading involves substantial risk. Use at your own risk. The authors assume no responsibility for financial losses.

Getting Started Guide

Quick start guide for installing and running the Wealth Trading Bot

Prerequisites

Operating System: Linux, macOS, or Windows
Exchange Accounts: Binance Futures, Bybit Perpetuals, or HyperLiquid (optional for paper trading)

Installation

Option 1: Download Binary (Recommended)

Download the latest release for your platform:

# Linux (x86_64)
curl -LO https://github.com/thiras/wealth/releases/latest/download/wealth-linux-x86_64.tar.gz
tar -xzf wealth-linux-x86_64.tar.gz
chmod +x wealth
sudo mv wealth /usr/local/bin/

# macOS (Apple Silicon)
curl -LO https://github.com/thiras/wealth/releases/latest/download/wealth-darwin-arm64.tar.gz
tar -xzf wealth-darwin-arm64.tar.gz
chmod +x wealth
sudo mv wealth /usr/local/bin/

# macOS (Intel)
curl -LO https://github.com/thiras/wealth/releases/latest/download/wealth-darwin-x86_64.tar.gz
tar -xzf wealth-darwin-x86_64.tar.gz
chmod +x wealth
sudo mv wealth /usr/local/bin/

Verify installation:

wealth --version

Option 2: Docker (Recommended for Production)

# Pull the latest image
docker pull ghcr.io/thiras/wealth:latest

# Run with Docker Compose (recommended)
curl -LO https://raw.githubusercontent.com/thiras/wealth/main/compose.yml
docker compose up -d

# Or run directly
docker run -d --name wealth \
  -v $(pwd)/config.toml:/app/config.toml \
  -v $(pwd)/credentials.encrypted.json:/app/credentials.encrypted.json \
  -e CREDENTIALS_PASSPHRASE="your_passphrase" \
  ghcr.io/thiras/wealth:latest

Option 3: Build from Source

For developers who want to build from source, see the Development Guide.

First Run (Paper Trading)

The bot runs in paper trading mode by default, which is safe for testing without real money.

# Run with defaults (paper trading)
wealth run

# Show help and available commands
wealth --help

You should see output like:

INFO  Initializing bot | mode=paper_trading
INFO  Starting WebSocket price streams
INFO  Binance WebSocket connected
INFO  Bybit WebSocket connected
INFO  Bot running | metrics_port=9090

Check Health

In another terminal:

# Health endpoint
curl http://localhost:9090/health | jq

# Metrics endpoint (JSON info about OTLP config)
curl http://localhost:9090/metrics | jq
# Query actual metrics in Grafana/Prometheus backend

Configuration

Paper Trading (Default)

No credentials needed! Just run:

wealth run

Live Trading Setup

The bot supports two methods for providing credentials, and automatically chooses the right one:

Encrypted Credentials (Recommended for production)
- Set CREDENTIALS_PASSPHRASE in .env
- Use wealth credentials commands to manage
- Credentials stored in AES-256-GCM encrypted file
Environment Variables (For development/testing)
- Set exchange-specific variables in .env
- Simpler but less secure
- Bot automatically falls back to this if no encrypted file

No configuration needed - just set up one method and the bot handles the rest!

Step 1: Get API Keys

Binance API Keys - API Key + Secret Key
Bybit API Keys - API Key + Secret Key
HyperLiquid: Use your Ethereum wallet private key (0x... format)
- Direct mode: Just the private key
- Subkey mode (recommended): Master wallet address + API wallet private key
- Vault mode: Vault address + API wallet private key
- See HyperLiquid Integration Guide for authorization details

Step 2: Create Encrypted Credentials

# Create encrypted config
wealth credentials create

# Add exchange credentials
export CREDENTIALS_PASSPHRASE="your_secure_passphrase"
wealth credentials add binance YOUR_API_KEY YOUR_SECRET
wealth credentials add bybit YOUR_API_KEY YOUR_SECRET

# HyperLiquid (choose one mode):

# Direct wallet mode (simplest)
wealth credentials add hyperliquid "" 0xYourPrivateKey

# API wallet (subkey) mode (recommended - better security)
wealth credentials add hyperliquid 0xMasterWallet 0xAPIWalletPrivateKey

# Vault mode (for HyperLiquid vaults)
wealth credentials add hyperliquid 0xVaultAddress 0xAPIWalletPrivateKey --vault-mode

# Verify
wealth credentials verify

Important: If using HyperLiquid subkey mode, you must authorize the API wallet in the HyperLiquid UI. See the detailed HyperLiquid Authorization Guide for step-by-step instructions.

Step 3: Run in Live Mode

export WEALTH__EXECUTION__MODE=live
export CREDENTIALS_PASSPHRASE="your_secure_passphrase"
wealth run

Alternative: Using Environment Variables

For development or testing, you can use environment variables instead of encrypted credentials:

# 1. Copy example configuration
cp .env.example .env

# 2. Edit .env and set exchange credentials
nano .env

# Example .env content:
# BINANCE_API_KEY=your_api_key
# BINANCE_SECRET_KEY=your_secret_key
# BYBIT_API_KEY=your_api_key
# BYBIT_SECRET_KEY=your_secret_key

# 3. Run (no CREDENTIALS_PASSPHRASE needed)
export WEALTH__EXECUTION__MODE=live
wealth run

Note: The bot automatically detects and uses this method when CREDENTIALS_PASSPHRASE is not set.

Basic Commands

# Run the bot
wealth run

# Verify configuration
wealth verify

# Display current config
wealth config

# Manage credentials
wealth credentials create
wealth credentials add binance YOUR_KEY YOUR_SECRET
wealth credentials verify

# Generate shell completions
wealth completions bash > ~/.local/share/bash-completion/completions/wealth

💡 Tip: Once running, you can edit config.toml and changes will apply automatically within 500ms without restart. See Configuration Hot-Reload for details.

Testing the Setup

1. Verify Configuration

wealth verify --encrypted

2. Check Exchange Connectivity

wealth health --all

3. View Current Config

wealth config

4. Monitor Metrics

# In your browser
open http://localhost:9090/metrics

Common First-Run Issues

Issue: "Failed to decrypt credentials"

Solution: Set CREDENTIALS_PASSPHRASE environment variable

export CREDENTIALS_PASSPHRASE="your_passphrase"

Issue: "Configuration validation failed"

Solution: Check your configuration values

wealth verify

Issue: WebSocket connection failed

Solution: Check internet connection and firewall settings

Next Steps

Once your bot is running:

Monitor Performance: Visit http://localhost:9090/metrics
Set Up AI Monitoring: See Grafana MCP Setup 🤖
View Logs: Check terminal output for trading activity
Configure Strategy: See Configuration Guide
Set Up Dashboards: See Monitoring Guide
Deploy to Production: See Deployment Guide

Quick Reference

Paper Trading (Safe):

wealth run  # That's it!

Live Trading (Requires credentials):

export WEALTH__EXECUTION__MODE=live
export CREDENTIALS_PASSPHRASE="your_passphrase"
wealth run

Custom Configuration:

# Use environment variables or config.toml for customization
export WEALTH__OBSERVABILITY__METRICS_PORT=8080
wealth run --verbose

AI-Powered Monitoring (After setup):

# Ask AI natural language questions like:
# "What's my current P&L?"
# "Show funding rates for BTCUSDT"
# "Are there any firing alerts?"

Stop the Bot: Press Ctrl+C for graceful shutdown (takes ~6-8 seconds to close all positions safely)

Note: Shutdown is fast thanks to shutdown-aware background tasks that respond instantly to signals.

Practical Examples & Common Workflows

This guide provides practical, real-world examples for using the Wealth trading bot. All examples assume you've completed the Getting Started guide.

Quick Start Example

Complete Workflow from Scratch

This example shows the complete workflow from installation to your first trade:

# 1. Download and install (or use Docker)
# See Getting Started guide for installation options

# 2. Set up credentials (encrypted storage)
export CREDENTIALS_PASSPHRASE="your-secure-passphrase-here"
wealth credentials create

# 3. Add exchange API credentials
wealth credentials add binance "your-binance-api-key" "your-binance-secret-key"
wealth credentials add hyperliquid "your-hyperliquid-wallet" "your-hyperliquid-key"

# 4. Configure trading parameters (edit config.toml)
# See Configuration Guide for all options

# 5. Verify configuration
wealth verify

# 6. Check balances (paper trading mode by default)
export WEALTH__EXECUTION__MODE=paper
wealth balance

# 7. Run the bot
wealth run

Expected Output:

2025-11-07T10:30:00Z INFO  Wealth Trading Bot v0.33.0
2025-11-07T10:30:00Z INFO  Mode: PAPER TRADING (no real orders)
2025-11-07T10:30:01Z INFO  Initialized leverage: BTCUSDT = 15x
2025-11-07T10:30:02Z INFO  WebSocket connected: Binance ticker stream
2025-11-07T10:30:03Z INFO  WebSocket connected: HyperLiquid ticker stream
2025-11-07T10:30:04Z INFO  Fetching funding rates...
2025-11-07T10:30:05Z INFO  Binance BTCUSDT: 0.0100%
2025-11-07T10:30:05Z INFO  HyperLiquid BTC: 0.0500%
2025-11-07T10:30:05Z INFO  Spread detected: 0.0400% (above 0.0300% threshold)
2025-11-07T10:30:05Z INFO  Opening arbitrage position...

Opening Your First Position

Manual Position Opening (Development/Testing)

# Check current funding rates
wealth funding

# Output:
# Exchange      Symbol      Rate        Next Payment
# binance       BTCUSDT     0.0100%     2025-11-07 12:00:00 UTC
# hyperliquid   BTC         0.0500%     2025-11-07 11:00:00 UTC
# Spread: 0.0400% ✓ (above 0.0300% threshold)

# Check available balance
wealth balance

# Output:
# Exchange      Asset    Free         Total        Reserved
# binance       USDT     10000.00     10000.00     0.00
# hyperliquid   USDT     10000.00     10000.00     0.00

# Start the bot (it will auto-open positions when spreads are found)
wealth run

Understanding Position Logic

The bot opens positions automatically when:

Spread Check: funding_rate_high - funding_rate_low >= 0.0300% (configurable)
Balance Validation: Sufficient collateral with 20% safety buffer
Leverage Validation: Within exchange limits (1-125x Binance, 1-50x HyperLiquid)
Position Limits: Under max concurrent pairs (default: 3)

Example Calculation:

Given:
- Position size: $1,000
- Leverage: 10x
- Safety buffer: 20%

Required collateral = ($1,000 / 10) * 1.2 = $120 per exchange
Total required = $120 + $120 = $240

If available balance >= $240, position opens

Monitoring Active Positions

Check Current Positions

# View all open positions
wealth positions

# Output:
# ID    Symbol      Long Exchange  Short Exchange  Size       Entry Spread  Current Spread  Funding Collected  P&L      Age
# 1     BTCUSDT     binance       hyperliquid     0.1 BTC    0.0400%      0.0350%         $2.40             $12.50   2h 15m
# 2     ETHUSDT     bybit         binance         2.0 ETH    0.0380%      0.0420%         $1.80             $8.30    1h 45m

Real-Time Monitoring with OpenTelemetry

The bot exports metrics via OTLP to your observability backend. Check configuration:

# View OTLP configuration (JSON response)
curl -s localhost:9090/metrics | jq

# Output:
# {
#   "status": "operational",
#   "backend": "OpenTelemetry OTLP",
#   "endpoint": "http://localhost:4317",
#   "protocol": "gRPC"
# }

# Query actual metrics in Grafana or your OTLP-compatible backend:
# wealth_funding_rate{exchange="binance",symbol="BTCUSDT"}
# wealth_positions_active
# wealth_profit_loss_usd

Using Grafana (Optional)

# Start Grafana with docker-compose
docker-compose up -d grafana

# Access dashboard at http://localhost:3000
# Default credentials: admin/admin
# Pre-configured dashboard shows:
# - Active positions
# - Funding rate spreads
# - P&L over time
# - WebSocket health
# - API latency

Closing Positions

Automatic Closing (Normal Operation)

Positions close automatically when any condition is met:

Target Profit Reached: 5% profit (configurable via target_profit)
Spread Reversal: Spread drops below 50% of entry spread
Emergency: Critical errors or shutdown signal

Example Auto-Close:

2025-11-07T14:30:00Z INFO  Position #1 BTCUSDT reached target profit (5.2%)
2025-11-07T14:30:01Z INFO  Closing position atomically...
2025-11-07T14:30:02Z INFO  Close order placed: Binance (sell 0.1 BTC)
2025-11-07T14:30:03Z INFO  Close order placed: HyperLiquid (buy 0.1 BTC)
2025-11-07T14:30:04Z INFO  Position closed successfully. Final P&L: $52.40

Manual Emergency Close

# Close positions on specific exchange
wealth close --exchange binance --confirm

# Close positions for specific symbol
wealth close --symbol BTCUSDT --confirm

# Emergency close ALL positions (requires confirmation)
wealth close --all --confirm

# Graceful shutdown (closes all positions cleanly)
# Press Ctrl+C once and wait (~6-8 seconds typical)
^C
2025-11-07T14:35:00Z INFO  Shutdown signal received
2025-11-07T14:35:01Z INFO  Closing all positions gracefully...
2025-11-07T14:35:05Z INFO  All positions closed. Exiting.

Note: All close commands require --confirm flag for safety.

Verifying Position Closure

# Check positions after close
wealth positions

# Output should be empty:
# 📊 Current Positions
#
#   No active positions

# Check balances are back to normal
wealth balance

Alert Integration

Discord Notifications

Set up Discord alerts for important events:

# Set your Discord webhook URL
export DISCORD_WEBHOOK_URL="https://discord.com/api/webhooks/..."

# The bot will send notifications for:
# - Position opened/closed
# - Target profit reached
# - Errors or warnings

Slack Notifications

# Set your Slack webhook URL
export SLACK_WEBHOOK_URL="https://hooks.slack.com/services/..."

Troubleshooting Common Issues

Issue: "Insufficient balance" error

Solution:

# Check current balance
wealth balance

# If balance is low, deposit more funds or reduce position size
# Edit config.toml:
[trading]
position_size_percentage = 0.1  # Reduce from 30% to 10%
max_position_usd = 5000.0       # Reduce cap

Issue: Position not opening despite good spread

Possible causes:

Balance too low

# Check required collateral
# For $1000 position at 10x leverage with 20% buffer:
# Required = ($1000 / 10) * 1.2 = $120 per exchange

Leverage not initialized

# Check logs for:
grep "Initialized leverage" logs/wealth.log

# If missing, verify config.json has leverage section

Max concurrent positions reached

# Check current count
wealth positions | wc -l

# Increase in config if needed:
"max_concurrent_pairs": 5

Issue: WebSocket keeps disconnecting

Solution:

# Check network connectivity
curl -I https://fstream.binance.com

# Enable detailed logging
export RUST_LOG=wealth=debug
wealth run

# If rate limited, increase reconnection delay in config

Issue: Orders rejected by exchange

Common reasons:

Quantity too small

# Binance minimum: 0.001 BTC
# HyperLiquid minimum: 0.01 BTC (varies by symbol)

# Increase position size or use different symbol

Insufficient margin

# Check exchange-specific requirements:
wealth verify --check-margin

Position mode not set (Binance)

# Bot auto-initializes hedge mode, but if it fails:
# Manually set in Binance UI: Futures > Preferences > Position Mode > Hedge Mode

Getting Started - Initial setup
Configuration Guide - Detailed config options
CLI Reference - All commands
API Reference - Programmatic usage
Troubleshooting - Detailed problem solving
Runbook - Operational procedures

Next Steps

Test in Paper Trading Mode: Run with WEALTH__EXECUTION__MODE=paper for 1-2 weeks
Monitor Metrics: Set up Grafana dashboard for visualization
Set Up Alerts: Configure Discord/Slack notifications for important events
Gradual Scaling: Start with small position sizes, increase gradually
Review Performance: Use wealth stats to analyze historical performance

Questions? See Troubleshooting or check the logs in logs/wealth.log.

Configuration Guide

Complete reference for configuration settings

Overview

The bot uses a unified configuration system with support for TOML configuration files and environment variables. All variables have sensible defaults and are validated on startup.

Current Version: v0.33.0 uses simplified configuration structure with direct field access.

Configuration Files:

config.toml - Main configuration file (instruments, leverage, trading params) - Preferred
config.json - Legacy JSON format (still supported for backward compatibility)
.env - Local environment configuration for sensitive values

Hot-Reload Support: The bot supports runtime configuration updates for safe parameters without requiring a restart. See Configuration Hot-Reload for details on which parameters can be hot-reloaded.

Configuration Architecture:

AppConfig - Single unified configuration structure
Direct field access: config.trading.*, config.risk.*, config.leverage.*
Environment variables with WEALTH__ prefix override TOML/JSON settings

Loading Priority:

Environment variables with WEALTH__ prefix (highest priority)
TOML/JSON configuration file (config.toml or config.json)
Default values (fallback)

Configuration Sections

The configuration is organized into clear sections:

Section	Description
`instruments`	Trading pairs to monitor
`leverage`	Per-symbol leverage settings
`trading`	Strategy parameters (spread thresholds, position sizing)
`risk`	Risk management (slippage, trailing stops, fees)
`execution`	Execution mode (paper/live) and resilience
`observability`	Metrics, logging, telemetry
`licensing`	License configuration

Environment Variable Overrides:

All configuration can be overridden via environment variables
Prefix pattern: WEALTH__SECTION__FIELD (e.g., WEALTH__TRADING__MIN_FUNDING_SPREAD)

Trading Configuration

WEALTHTRADINGMIN_FUNDING_SPREAD

Minimum funding rate spread required to open a position.

Type: Decimal (0.0-1.0)
Default: 0.0004 (0.04%)
TOML Path: trading.min_funding_spread
Example: WEALTH__TRADING__MIN_FUNDING_SPREAD=0.0005

WEALTHTRADINGMAX_POSITION_USD

Maximum position size cap in USD (safety limit).

Type: Decimal
Default: 10000 ($10,000)
TOML Path: trading.max_position_usd
Example: WEALTH__TRADING__MAX_POSITION_USD=20000

WEALTHTRADINGPOSITION_SIZE_PERCENT

Percentage of available balance to use per position.

Type: Decimal (0.0 - 1.0)
Default: 0.3 (30%)
TOML Path: trading.position_size_percent
Example: WEALTH__TRADING__POSITION_SIZE_PERCENT=0.25

WEALTHTRADINGTARGET_PROFIT_PERCENT

Target profit percentage per position.

Type: Decimal (0.0 - 1.0)
Default: 0.05 (5%)
TOML Path: trading.target_profit_percent
Example: WEALTH__TRADING__TARGET_PROFIT_PERCENT=0.08

WEALTHTRADINGMAX_POSITION_DURATION_HOURS

Maximum position duration in hours before forced exit. Prevents capital from being tied up indefinitely.

Type: Integer
Default: 168 (7 days)
TOML Path: trading.max_position_duration_hours
Example: WEALTH__TRADING__MAX_POSITION_DURATION_HOURS=336
Note: Set to 0 to disable duration-based exits

WEALTHTRADINGNARROW_SPREAD_EXIT_PERIODS

Number of consecutive narrow spread periods (8-hour intervals) before closing position.

Type: Integer
Default: 3 (24 hours of narrow spread)
TOML Path: trading.narrow_spread_exit_periods
Example: WEALTH__TRADING__NARROW_SPREAD_EXIT_PERIODS=2
Note: Set to 0 for immediate spread narrowing exit

WEALTHTRADINGMAX_CONCURRENT_POSITIONS

Maximum number of concurrent arbitrage pairs.

Type: Integer
Default: 5
TOML Path: trading.max_concurrent_positions
Example: WEALTH__TRADING__MAX_CONCURRENT_POSITIONS=3

WEALTHTRADINGUPDATE_INTERVAL_SECS

Funding rate update interval in seconds.

Type: Integer
Default: 60
TOML Path: trading.update_interval_secs
Example: WEALTH__TRADING__UPDATE_INTERVAL_SECS=30

WEALTHTRADINGMAX_HEDGE_ATTEMPTS

Maximum number of hedging attempts for partial fills.

Type: Integer
Default: 5
TOML Path: trading.max_hedge_attempts
Example: WEALTH__TRADING__MAX_HEDGE_ATTEMPTS=3
Purpose: Prevents infinite hedging loops when orders partially fill

WEALTHTRADINGHEDGE_TOLERANCE_PERCENT

Hedging tolerance as percentage (when to stop hedging).

Type: Decimal (0.0-1.0)
Default: 0.001 (0.1%)
TOML Path: trading.hedge_tolerance_percent
Example: WEALTH__TRADING__HEDGE_TOLERANCE_PERCENT=0.0005
Purpose: Allows small quantity mismatches without additional hedging orders

Note: Leverage is configured per-symbol in the [leverage] section, not as a global trading parameter.

Expected Value (EV) Gating

The bot uses expected value analysis to filter opportunities before execution. Only opportunities with positive risk-adjusted EV after costs are considered.

WEALTHTRADINGMIN_EXPECTED_VALUE

Minimum expected value threshold (net profit after costs).

Type: Decimal (0.0-1.0)
Default: 0.0005 (0.05%)
TOML Path: trading.min_expected_value
Example: WEALTH__TRADING__MIN_EXPECTED_VALUE=0.0008
Purpose: Filters out low-quality opportunities with insufficient edge
Formula: EV = (time_weighted_spread × 3 funding payments) - (entry_fees + exit_fees + slippage)
Note: Time weighting applies linear decay based on minutes until next funding payment

WEALTHTRADINGSTALENESS_PENALTY

Discount applied to EV calculation when data nears staleness threshold.

Type: Decimal (0.0-1.0)
Default: 0.0003 (0.03%)
TOML Path: trading.staleness_penalty
Example: WEALTH__TRADING__STALENESS_PENALTY=0.0005
Purpose: Conservative adjustment for potential data lag
When Applied: Automatically reduces gross EV when funding rate data is >4 minutes old

Example EV Calculation:

Funding spread: 90 bps (0.0090)
Time to funding: 240 minutes (50% of 8h period)
Time weight: 0.5
Gross EV: 90 bps × 0.5 = 45 bps

Entry fees (both sides): 4 bps
Exit fees (both sides): 4 bps
Estimated slippage: 10 bps
Total costs: 18 bps

Adjusted EV: 45 - 18 = 27 bps
Passes threshold: 27 bps > 5 bps ✅

Kelly Position Sizing

The bot supports two position sizing modes: Kelly Criterion (dynamic, risk-adjusted) or Fixed Percentage (simple, predictable).

WEALTHTRADINGUSE_KELLY_CRITERION

Enable or disable Kelly Criterion for position sizing.

Type: Boolean
Default: true
TOML Path: trading.use_kelly_criterion
Example: WEALTH__TRADING__USE_KELLY_CRITERION=false
When true: Uses Kelly Criterion formula for dynamic sizing based on EV and variance
When false: Uses fixed percentage sizing: size = min(balance) × max_exchange_utilization × leverage
Recommendation: true for optimal capital allocation, false for simpler, predictable sizing

WEALTHTRADINGKELLY_FRACTION

Fraction of full Kelly size to use (0.0 - 1.0).

Type: Decimal
Default: 0.25 (quarter-Kelly for conservative sizing)
TOML Path: trading.kelly_fraction
Example: WEALTH__TRADING__KELLY_FRACTION=0.5
Purpose: Reduces risk by using only a fraction of the mathematically optimal Kelly size
Recommendation: Keep at 0.25 or lower for funding rate arbitrage

WEALTHTRADINGMAX_NOTIONAL_PER_SYMBOL

Maximum notional value per symbol across all exchanges in USD.

Type: Decimal
Default: 10000 ($10,000)
TOML Path: trading.max_notional_per_symbol
Example: WEALTH__TRADING__MAX_NOTIONAL_PER_SYMBOL=20000
Purpose: Hard cap on position size regardless of Kelly calculation
Applied: Per symbol across all exchanges (e.g., total BTC exposure)

WEALTHTRADINGMAX_EXCHANGE_UTILIZATION

Maximum percentage of free balance to use for a single position (0.0 - 1.0).

Type: Decimal
Default: 0.5 (50%)
TOML Path: trading.max_exchange_utilization
Example: WEALTH__TRADING__MAX_EXCHANGE_UTILIZATION=0.3
Purpose: Prevents over-concentration on one exchange
Reference Capital: Uses min(long_exchange_balance, short_exchange_balance) for sizing

Example Kelly Sizing:

EV: 30 bps per 8h (0.003)
Variance: 0.0001 (estimated spread volatility)
Kelly fraction: EV / variance = 0.003 / 0.0001 = 30
Reference capital: min($50,000, $80,000) = $50,000
Leverage: 10x
Full Kelly size: $50,000 × 30 × 10 = $15,000,000 (unrealistic)

Apply kelly_cap (0.25): $15,000,000 × 0.25 = $3,750,000
Apply max_notional cap: min($3,750,000, $10,000) = $10,000
Apply utilization cap: min($10,000, $50,000 × 0.5) = $10,000

Final position size: $10,000 (binding constraint: notional cap)

Slippage Protection Configuration

WEALTHRISKMAX_SLIPPAGE_BPS

Maximum allowed slippage in basis points.

Type: Integer
Default: 50 (0.5%)
TOML Path: risk.max_slippage_bps
Example: WEALTH__RISK__MAX_SLIPPAGE_BPS=30

WEALTHRISKMIN_SLIPPAGE_BPS

Minimum slippage buffer in basis points (always applied).

Type: Integer
Default: 10 (0.1%)
TOML Path: risk.min_slippage_bps
Example: WEALTH__RISK__MIN_SLIPPAGE_BPS=5

WEALTHRISKMARKET_ORDER_FALLBACK_ENABLED

Whether to fall back to MARKET orders if LIMIT order doesn't fill.

Type: Boolean
Default: true
TOML Path: risk.market_order_fallback_enabled
Values: true | false
Example: WEALTH__RISK__MARKET_ORDER_FALLBACK_ENABLED=false

WEALTHRISKLIMIT_ORDER_TIMEOUT_SECS

Timeout before falling back to MARKET order (seconds).

Type: Integer
Default: 5
TOML Path: risk.limit_order_timeout_secs
Example: WEALTH__RISK__LIMIT_ORDER_TIMEOUT_SECS=10

WEALTHRISKSLIPPAGE_VOLATILITY_MULTIPLIER

Multiplier to increase slippage tolerance in volatile markets.

Type: Float
Default: 1.5 (50% wider)
TOML Path: risk.slippage_volatility_multiplier
Example: WEALTH__RISK__SLIPPAGE_VOLATILITY_MULTIPLIER=2.0

Trailing Stop Configuration

Note: Trailing stop configuration is defined in config.toml or via environment variables below. See Phase 1 Features for detailed usage.

WEALTHRISKTRAILING_STOPS_ENABLED

Enable trailing stop loss functionality.

Type: Boolean
Default: true
TOML Path: risk.trailing_stops_enabled
Values: true | false
Example: WEALTH__RISK__TRAILING_STOPS_ENABLED=true

WEALTHRISKTRAILING_STOP_ACTIVATION

Profit threshold to activate trailing stop (as decimal percentage).

Type: Decimal (0.0-1.0)
Default: 0.03 (3%)
TOML Path: risk.trailing_stop_activation
Example: WEALTH__RISK__TRAILING_STOP_ACTIVATION=0.05

WEALTHRISKTRAILING_STOP_DISTANCE

Maximum allowed profit retracement before exit (as decimal percentage).

Type: Decimal (0.0-1.0)
Default: 0.40 (40% from peak)
TOML Path: risk.trailing_stop_distance
Example: WEALTH__RISK__TRAILING_STOP_DISTANCE=0.3

WEALTHRISKTRAILING_STOP_MIN_LOCK

Minimum profit to lock in (prevents exit below this level).

Type: Decimal (0.0-1.0)
Default: 0.02 (2%)
TOML Path: risk.trailing_stop_min_lock
Example: WEALTH__RISK__TRAILING_STOP_MIN_LOCK=0.015

WEALTHRISKSLIPPAGE_AWARE_EXITS

Consider estimated slippage cost in exit decisions. When enabled, positions with remaining potential profit less than expected closing costs may be held for one more funding period.

Type: Boolean
Default: true
TOML Path: risk.slippage_aware_exits
Example: WEALTH__RISK__SLIPPAGE_AWARE_EXITS=false

Fee Configuration

WEALTHRISKFEES__ESTIMATED_SLIPPAGE

Estimated slippage for market orders (as decimal).

Type: Decimal (0.0-1.0)
Default: 0.001 (0.1%)
TOML Path: risk.fees.estimated_slippage
Example: WEALTH__RISK__FEES__ESTIMATED_SLIPPAGE=0.002

Exchange Fee Overrides

Override default exchange fee rates in config.toml under [risk.fees.<exchange>]:

Binance

[risk.fees.binance]
maker = 0.0002  # 0.02% - Maker fee
taker = 0.0004  # 0.04% - Taker fee
funding_interval_hours = 8

Bybit

[risk.fees.bybit]
maker = 0.0001  # 0.01% - Maker fee
taker = 0.0006  # 0.06% - Taker fee
funding_interval_hours = 8

HyperLiquid

[risk.fees.hyperliquid]
maker = 0.0000   # 0.00% - Maker rebate
taker = 0.00035  # 0.035% - Taker fee
funding_interval_hours = 8

Metrics Server Configuration

WEALTHOBSERVABILITYMETRICS_PORT

Port for metrics and health check server.

Type: Integer
Default: 9090
TOML Path: observability.metrics_port
Example: WEALTH__OBSERVABILITY__METRICS_PORT=8080

WEALTHOBSERVABILITYMETRICS_BIND_ADDRESS

Bind address for metrics server.

Type: String (IP address)
Default: 0.0.0.0 (all interfaces)
TOML Path: observability.metrics_bind_address
Example: WEALTH__OBSERVABILITY__METRICS_BIND_ADDRESS=127.0.0.1

Execution Mode

WEALTHEXECUTIONMODE

Execution mode for order placement.

Type: String ("paper" | "live" | "dryrun")
Default: paper
TOML Path: execution.mode
Example: WEALTH__EXECUTION__MODE=live

Modes:

paper - Simulated trading with mock order fills
live - Real trading with actual exchange orders
dryrun - Logs order intent but doesn't execute (useful for testing strategy logic)

Note: The old PAPER_TRADING and DRY_RUN boolean flags have been replaced with WEALTH__EXECUTION__MODE in v0.33.0 for clearer semantics.

Exchange Credentials

Credential Loading Strategy

The bot automatically tries to load credentials in the following order:

Encrypted Credentials File (if CREDENTIALS_PASSPHRASE is set)
- Uses credentials.encrypted.json and .credentials.salt
- Most secure, recommended for production
- Managed via wealth credentials CLI commands
Environment Variables (fallback)
- Exchange-specific variables documented below
- Simpler for development and testing

No configuration needed - the bot seamlessly uses whichever method you set up.

Setup Option 1: Encrypted Credentials (Recommended)

Initial Setup:

# 1. Set passphrase in .env
echo "CREDENTIALS_PASSPHRASE=your-secure-passphrase" >> .env

# 2. Create encrypted credentials file
wealth credentials create

# 3. Add exchange credentials
wealth credentials add binance YOUR_API_KEY YOUR_SECRET_KEY
wealth credentials add bybit YOUR_API_KEY YOUR_SECRET_KEY
wealth credentials add hyperliquid 0xWALLET_ADDRESS 0xPRIVATE_KEY

# 4. Verify
wealth verify --encrypted

Advantages:

AES-256-GCM encryption
Credentials never stored in plaintext
Secure key derivation from passphrase (PBKDF2 with 100,000 iterations)
Isolated from source code and version control

Security:

Never commit credentials.encrypted.json or .credentials.salt to version control
Use a strong passphrase (12+ characters, mix of types)
Store passphrase securely (password manager, secrets vault)

CREDENTIALS_PASSPHRASE

Passphrase for encrypting/decrypting the credentials file.

Type: String
Default: None (not set)
Example: CREDENTIALS_PASSPHRASE=my-super-secure-passphrase-2024
Required for: Encrypted credentials (Option 1)
Not needed for: Direct environment variables (Option 2)

When set: Bot loads credentials from credentials.encrypted.json
When not set: Bot loads credentials from environment variables (see below)

Setup Option 2: Environment Variables (Development)

⚠️ Deprecated: Direct exchange credential environment variables are deprecated in v0.21.0+. Use encrypted credentials instead.

For backwards compatibility, you can still set exchange-specific variables:

Binance Futures

BINANCE_API_KEY="your-api-key"
BINANCE_SECRET_KEY="your-secret-key"
BINANCE_TESTNET="false"  # or "true" for testnet

URLs:

Production: https://fapi.binance.com
Testnet: https://testnet.binancefuture.com

Bybit Perpetuals

BYBIT_API_KEY="your-api-key"
BYBIT_SECRET_KEY="your-secret-key"
BYBIT_TESTNET="false"

URLs:

Production: https://api.bybit.com
Testnet: https://api-testnet.bybit.com

HyperLiquid (official SDK)

HYPERLIQUID_SECRET_KEY="0x1234567890abcdef..."  # API wallet private key (64 hex chars)
HYPERLIQUID_API_KEY="0x..."                     # Optional: master wallet address for subkey mode
HYPERLIQUID_TESTNET="false"                     # false=mainnet, true=testnet
HYPERLIQUID_VAULT_MODE="false"                  # false=subkey mode, true=vault mode

Authentication Modes:

Direct Wallet Mode (simplest):
- Only set HYPERLIQUID_SECRET_KEY
- Trades directly with the wallet derived from the private key
- Use for single-wallet trading
API Wallet (Subkey) Mode (recommended):
- Set HYPERLIQUID_SECRET_KEY (API wallet private key)
- Set HYPERLIQUID_API_KEY (master wallet address starting with 0x)
- Set HYPERLIQUID_VAULT_MODE=false (or omit, this is default)
- API wallet must be authorized via HyperLiquid UI under Settings → API
- Queries use master wallet, signing uses API wallet
- Provides better security isolation
Vault Mode:
- Set HYPERLIQUID_SECRET_KEY (vault's API wallet private key)
- Set HYPERLIQUID_API_KEY (vault address starting with 0x)
- Set HYPERLIQUID_VAULT_MODE=true
- For trading on behalf of HyperLiquid vaults only

Note: Signing is handled by the official hyperliquid_rust_sdk (git). It uses Alloy v1.0 PrivateKeySigner under the hood. No custom signing code is implemented in this repository.

Networks:

Mainnet: Arbitrum One (Chain ID 42161)
Testnet: Arbitrum Sepolia (Chain ID 421614)

Security: Private key is stored in secrecy::Secret to prevent accidental logging.

Resilience Configuration

WEALTHEXECUTIONRESILIENCE__CIRCUIT_BREAKER_FAILURES

Number of failures before circuit breaker opens.

Type: Integer
Default: 5
TOML Path: execution.resilience.circuit_breaker_failures
Example: WEALTH__EXECUTION__RESILIENCE__CIRCUIT_BREAKER_FAILURES=10

WEALTHEXECUTIONRESILIENCE__CIRCUIT_BREAKER_TIMEOUT_SECS

Time to wait before testing recovery.

Type: Integer
Default: 60
TOML Path: execution.resilience.circuit_breaker_timeout_secs
Example: WEALTH__EXECUTION__RESILIENCE__CIRCUIT_BREAKER_TIMEOUT_SECS=120

WEALTHEXECUTIONRESILIENCE__CIRCUIT_BREAKER_SUCCESS_THRESHOLD

Number of successful calls needed to close circuit breaker.

Type: Integer
Default: 2
TOML Path: execution.resilience.circuit_breaker_success_threshold
Example: WEALTH__EXECUTION__RESILIENCE__CIRCUIT_BREAKER_SUCCESS_THRESHOLD=3

WEALTHEXECUTIONRESILIENCE__MAX_RETRIES

Maximum retry attempts for API calls.

Type: Integer
Default: 3
TOML Path: execution.resilience.max_retries
Example: WEALTH__EXECUTION__RESILIENCE__MAX_RETRIES=5

WEALTHEXECUTIONRESILIENCE__RETRY_INITIAL_BACKOFF_MS

Initial retry backoff in milliseconds.

Type: Integer
Default: 100
TOML Path: execution.resilience.retry_initial_backoff_ms
Example: WEALTH__EXECUTION__RESILIENCE__RETRY_INITIAL_BACKOFF_MS=200

WEALTHEXECUTIONRESILIENCE__RETRY_MAX_BACKOFF_SECS

Maximum retry backoff in seconds.

Type: Integer
Default: 10
TOML Path: execution.resilience.retry_max_backoff_secs
Example: WEALTH__EXECUTION__RESILIENCE__RETRY_MAX_BACKOFF_SECS=30

WEALTHEXECUTIONRESILIENCE__WEBSOCKET_MAX_RECONNECTS

Maximum WebSocket reconnection attempts.

Type: Integer
Default: 100
TOML Path: execution.resilience.websocket_max_reconnects
Example: WEALTH__EXECUTION__RESILIENCE__WEBSOCKET_MAX_RECONNECTS=50

Observability Configuration

WEALTHOBSERVABILITYMETRICS_PORT

Port number for the metrics/API server.

Type: u16
Default: 9090
TOML Path: observability.metrics_port
Example: WEALTH__OBSERVABILITY__METRICS_PORT=9090

The metrics server exposes:

/metrics - OpenTelemetry metrics info (JSON)
/health, /ready, /live - Health check endpoints
/api/positions, /api/funding_rates, /api/balances - HTTP API for querying bot state
/api/arbitrage_positions - Cached arbitrage positions with funding collected (fast, no API quota)

WEALTHOBSERVABILITYMETRICS_BIND_ADDRESS

IP address for the metrics/API server to bind to.

Type: IP Address
Default: 0.0.0.0 (all network interfaces)
TOML Path: observability.metrics_bind_address
Example: WEALTH__OBSERVABILITY__METRICS_BIND_ADDRESS=127.0.0.1

Set to 127.0.0.1 for localhost-only access (recommended for production with external proxy).

WEALTHOBSERVABILITYOTLP_ENDPOINT

OpenTelemetry Protocol (OTLP) endpoint URL for exporting traces, metrics, and logs.

Type: String (URL)
Default: None (OTLP export disabled, local logging only)
TOML Path: observability.otlp_endpoint
Example: WEALTH__OBSERVABILITY__OTLP_ENDPOINT=http://localhost:4317

When set, the bot will export all telemetry (traces, metrics, logs) to an OpenTelemetry Collector or compatible backend (Tempo, Jaeger, etc.) via gRPC. This replaces the previous Loki push logging with a unified observability pipeline.

URL Validation:

Must start with http:// or https://
Default port for OTLP gRPC is 4317
Supports authentication via standard OTLP headers (configure in collector)
Can be configured via CLI: --loki-url <URL> (flag name preserved for compatibility)

Resource Attributes: You can add custom resource attributes via the OTEL_RESOURCE_ATTRIBUTES environment variable:

export OTEL_RESOURCE_ATTRIBUTES="service.name=wealth-bot,deployment.environment=production,service.version=0.16.0"

WEALTHOBSERVABILITYSERVICE_NAME

Service name for telemetry resource attributes.

Type: String
Default: wealth
TOML Path: observability.service_name
Example: WEALTH__OBSERVABILITY__SERVICE_NAME=wealth-prod

Used as the service.name resource attribute in OpenTelemetry spans, metrics, and logs. Useful when running multiple bot instances.

WEALTHOBSERVABILITYENVIRONMENT

Environment name for telemetry resource attributes (e.g., production, development, staging).

Type: String
Default: development
TOML Path: observability.environment
Example: WEALTH__OBSERVABILITY__ENVIRONMENT=production

Used as a custom resource attribute for environment-based filtering in observability backends.

WEALTHOBSERVABILITYENABLE_TRACES

Enable distributed tracing via OTLP.

Type: Boolean
Default: true
TOML Path: observability.enable_traces
Example: WEALTH__OBSERVABILITY__ENABLE_TRACES=false

When enabled alongside otlp_endpoint, exports trace spans to the OTLP endpoint for distributed tracing visualization in Tempo, Jaeger, or similar backends.

WEALTHOBSERVABILITYTRACE_SAMPLE_RATE

Trace sampling rate (0.0 = never sample, 1.0 = always sample).

Type: Decimal (0.0 - 1.0)
Default: 1.0 (100% of traces)
TOML Path: observability.trace_sample_rate
Example: WEALTH__OBSERVABILITY__TRACE_SAMPLE_RATE=0.1

Lower values reduce observability costs in high-throughput scenarios. Recommended: 1.0 for development, 0.1-0.01 for high-volume production.

WEALTHOBSERVABILITYLOG_FILE

Path to log file for file-based logging.

Type: String (file path)
Default: None (logs to stdout only)
TOML Path: observability.log_file
Example: WEALTH__OBSERVABILITY__LOG_FILE=/var/log/wealth.log

When set, logs are written to both stdout and the specified file. Useful for debugging without disrupting OTLP telemetry.

Configuration Methods:

Environment Variables:

export WEALTH__OBSERVABILITY__OTLP_ENDPOINT=http://localhost:4317
export WEALTH__OBSERVABILITY__SERVICE_NAME=wealth-prod
export WEALTH__OBSERVABILITY__ENVIRONMENT=production

TOML Config:

[observability]
otlp_endpoint = "http://localhost:4317"
service_name = "wealth-prod"
environment = "production"

JSON Config (Legacy):

{
  "observability": {
    "otlp_endpoint": "http://localhost:4317",
    "service_name": "wealth-prod",
    "environment": "production"
  }
}

Configuration Precedence: Environment variables > TOML config > JSON config (legacy) > Defaults

See Logging Guide for complete OpenTelemetry setup instructions.

Configuration Examples

Development (Paper Trading)

# Trading
WEALTH__TRADING__MIN_FUNDING_SPREAD=0.0004
WEALTH__TRADING__MAX_POSITION_USD=10000
WEALTH__TRADING__POSITION_SIZE_PERCENT=0.3
WEALTH__TRADING__TARGET_PROFIT_PERCENT=0.05

# Execution
WEALTH__EXECUTION__MODE=paper

# Observability
RUST_LOG=info
# Optional: Enable OTLP export for development
WEALTH__OBSERVABILITY__OTLP_ENDPOINT=http://localhost:4317
WEALTH__OBSERVABILITY__SERVICE_NAME=wealth-dev
WEALTH__OBSERVABILITY__ENVIRONMENT=development

Production (Live Trading)

# Trading
WEALTH__TRADING__MIN_FUNDING_SPREAD=0.0004
WEALTH__TRADING__MAX_POSITION_USD=50000
WEALTH__TRADING__POSITION_SIZE_PERCENT=0.25
WEALTH__TRADING__TARGET_PROFIT_PERCENT=0.05

# Execution
WEALTH__EXECUTION__MODE=live

# Exchange Credentials (use encrypted credentials file instead)
CREDENTIALS_PASSPHRASE="your-secure-passphrase"

# Observability
RUST_LOG=info
WEALTH__OBSERVABILITY__OTLP_ENDPOINT=https://otlp.example.com:4317
WEALTH__OBSERVABILITY__SERVICE_NAME=wealth-prod
WEALTH__OBSERVABILITY__ENVIRONMENT=production

Testing Configuration

# Execution
WEALTH__EXECUTION__MODE=paper

# Conservative settings
WEALTH__TRADING__MAX_POSITION_USD=100
WEALTH__TRADING__POSITION_SIZE_PERCENT=0.1

# Verbose logging
RUST_LOG=debug

Configuration Validation

All configuration is validated on startup with fail-fast behavior.

Validation Process:

Type validation during JSON parsing (serde)
Cross-field validation after loading
Clear error messages indicating which rule failed

Validated Constraints:

All numeric thresholds must be positive
Percentages must be between 0 and 1
Port numbers must be non-zero
At least one instrument must be configured
Retry counts and timeouts must be valid

Example Error:

ERROR Configuration validation failed: Metrics port must be non-zero
ERROR Configuration validation failed: At least one instrument must be configured

Debugging Configuration

Use the config show --show-sources command to see where each configuration value comes from:

$ wealth config show --show-sources

📋 Configuration Sources

Shows: default → file (config.toml) → env vars → ✓ final value

🎯 Trading Configuration:
  min_funding_spread:
    default:  0.0004
    file:     0.0005
    env:      0.0006
    ✓ final:  0.0006 ← using environment override

  max_position_usd:
    default:  10000
    ✓ final:  10000 ← using default

This helps identify configuration issues and understand which values are active.

Environment Variable Precedence

Configuration loading happens in this order:

Environment variables with WEALTH__ prefix (highest priority) - Override all other settings
TOML/JSON configuration file (config.toml or config.json) - Provides structured config
Default values (lowest priority) - Built-in sensible defaults

Example: If you set WEALTH__TRADING__MIN_FUNDING_SPREAD=0.001 in .env and min_funding_spread = 0.0004 in config.toml, the environment variable takes precedence.

Best Practices:

Use config.toml for static configuration (instruments, leverage, strategy parameters)
Use environment variables with WEALTH__ prefix for deployment-specific settings
Use encrypted credentials file for API keys (safer than environment variables)
Use .env file for local development overrides

Using with Different Tools

Systemd

[Service]
Environment="WEALTH__EXECUTION__MODE=live"
Environment="CREDENTIALS_PASSPHRASE=your-passphrase"

Docker

docker run -e WEALTH__EXECUTION__MODE=live \
  -e CREDENTIALS_PASSPHRASE=xxx \
  -v $(pwd)/config.toml:/app/config.toml \
  wealth

Docker Compose

environment:
  - WEALTH__EXECUTION__MODE=live
  - CREDENTIALS_PASSPHRASE=${CREDENTIALS_PASSPHRASE}
volumes:
  - ./config.toml:/app/config.toml:ro
  - ./credentials.encrypted.json:/app/credentials.encrypted.json:ro

Kubernetes

env:
- name: WEALTH__EXECUTION__MODE
  value: "live"
- name: CREDENTIALS_PASSPHRASE
  valueFrom:
    secretKeyRef:
      name: wealth-secrets
      key: credentials-passphrase
volumeMounts:
- name: config
  mountPath: /app/config.toml
  subPath: config.toml

Configuration Debugging

Config Show Sources Command

Display where each configuration value comes from (default → file → env):

wealth config --show-sources

Example Output:

📋 Configuration Sources

Shows: default → file (config.toml) → env vars → ✓ final value

🎯 Trading Configuration:
  min_funding_spread:
    default:  0.0004
    env:      0.0008
    ✓ final:  0.0008 ← using environment override
    
  max_position_usd:
    default:  10000
    file:     15000
    ✓ final:  15000 ← from config file

Use cases:

Debug which source is being used for each config value
Verify environment variable overrides are working
Identify misconfigured values before running the bot

Validation Error Messages

Configuration errors now include source tracking (v0.33.0+):

File-based error:

Error: Invalid value for leverage.default: Must be greater than 0 
(from config file: config.toml)

Environment variable error:

Error: Invalid value for trading.min_funding_spread: Cannot be negative
(from environment variable: WEALTH__TRADING__MIN_FUNDING_SPREAD)

Default value error:

Error: Invalid value for risk.max_slippage_bps: must be positive
(using default value)

Benefits:

Quickly identify where the problematic value is configured
Avoid checking multiple configuration sources manually
Clear distinction between file errors and env var errors

Config Validation

Validate configuration without starting the bot:

wealth config --validate

Output:

🔍 Validating configuration...

  ✓ Loading config.toml
  ✓ TOML parsing successful
  ✓ Environment variable parsing successful
  ✓ Validating configuration... OK

✅ Configuration is valid!
   Instruments: 6
   Metrics Port: 9090
   Execution Mode: Paper

Common validation errors:

Negative values (spreads, slippage, leverage)
Out-of-range percentages (must be 0.0-1.0)
Invalid exchange names (must be: binance, bybit, hyperliquid)
Missing required instruments

CLI Reference

Complete command-line interface reference for the Wealth Trading Bot

Overview

The wealth trading bot features a comprehensive command-line interface (CLI) built with clap. The CLI provides subcommands for all major operations, making it easy to manage the bot from the terminal.

Global Options

Available on all commands:

-v                         # Info-level logging
-vv                        # Debug-level logging
-vvv                       # Trace-level logging (very verbose)
-q, --quiet                # Error-level only (warnings suppressed)
-l, --log-file <FILE>      # Path to log file (default: stdout)
-c, --config <FILE>        # Path to custom configuration file
-h, --help                 # Show help information
-V, --version              # Show version information

Environment Variables:

WEALTH__OBSERVABILITY__LOG_FILE   # Default log file path
WEALTH_CONFIG                      # Default configuration file path

Commands

`wealth run` - Run the Trading Bot

Start the trading bot. All configuration is managed via config.toml or environment variables.

Usage:

wealth run [OPTIONS]

Options:

--enable-discovery     # Enable dynamic pair discovery (overrides config)

Note: The run command is required. Running wealth without a command displays help.

Configuration:

All runtime settings are configured via:

Config file: config.toml (persistent settings)
Environment variables: WEALTH__* prefix (see Configuration Guide)

Common Configuration Examples:

# Paper trading mode (via environment variable)
export WEALTH__EXECUTION__MODE=paper
wealth run

# Live trading mode (via environment variable)
export WEALTH__EXECUTION__MODE=live
wealth run

# Custom metrics port
export WEALTH__OBSERVABILITY__METRICS_PORT=8080
wealth run

# OpenTelemetry export
export WEALTH__OBSERVABILITY__OTLP_ENDPOINT=http://localhost:4317
export WEALTH__OBSERVABILITY__SERVICE_NAME=wealth-prod
export WEALTH__OBSERVABILITY__ENVIRONMENT=production
wealth run

Examples:

# Run with default settings from config.toml
wealth run

# Show help
wealth

# Log to file
wealth run --log-file /tmp/wealth.log

# Verbose logging with file output
wealth run -v --log-file /var/log/wealth.log

# Very verbose (debug level)
export WEALTH__OBSERVABILITY__SERVICE_NAME=wealth-prod
export WEALTH__OBSERVABILITY__ENVIRONMENT=production
wealth run -vv

# Hybrid mode: file + Loki push
wealth run --log-file /tmp/wealth.log \
  --loki-url http://localhost:3100

Environment Variables:

WEALTH__EXECUTION__MODE - Execution mode: "paper", "live", or "dryrun" (default: paper)
WEALTH__OBSERVABILITY__METRICS_PORT - Metrics port (default: 9090)
WEALTH__OBSERVABILITY__METRICS_BIND_ADDRESS - Bind address (default: 0.0.0.0)
WEALTH__OBSERVABILITY__OTLP_ENDPOINT - OpenTelemetry OTLP endpoint URL (e.g., http://localhost:4317)
WEALTH__OBSERVABILITY__SERVICE_NAME - Service name for telemetry resource attributes (default: wealth)
WEALTH__OBSERVABILITY__ENVIRONMENT - Environment name for telemetry resource attributes (default: development)

Note: All configuration can also be set in config.toml. See Configuration Guide for details.

`wealth dashboard` - Interactive TUI Dashboard

🖥️ Status: Fully implemented (requires tui feature)

Run the trading bot with an interactive terminal dashboard for real-time monitoring and control.

Wealth TUI Dashboard

Usage:

wealth dashboard [OPTIONS]

Options:

--enable-discovery     # Enable dynamic pair discovery
--tick-rate <MS>       # UI refresh rate in milliseconds (default: 250)

Examples:

# Run with TUI dashboard (default settings)
wealth dashboard

# Enable dynamic pair discovery
wealth dashboard --enable-discovery

# Slower refresh rate (for remote connections)
wealth dashboard --tick-rate 500

📖 See TUI Dashboard Guide for full documentation including keybindings, tabs, and troubleshooting.

`wealth verify` - Verify Configuration

✅ Status: Fully implemented

Verify that configuration files and credentials are valid, with optional API validation.

Usage:

wealth verify [OPTIONS]

Options:

--check-credentials    # Make test API calls to verify credentials work
--encrypted            # Verify encrypted credential decryption

Examples:

# Basic verification
wealth verify

# Verify with real API calls (recommended)
wealth verify --check-credentials

# Verify encrypted credentials
export CREDENTIALS_PASSPHRASE="your_passphrase"
wealth verify --encrypted --check-credentials

Output Format:

🔍 Configuration Verification

✅ Configuration file loaded successfully
✅ Strategy parameters validated
✅ Instruments configured: 3

🔐 Credential Verification:
  ✅ BinanceFutures: Valid (API call successful)
  ✅ BybitPerpetualsUsd: Valid (API call successful)
  ✅ HyperLiquid: Valid (API call successful)

All checks passed!

`wealth config` - Display Configuration

Show current configuration settings.

Usage:

wealth config [OPTIONS]

Options:

--strategy       # Show only strategy configuration
--execution      # Show only execution configuration
--credentials    # Show credential status (redacted)
--validate       # Validate configuration file
--show-sources   # Show configuration sources (defaults, file, env vars)

Examples:

# Show all configuration
wealth config

# Validate configuration
wealth config --validate

# Debug configuration with source tracking
wealth config --show-sources

Sample Output (with --show-sources):

📋 Configuration Sources

Shows: default → file (config.toml) → env vars → ✓ final value

🎯 Trading Configuration:
  min_funding_spread:
    default:  0.0004
    file:     0.0005
    env:      0.0006
    ✓ final:  0.0006 ← using environment override

  max_position_usd:
    default:  10000
    ✓ final:  10000 ← using default

`wealth license` - Check License Status

✅ Status: Fully implemented

Check license status, validate connectivity, and show machine fingerprint for support.

Subcommands

`check` - Check License Status

wealth license check

Displays:

License configuration (account, product)
Machine fingerprint
License validity status
Expiration information (if available)

Example Output:

📋 License Information:
   Account:     fcaa9cfa-8236-4270-81b6-da3ddd238d70
   Product:     6f04c081-10d2-4222-8310-b314d82ac410
   Machine ID:  a1b2c3d4-e5f6-7890-abcd-ef1234567890

✅ License is valid!

`show-fingerprint` - Show Machine Fingerprint

wealth license show-fingerprint

Displays your machine's unique identifier. Use this when:

Contacting support about licensing issues
Requesting additional machine activations
Troubleshooting activation problems

Example Output:

📌 Machine ID: a1b2c3d4-e5f6-7890-abcd-ef1234567890

This unique identifier is used to activate your license on this machine.
Share this ID with support when requesting license assistance.

License Status: ✅ Configured
Account: fcaa9cfa-8236-4270-81b6-da3ddd238d70

`validate` - Validate License Connectivity

wealth license validate

Performs comprehensive license validation:

Tests connection to Keygen API
Validates license configuration
Checks machine activation status
Provides detailed error messages with troubleshooting guidance

Example Output (Success):

🔍 Configuration:
   Account:  fcaa9cfa-8236-4270-81b6-da3ddd238d70
   Product:  6f04c081-10d2-4222-8310-b314d82ac410
   API URL:  https://api.keygen.sh

🖥️  Machine ID: a1b2c3d4-e5f6-7890-abcd-ef1234567890

📡 Connecting to Keygen API...
✅ License validation successful!

Your license is active and this machine is properly configured.

Example Output (Error with Guidance):

❌ License validation failed: Network error contacting license server

💡 Network Issue:
   - Check internet connectivity
   - Verify firewall allows HTTPS to api.keygen.sh
   - Check proxy configuration if applicable

Environment Variables:

WEALTH__LICENSING__LICENSE_KEY - Your license key (required)
WEALTH__LICENSING__API_URL - Custom Keygen API URL (optional, debug builds only)

`deactivate` - Deactivate License from Current Machine

wealth license deactivate --confirm

Deactivates your license from the current machine, allowing you to transfer it to another machine.

Use Cases:

Migrating to a new server
Replacing hardware
Switching between development and production machines
Freeing up a machine activation slot

Safety Features:

Requires --confirm flag to prevent accidental deactivation
Shows warning and confirmation prompt without flag
Verifies license status before deactivation
Provides clear next steps after deactivation

Example Usage:

# Show warning and instructions (safe dry-run)
wealth license deactivate

# Actually deactivate (requires confirmation)
wealth license deactivate --confirm

Example Output (Dry-Run):

⚠️  License Deactivation

This will deactivate your license from this machine.
You can reactivate it on another machine by running the bot there.

To confirm, run:
  wealth license deactivate --confirm

Example Output (Successful Deactivation):

📋 Deactivation Details:
  Account:     fcaa9cfa-8236-4270-81b6-da3ddd238d70
  Product:     6f04c081-10d2-4222-8310-b314d82ac410
  Machine ID:  a1b2c3d4-e5f6-7890-abcd-ef1234567890

🔍 Finding machine activation...
✅ License found: lic_1234567890

✅ License deactivated successfully!

📦 Your license is now available for activation on another machine.
  To activate on a new machine:
  1. Copy your license key to the new machine
  2. Set WEALTH__LICENSING__LICENSE_KEY environment variable
  3. Run: wealth run
  (The license will activate automatically)

Example Output (Already Deactivated):

ℹ️  License is not currently activated on this machine.

No action needed - you can activate on another machine anytime.

Error Handling:

Invalid license key → Clear error message
Network issues → Troubleshooting guidance
Already deactivated → Informational message
License not found → Verification steps

Environment Variables:

WEALTH__LICENSING__LICENSE_KEY - Your license key (required)

Notes:

Deactivation is immediate and cannot be undone
After deactivation, simply run the bot on a new machine to reactivate
No limit on number of deactivations/reactivations
Your license key remains valid after deactivation

Related Documentation:

Licensing Guide - Complete licensing documentation
Troubleshooting - Common licensing issues

`wealth credentials` - Manage Encrypted Credentials

Manage encrypted credential storage.

Subcommands

`create` - Create New Encrypted Config

wealth credentials create

Creates .credentials.salt and credentials.encrypted.json files.

`add` - Add Exchange Credentials

wealth credentials add <EXCHANGE> [OPTIONS]

Arguments:

<EXCHANGE> - Exchange name (binance, bybit, hyperliquid)

Options:

--api-key <KEY> - API key (or master wallet/vault address for HyperLiquid)
--secret-key <KEY> - Secret key (or API wallet private key for HyperLiquid)
--testnet - Use testnet mode
--vault-mode - HyperLiquid only: Enable vault mode (API_KEY is vault address)

Security Recommendation:

For maximum security, omit the --api-key and --secret-key options to use interactive prompts. This prevents secrets from appearing in shell history.

# Recommended: Interactive mode (secrets not in shell history)
wealth credentials add binance
# Prompts for:
#   Enter API key: <your-api-key>
#   Enter secret key (hidden): <your-secret-key>

# Less secure: CLI arguments (visible in shell history)
wealth credentials add binance --api-key YOUR_API_KEY --secret-key YOUR_SECRET_KEY

HyperLiquid Examples:

# Direct wallet mode (simplest) - interactive
wealth credentials add hyperliquid
# API key: 0xYourWalletAddress
# Secret key: 0xYourPrivateKey

# API wallet (subkey) mode (recommended) - interactive
wealth credentials add hyperliquid
# API key: 0xYourMasterWalletAddress
# Secret key: 0xAPIWalletPrivateKey

# Vault mode - interactive
wealth credentials add hyperliquid --vault-mode
# API key: 0xVaultAddress
# Secret key: 0xAPIWalletPrivateKey

`verify` - Verify Encrypted Credentials

wealth credentials verify

Requires CREDENTIALS_PASSPHRASE environment variable.

`list` - List Configured Exchanges

wealth credentials list

`remove` - Remove Credentials

wealth credentials remove <EXCHANGE> --confirm

Safety: Requires --confirm flag.

`wealth health` - System Health Checks

✅ Status: Fully implemented

Check system health including exchange connectivity, WebSocket connections, and data freshness.

Usage:

wealth health [OPTIONS]

Options:

--exchanges     # Check exchange API connectivity (validates credentials)
--websockets    # Check WebSocket connectivity
--all           # Run all health checks

Examples:

# Check all health indicators
wealth health --all

# Check only exchange connectivity
wealth health --exchanges

# Check only WebSocket connections
wealth health --websockets

Output Format:

🏥 Health Check

📡 Exchange API Connectivity:
  🟡 BinanceFutures ... ✅ OK (142ms)
  🟠 BybitPerpetualsUsd ... ✅ OK (98ms)
  🔵 HyperLiquid ... ✅ OK (76ms)

  Summary: 3/3 exchanges accessible

🔌 WebSocket Connectivity:
  ⚠️  WebSocket health checks require running bot instance
  💡 Tip: Use 'curl http://localhost:9090/ready' while bot is running

✅ Health check complete

`wealth balance` - Display Account Balances

✅ Status: Fully implemented

Usage:

wealth balance [OPTIONS]

Options:

--exchange <NAME>    # Filter by exchange (binance, bybit, hyperliquid)
--total              # Show total USD equivalent balance
--detailed           # Show detailed asset breakdown with total and free balances

Examples:

# View all balances
wealth balance

# View specific exchange
wealth balance --exchange binance

# Show total USD equivalent
wealth balance --total

# Detailed breakdown
wealth balance --detailed

Output Format:

💰 Account Balances

📈 BinanceFutures
  USDT: 5000.00

📈 BybitPerpetualsUsd  
  USDT: 3000.00

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
💵 Total Available (USDT equivalent): $8000.00

`wealth positions` - Display Current Positions

✅ Status: Fully implemented

Display current open positions across exchanges with funding rates and projected PnL. Automatically uses the bot's HTTP API when running for fast access to in-memory state.

Usage:

wealth positions [OPTIONS]

Options:

--exchange <NAME>    # Filter by exchange (binance, bybit, hyperliquid)
--pnl                # Show P&L summary for all positions
--funding            # Show detailed funding rate breakdown

Smart Data Fetching:

Bot running: Uses /api/positions and /api/funding_rates (fast, no API quota)
Bot not running: Queries exchanges directly (slower, uses API quota)
API port is configurable (default: 9090, set via WEALTH__OBSERVABILITY__METRICS_PORT)

Examples:

# View all positions (uses API if bot running)
wealth positions

# View positions on specific exchange
wealth positions --exchange binance

# Show detailed funding breakdown
wealth positions --funding

# Show with P&L summary
wealth positions --pnl

# Combine funding and P&L
wealth positions --funding --pnl

Output Format:

Positions table now includes funding rate information by default:

📊 Current Positions

  🔗 Using live data from running bot instance

# OR (if bot not running):

📊 Current Positions

  📡 Querying exchanges directly (bot not running or API unavailable)

╭──────────┬──────────┬───────┬────────┬─────────────┬────────────┬────────────────┬──────────────┬──────────╮
│ Exchange │ Symbol   │ Side  │ Size   │ Entry Price │ Mark Price │ Unrealized PnL │ Funding Rate │ Est. 8h  │
├──────────┼──────────┼───────┼────────┼─────────────┼────────────┼────────────────┼──────────────┼──────────┤
│ binance  │ BTCUSDT  │ LONG  │ 0.1000 │ $95000.00   │ $96000.00  │ $100.00        │ 0.0100%      │ -9.50    │
│ bybit    │ BTCUSDT  │ SHORT │ 0.1000 │ $95200.00   │ $96000.00  │ -$80.00        │ 0.0150%      │ +14.28   │
╰──────────┴──────────┴───────┴────────┴─────────────┴────────────┴────────────────┴──────────────┴──────────╯

Columns:

Exchange: Exchange name
Symbol: Trading pair
Side: LONG or SHORT
Size: Position size in base asset
Entry Price: Average entry price
Mark Price: Current mark price
Unrealized PnL: Unrealized profit/loss
Funding Rate: Current funding rate percentage
Est. 8h: Estimated funding payment in next 8h (+ = receive, - = pay)

With --funding flag:

Additional detailed summary section:

📈 Summary:
  Total Long Notional:  $9600.00
  Total Short Notional: $9600.00
  Net Exposure:         $0.00

💰 Funding Rate PnL (Projected):
  binance BTCUSDT LONG: 0.0100% → -$9.50 USD (next 8h)
  bybit BTCUSDT SHORT: 0.0150% → +$14.28 USD (next 8h)

  Projected Next 8h:    +$4.78 USD
  Est. Daily Projected: +$14.34 USD (3 payments/day)

💹 Funding Collected (Actual):
  BTCUSDT (binance ↔ hyperliquid): +$15.50 (held 2d 5h)
  ETHUSDT (bybit ↔ binance): +$8.25 (held 1d 12h)

  Total Funding Collected: +$23.75

Funding PnL Details:

Projected: Estimated funding payment for next 8h period based on current rates
Actual (Collected): Real funding fees collected from active arbitrage positions (requires bot to be running)
Long positions pay funding when rate is positive (negative PnL)
Short positions receive funding when rate is positive (positive PnL)
Funding collected is tracked by the bot's FundingTracker background task

Note:

Position data fetched from bot's API or directly from exchange APIs
Funding rates always displayed in table for immediate visibility
Use --funding flag for detailed breakdown and projections

`wealth funding` - Display Funding Rates

✅ Status: Fully implemented

Display current funding rates, spreads, and calculate funding PnL for active positions.

Usage:

wealth funding [OPTIONS]

Options:

--symbol <SYMBOL>    # Filter by specific symbol (e.g., BTCUSDT)
--spreads            # Show funding rate spreads between exchanges
--pnl                # Show PnL from funding rates for current positions

Examples:

# View all funding rates
wealth funding

# View funding rate for specific symbol
wealth funding --symbol BTCUSDT

# Show spreads between exchanges
wealth funding --spreads

# Show funding rate PnL for current positions
wealth funding --pnl

# Combine spreads and PnL
wealth funding --spreads --pnl

Output Format:

📈 Funding Rates

┌─────────────┬───────────────┬────────────┬──────────────────────┐
│ Symbol      │ Exchange      │ Rate       │ Next Funding         │
├─────────────┼───────────────┼────────────┼──────────────────────┤
│ BTCUSDT     │ Binance       │    0.0100% │ in 5h 23m            │
│ BTCUSDT     │ Bybit         │    0.0500% │ in 5h 23m            │
│             │ Spread: 0.0400% (Bybit - Binance)                  │
└─────────────┴───────────────┴────────────┴──────────────────────┘

💰 Funding Rate PnL:
  Binance BTCUSDT LONG (0.5000 @ $95000.00): 0.0100% → -$4.75 USD (8h)
  Bybit BTCUSDT SHORT (0.5000 @ $95000.00): 0.0500% → +$23.75 USD (8h)

  Net 8h Funding:       +$19.00 USD
  Est. Daily Funding:   +$57.00 USD (3 payments/day)
  Est. Monthly Funding: +$1710.00 USD (30 days)

PnL Calculation:

Long positions: Pay funding when rate is positive (cost), receive when negative (income)
Short positions: Receive funding when rate is positive (income), pay when negative (cost)
Formula: funding_payment = notional_value × funding_rate / 100
Notional value: position_size × entry_price

Note: The --pnl flag fetches both funding rates and current positions from all configured exchanges to calculate expected funding payments.

`wealth close` - Close Positions

✅ Status: Fully implemented

Close open positions on exchanges with reduce-only market orders.

Usage:

wealth close [OPTIONS] --confirm

Options:

--exchange <NAME>    # Close positions on specific exchange (binance, bybit, hyperliquid)
--symbol <SYMBOL>    # Close positions for specific symbol (e.g., BTCUSDT)
--all                # Close all positions (if --symbol not specified)
--confirm            # Required confirmation flag (safety)

Examples:

# Close all positions on Binance
wealth close --exchange binance --all --confirm

# Close specific symbol on all exchanges
wealth close --symbol BTCUSDT --all --confirm

# Close all positions on all exchanges
wealth close --all --confirm

# Close specific symbol on specific exchange
wealth close --exchange bybit --symbol ETHUSDT --confirm

Output Format:

🔴 Closing Positions

📊 Checking BinanceFutures positions...
  Found 2 position(s) to close:
    BTCUSDT Long 0.1 @ 35000.00 (PnL: $125.50)
      ✅ Closed (Order ID: 12345678)
    ETHUSDT Short 2.0 @ 2000.00 (PnL: $45.20)
      ✅ Closed (Order ID: 12345679)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Successfully closed: 2

Safety Features:

Requires --confirm flag for all operations
Uses reduce-only market orders (never increases position)
Skips dust positions automatically
Detailed error reporting per position

Note: This command fetches current positions from exchanges and closes them immediately. For graceful shutdown with automatic position closing, use Ctrl+C instead.

`wealth stats` - Trading Statistics

✅ Status: Fully implemented

Display trading performance statistics from the in-memory performance history repository.

Usage:

wealth stats [OPTIONS]

Options:

--period <PERIOD>    # Time period: 1h, 24h, 7d, 30d, 90d, all (default: 24h)
--detailed           # Show detailed breakdown with recent trades

Examples:

# View last 24 hours (default)
wealth stats

# View last 7 days
wealth stats --period 7d

# View all-time with details
wealth stats --period all --detailed

Output Format:

📊 Trading Statistics

  📅 Period: Last 7 Days (2025-11-07 10:00 - 2025-11-14 10:00)

  📈 Performance Summary:
     Total Trades: 42
     Winning Trades: 35 (83%)
     Losing Trades: 7
     Total P&L: $1,234.56
     Avg Win: $52.30
     Avg Loss: -$12.45
     Sharpe Ratio: 2.15

  � Recent Trades: (with --detailed)
     1. ✅ BTCUSDT → 2025-11-14 09:45 | $65.23 | WIN
     2. ✅ ETHUSDT → 2025-11-14 08:30 | $42.10 | WIN
     ...

Note: Data is collected while the bot is running from the InMemoryPerformanceHistoryRepository. Not persisted between restarts.

Important: Data Availability

The stats command displays data from an in-memory performance history repository. This has important implications:

Data is not persisted: Trade statistics are stored in memory and reset when the bot restarts
CLI creates new instance: Each CLI command creates a fresh, empty repository instance
Bot must be running: To see meaningful data, the bot must be actively running and have executed trades
Consider external monitoring: For persistent statistics, use the Prometheus metrics endpoint with Grafana or similar tools

If you see "No trading history found", this is expected behavior – it means:

The bot hasn't executed any trades in the specified period, OR
The bot was restarted (clearing in-memory history), OR
You're running the CLI while the bot is stopped

`wealth profit` - Total Profit Tracking

✅ Status: Fully implemented

Display cumulative profit/loss tracking with realized and unrealized PnL.

Usage:

wealth profit [OPTIONS]

Options:

--period <PERIOD>         # Time period: 1h, 24h, 7d, 30d, 90d, all (default: all)
--by-exchange             # Show breakdown by exchange
--by-symbol               # Show breakdown by symbol
--include-unrealized      # Include unrealized PnL from open positions

Examples:

# View all-time profit (default)
wealth profit

# View last 7 days with exchange breakdown
wealth profit --period 7d --by-exchange

# View total profit including unrealized PnL
wealth profit --include-unrealized

# Detailed view with all breakdowns
wealth profit --by-exchange --by-symbol --include-unrealized

Output Format:

💰 Profit Tracking

  📅 Period: All Time (1970-01-01 00:00 - 2025-11-28 12:00)

  📊 Realized Profit/Loss (Closed Positions):
     Total Trades: 156
     Total Realized PnL: $2847.92

  📈 Breakdown by Exchange: (with --by-exchange)
     binance (LONG): +$1523.45
     hyperliquid (SHORT): +$1324.47
     ...

  📈 Breakdown by Symbol: (with --by-symbol)
     BTCUSDT: +$1245.30
     ETHUSDT: +$892.15
     SOLUSDT: +$710.47

  💹 Funding Rate Profit/Loss (Active Positions):
     Active Arbitrage Pairs: 3
     Total Funding Collected: +$156.78

     Per-Position Breakdown: (with --by-symbol)
       BTCUSDT (binance ↔ hyperliquid): +$85.50
       ETHUSDT (bybit ↔ binance): +$45.28
       SOLUSDT (binance ↔ hyperliquid): +$26.00

  📊 Unrealized Profit/Loss (Open Positions): (with --include-unrealized)
     Open Positions: 4
     Total Unrealized PnL: $123.45
       binance BTCUSDT LONG: +$45.20
       hyperliquid BTCUSDT SHORT: +$78.25

  💵 Total Profit/Loss:
     Realized PnL:       $2847.92
     Funding Collected:  $156.78
     Unrealized PnL:     $123.45
     ─────────────────────────────
     Total PnL:          +$3128.15

Notes:

Realized PnL comes from InMemoryPerformanceHistoryRepository (closed trades)
Funding Collected shows actual funding fees received from active arbitrage positions (requires bot to be running)
Important: The CLI creates a new repository instance, so it will show zero trades unless the bot is configured with persistent storage that the CLI can access
Unrealized PnL requires bot to be running (fetched via API)
Data is collected during bot runtime and not persisted between restarts
Use --include-unrealized to see current position PnL
Exchange breakdown splits PnL equally (50/50) between long and short legs of arbitrage pairs

Data Availability Summary:

Data Type	Bot Running	Bot Stopped
Realized PnL	Shows current session trades	Always $0.00
Funding Collected	Shows actual funding fees	"Bot not running" message
Unrealized PnL	Shows live position PnL	"Bot not running" message

For persistent profit tracking across restarts, consider:

Exporting data regularly with wealth export
Using Grafana dashboards with Prometheus metrics
Implementing external trade logging

`wealth export` - Export Trading Data

⚠️ Status: Not yet implemented

Export trading data to files for external analysis.

Usage:

wealth export [OPTIONS]

Options:

--format <FORMAT>       # Output format: json or csv (default: json)
--output <FILE>         # Output file path (required)
--data-type <TYPE>      # Data to export: trades, positions, funding, all (default: all)

Examples:

# Export all data to JSON
wealth export --output trades.json

# Export positions to CSV
wealth export --format csv --output positions.csv --data-type positions

Current Status: Command is defined but displays "Not yet implemented" message. Planned for future releases.

`wealth completions` - Generate Shell Completions

✅ Status: Fully implemented

Usage:

wealth completions <SHELL>

Supported Shells: bash, zsh, fish, powershell, elvish

Installation Examples:

Bash:

wealth completions bash > ~/.local/share/bash-completion/completions/wealth
# OR
echo 'source <(wealth completions bash)' >> ~/.bashrc

Zsh:

mkdir -p ~/.zfunc
wealth completions zsh > ~/.zfunc/_wealth
echo 'fpath=(~/.zfunc $fpath)' >> ~/.zshrc

Fish:

wealth completions fish > ~/.config/fish/completions/wealth.fish

Configuration

Custom Config Files

# Use custom configuration file
wealth run --config config.production.json

Environment Variables

All CLI options can be set via environment variables:

export WEALTH__EXECUTION__MODE=live
export WEALTH__OBSERVABILITY__METRICS_PORT=8080
wealth run

Logging Levels

# Default: INFO level
wealth run

# Debug level
wealth run --verbose

# Trace level
wealth run --trace

# Via environment
export RUST_LOG=debug
wealth run

Exit Codes

0 - Success
1 - General error
2 - Configuration error
130 - Interrupted by Ctrl+C (graceful shutdown)

TUI Dashboard

Interactive terminal interface for real-time monitoring and control

Wealth TUI Dashboard

The Wealth Trading System includes a powerful terminal user interface (TUI) built with Ratatui. It provides real-time monitoring of your trading bot with an intuitive keyboard-driven interface.

Quick Start

# Run the TUI dashboard
wealth dashboard

# With dynamic pair discovery enabled
wealth dashboard --enable-discovery

# Custom refresh rate (for slow connections)
wealth dashboard --tick-rate 500

Building with TUI Support

The TUI feature must be enabled at compile time:

cargo build --release --features tui

Pre-built binaries from GitHub releases include TUI support by default.

Dashboard Tabs

The dashboard has four main tabs, accessible with Tab and Shift+Tab:

📊 Dashboard Tab

Dashboard Tab

The overview tab shows:

Bot Status - Running state, uptime, mode (paper/live)
Health Summary - Exchange connectivity, WebSocket status
Key Metrics - Total P&L, open positions, funding collected
Recent Opportunities - Latest arbitrage opportunities detected

📈 Positions Tab

Positions Tab

Monitor your active positions:

Position symbol and direction (long/short sides)
Entry prices and current prices
Unrealized P&L with percentage
Funding collected per position
Position age and last funding payment

Use j/k or arrow keys to select a position, then x to close it.

💰 Funding Tab

Funding Tab

Live funding rates across all exchanges:

Real-time rates updated every tick
Color-coded rates (green positive, red negative)
Annualized rate display
Next funding countdown timer
Spread between exchanges highlighted

📜 Logs Tab

Logs Tab

Real-time event viewer:

Scrollable log history
Color-coded log levels (INFO, WARN, ERROR)
Trade executions, funding payments, system events
Use j/k to scroll through history

Keybindings

Key	Action
`Tab`	Switch to next tab
`Shift+Tab`	Switch to previous tab
`1`-`4`	Jump to specific tab

Scrolling

Key	Action
`j` or `↓`	Scroll down / Next item
`k` or `↑`	Scroll up / Previous item
`g`	Go to top
`G`	Go to bottom
`Page Up`	Scroll up one page
`Page Down`	Scroll down one page

Actions

Key	Action
`p`	Pause/Resume trading bot
`r`	Force refresh data
`x` or `Del`	Close selected position
`?` or `F1`	Toggle help overlay
`q`	Quit dashboard

Closing Positions

When you press x on a selected position, a confirmation dialog appears:

┌─ Close Position ─────────────────┐
│ Close position BTCUSDT?          │
│                                  │
│ Long: Binance                    │
│ Short: Bybit                     │
│ Size: $1000.00                   │
│                                  │
│ [Y] Yes  [N] No  [Esc] Cancel    │
└──────────────────────────────────┘

Press Y to confirm and close the position
Press N or Esc to cancel

The bot will atomically close both legs of the position with rollback protection.

Configuration Options

Command Line Options

wealth dashboard [OPTIONS]

Options:
  --enable-discovery     Enable dynamic pair discovery
  --tick-rate <MS>       UI refresh rate in milliseconds (default: 250)
  -v, -vv, -vvv          Verbosity level
  -c, --config <FILE>    Custom config file

Tick Rate Recommendations

Environment	Tick Rate	Notes
Local	100-250ms	Smooth updates
LAN	250-500ms	Good balance
Remote/SSH	500-1000ms	Reduces bandwidth
Slow connection	1000-2000ms	Minimal updates

Dashboard vs Headless Mode

Feature	`wealth dashboard`	`wealth run`
Interface	Interactive TUI	Headless (logs only)
Resource usage	Higher (rendering)	Lower
Best for	Development, monitoring	Production, servers
Position management	Visual with keyboard	API/CLI only

Troubleshooting

TUI Not Available

If you see "TUI feature not enabled":

# Check if binary was built with TUI
wealth dashboard
# Error: TUI feature not enabled

# Rebuild with TUI feature
cargo build --release --features tui

Display Issues

Garbled output:

# Reset terminal
reset

# Try different terminal emulator
# Recommended: kitty, alacritty, wezterm, iTerm2

Colors not showing:

# Ensure TERM is set correctly
export TERM=xterm-256color
wealth dashboard

Too small:

# Minimum recommended size: 80x24
# Resize terminal or reduce font size

SSH/Remote Issues

# Use slower tick rate
wealth dashboard --tick-rate 1000

# Enable SSH compression
ssh -C user@host

# Use mosh for better responsiveness
mosh user@host -- wealth dashboard

CLI Reference - Full command documentation
Getting Started - Initial setup
Monitoring - Grafana dashboards for production

Trading Strategy Guide

Understanding how the Wealth bot generates profits through funding rate arbitrage

Overview

The Wealth bot implements a funding rate arbitrage strategy that profits from differences in funding rates across cryptocurrency exchanges while maintaining delta-neutral positions (no exposure to price movements).

What Are Funding Rates?

Funding rates are periodic payments exchanged between traders holding long and short positions in perpetual futures contracts. They keep the perpetual futures price aligned with the spot price.

Positive funding rate: Longs pay shorts
Negative funding rate: Shorts pay longs
Payment frequency: Every 8 hours (3 times daily) on most exchanges

How the Strategy Works

The Arbitrage Opportunity

Different exchanges often have different funding rates for the same asset. The bot exploits this by:

Opening a LONG position on the exchange with the lower (or negative) funding rate
Opening a SHORT position on the exchange with the higher funding rate
Collecting the spread between the two rates every funding period

Delta-Neutral = Market-Neutral

Because you hold equal and opposite positions:

If BTC price goes up: Your long profits, your short loses → net zero
If BTC price goes down: Your long loses, your short profits → net zero
Your only P&L comes from the funding rate difference

Visual Example

Exchange A (Binance):    Funding Rate = +0.01%  → You go LONG  (receive 0.01%)
Exchange B (HyperLiquid): Funding Rate = +0.05% → You go SHORT (pay 0.05%)

Your position:
┌─────────────────────────────────────────────────────────────┐
│  LONG 0.1 BTC on Binance    ←→    SHORT 0.1 BTC on HyperLiquid  │
│  (Receive funding)                (Pay funding)                 │
└─────────────────────────────────────────────────────────────┘

Net result per 8-hour period:
  Receive: +0.01% on Binance
  Pay:     -0.05% on HyperLiquid
  ────────────────────────────
  Wait... that's negative!

Actually, you want the OPPOSITE - go SHORT where rates are HIGH:
  Pay:     -0.01% on Binance (you're long, you pay positive funding)
  Receive: +0.05% on HyperLiquid (you're short, you receive positive funding)
  ────────────────────────────
  Net: +0.04% profit per 8 hours

Profit Calculation Example

Position size: $10,000 per side
Funding spread: 0.04% (4 basis points)
Leverage: 10x

Per funding period (every 8 hours):
  Profit = $10,000 × 0.0004 = $4.00

Daily (3 funding periods):
  Profit = $4.00 × 3 = $12.00

Monthly:
  Profit = $12.00 × 30 = $360.00

Annualized APY:
  APY = 0.04% × 3 × 365 = 43.8%

When Does the Bot Open Positions?

The bot continuously monitors funding rates and opens positions when:

Spread exceeds threshold: The funding rate difference meets your configured minimum (default: 0.04%)
Sufficient balance: You have enough collateral on both exchanges
Position limits not reached: You haven't hit your maximum concurrent positions
Positive expected value: After accounting for fees and slippage, the trade is profitable

When Does the Bot Close Positions?

Positions are closed automatically when:

Target profit reached: Default 5% profit on position
Spread reverses: The funding rate advantage disappears or reverses
Trailing stop triggered: Profit retraces too much from peak (protects gains)
Manual shutdown: You stop the bot gracefully

Risk Management Features

Delta-Neutral Protection

Equal position sizes on both exchanges
No exposure to price movements
Profit comes only from funding rate spread

Slippage Protection

Uses limit orders with timeout
Falls back to market orders if needed
Configurable maximum slippage tolerance

Trailing Stop Loss

Activates after reaching profit threshold (default: 3%)
Locks in minimum profit (default: 2%)
Closes if profit retraces too much from peak

Atomic Execution

Opens both legs simultaneously
If one leg fails, the other is rolled back
Prevents unhedged (risky) positions

Position Size Limits

Maximum position size cap (USD)
Maximum percentage of balance per trade
Maximum concurrent positions

Configuring the Strategy

Key parameters in config.toml:

[trading]
# Minimum spread to open a position (0.04% = 4 basis points)
min_funding_spread = 0.0004

# Position sizing
position_size_percent = 0.30    # Use 30% of available balance
max_position_usd = 10000        # Cap at $10,000 per position
max_concurrent_positions = 5    # Maximum open positions

# Profit targets
target_profit_percent = 0.05    # Close at 5% profit

[risk]
# Trailing stop configuration
trailing_stops_enabled = true
trailing_stop_activation = 0.03  # Activate at 3% profit
trailing_stop_distance = 0.40    # Allow 40% retracement
trailing_stop_min_lock = 0.02    # Lock in minimum 2%

# Slippage protection
max_slippage_bps = 50           # Maximum 0.5% slippage

Understanding Expected Value (EV)

Before opening a position, the bot calculates the expected value - whether the trade will be profitable after costs:

Expected Value = (Funding Spread × Time Weight) - (Entry Fees + Exit Fees + Slippage)

Funding Spread: The rate difference between exchanges
Time Weight: Reduced value if close to next funding payment
Fees: Trading fees on both exchanges (entry and exit)
Slippage: Expected price impact

The bot only opens positions with positive expected value.

Supported Exchanges

Exchange	Type	Funding Interval	Max Leverage
Binance Futures	Perpetual	8 hours	125x
Bybit Perpetuals	Perpetual	8 hours	100x
HyperLiquid	Perpetual	8 hours	50x

Frequently Asked Questions

What happens if one exchange goes down?

The bot monitors connectivity and will:

Alert you to connection issues
Prevent new positions from opening
Existing positions remain open (both legs on different exchanges)

Can I lose money?

While the strategy is market-neutral, risks include:

Fee costs: If spread narrows before covering fees
Liquidation: If leverage is too high and price moves sharply
Exchange risk: Counterparty risk on centralized exchanges
Execution risk: Slippage during entry/exit

How much capital do I need?

Minimum recommended: $1,000 per exchange ($2,000 total)

With leverage, you can control larger positions:

$1,000 at 10x = $10,000 position size
Required margin = Position / Leverage + Safety Buffer

What's a good minimum spread to trade?

Conservative: 0.05% (5 basis points) - fewer trades, higher quality
Moderate: 0.04% (4 basis points) - balanced approach
Aggressive: 0.03% (3 basis points) - more trades, lower margins

How often are there opportunities?

This varies by market conditions:

High volatility: More funding rate divergence = more opportunities
Low volatility: Rates converge = fewer opportunities
Typical: 2-5 high-quality opportunities per day

Strategy Calculation Formulas

Mathematical reference for funding rate arbitrage calculations, position sizing, and profitability analysis

Funding Rate Arbitrage

Core Concept

Funding rate arbitrage exploits differences in funding rates across exchanges while maintaining delta-neutral positions.

Funding Rate Spread

$spread = ∣ rate_{short} - rate_{long} ∣$

Where:

$\text{rate}_{\text{short}}$ = funding rate on high-rate exchange (short position)
$\text{rate}_{\text{long}}$ = funding rate on low-rate exchange (long position)

Example:

Binance BTCUSDT: +0.0001 (1 basis point)
HyperLiquid BTCUSDT: +0.0006 (6 basis points)

spread = |0.0006 - 0.0001| = 0.0005 = 5 basis points = 0.05%

Annual Percentage Yield (APY)

$APY = spread \times funding_intervals_per_year$

$APY = spread \times 3 \times 365 = spread \times 1, 095$

Where:

Funding occurs 3 times per day (every 8 hours)
Spread is expressed as a decimal (e.g., 0.0005)

Example:

spread = 0.0005 (5 basis points)
APY = 0.0005 × 1,095 = 0.5475 = 54.75%

Expected Value (EV) Calculation

The bot uses risk-adjusted expected value to filter opportunities and ensure trades are profitable after all costs.

Time Weighting

$time_weight = min (\frac{minutes_to_funding}{480}, 1)$

Where:

480 minutes = 8 hours (full funding period)
Weight approaches 1 as funding time approaches
Weight approaches 0 immediately after funding

Gross Expected Value

$gross_ev = spread \times time_weight$

Transaction Costs

$costs = fee_{long_entry} + fee_{long_exit} + fee_{short_entry} + fee_{short_exit} + slippage$

For limit orders (maker fees): $costs = 2 \times (maker_{long} + maker_{short})$

For market orders (taker fees + slippage): $costs = 2 \times (taker_{long} + taker_{short}) + slippage_estimate$

Staleness Haircut

$staleness = {haircut 0 if data age > 4 minutes otherwise$

Default: haircut = 0.0003 (3 basis points)

Adjusted Expected Value

$adjusted_ev = gross_ev - costs - staleness$

Qualification Criteria

An opportunity qualifies if:

$adjusted_ev \geq min_ev_bps AND spread \geq min_spread_threshold$

Default thresholds:

$\text{min_ev_bps} = 0.0005$ (5 basis points)
$\text{min_spread_threshold} = 0.0004$ (4 basis points)

Complete EV Example

Scenario:

Binance BTCUSDT: +0.0001 (long position)
HyperLiquid BTCUSDT: +0.0010 (short position)
Time to funding: 480 minutes (full period)
Order type: Limit (maker fees)

Step 1: Calculate spread $spread = 0.0010 - 0.0001 = 0.0009 = 90 bps$

Step 2: Time weighting $time_weight = \frac{480}{480} = 1.0$

Step 3: Gross EV $gross_ev = 0.0009 \times 1.0 = 0.0009 = 90 bps$

Step 4: Calculate costs

Binance fees: 0.0002 maker (entry) + 0.0002 maker (exit) = 0.0004
HyperLiquid fees: 0.0000 maker (entry) + 0.0000 maker (exit) = 0.0000
Total costs: 0.0004 = 4 bps

Step 5: Staleness haircut

Data age: < 4 minutes
Staleness: 0

Step 6: Adjusted EV $adjusted_ev = 90 - 4 - 0 = 86 bps$

Step 7: Qualification check

adjusted_ev (86 bps) >= min_ev (5 bps) ✅
spread (90 bps) >= min_spread (4 bps) ✅

Result: QUALIFIES

Expected profit per $10,000 position: $profit = $10, 000 \times 0.0086 = $86 per funding period$

$daily_profit = $86 \times 3 = $258$

$monthly_profit = $258 \times 30 = $7, 740$

Position Sizing

Conservative Sizing Formula

The bot reserves margin for both opening AND closing positions:

$position_size = (\frac{balance \times percentage}{2}) \times leverage$

Where:

$\text{balance}$ = available balance on exchange (USDT)
$\text{percentage}$ = position size percentage (default: 0.30 = 30%)
$\text{leverage}$ = configured leverage (default: 10x)
Division by 2 reserves margin for closing

Example:

balance = $1,000 USDT
percentage = 0.30 (30%)
leverage = 10x

position_size = (1,000 × 0.30 / 2) × 10
              = (300 / 2) × 10
              = 150 × 10
              = $1,500 USD notional

Quantity Calculation

$quantity = \frac{position_size}{price}$

Must be rounded to exchange precision:

$quantity_rounded = ⌊ \frac{quantity}{1 0 ^{- precision}} ⌋ \times 1 0^{- precision}$

Example:

position_size = $1,500
BTC price = $50,000
precision = 3 decimals (0.001 BTC minimum)

quantity = 1,500 / 50,000 = 0.030 BTC
quantity_rounded = floor(0.030 / 0.001) × 0.001 = 0.030 BTC ✅

Kelly Criterion Sizing

The bot supports two position sizing modes:

Mode 1: Kelly Criterion (Dynamic)

Enabled when use_kelly_criterion = true (default). Sizes positions based on expected value and variance:

$kelly_fraction = \frac{EV}{variance}$

$raw_kelly_size = kelly_fraction \times reference_capital \times leverage$

With safety caps:

$final_size = min (raw_kelly_size \times kelly_cap, max_notional, balance \times max_utilization)$

Where:

$\text{kelly_cap} = 0.25$ (quarter-Kelly, default)
$\text{max_notional} = $10,000$ (per-symbol cap, default)
$\text{max_utilization} = 0.50$ (50% of balance, default)

Use when: You want dynamic sizing that increases with edge quality and decreases with variance.

Mode 2: Fixed Percentage (Simple)

Enabled when use_kelly_criterion = false. Uses fixed percentage of available capital:

$reference_capital = min (long_balance, short_balance)$

$fixed_size = reference_capital \times max_utilization \times leverage$

With safety cap:

$final_size = min (fixed_size, max_notional)$

Use when: You want predictable, consistent position sizing regardless of opportunity quality.

Configuration:

[trading]
use_kelly_criterion = true    # Toggle between Kelly and fixed modes
kelly_fraction = 0.25          # Only used in Kelly mode
max_exchange_utilization = 0.5 # Used in both modes
max_notional_per_symbol = 10000 # Hard cap for both modes

Fee Calculations

Exchange Fee Structure

Exchange	Maker Fee	Taker Fee	Funding Interval
Binance	0.02%	0.04%	8 hours
Bybit	0.01%	0.06%	8 hours
HyperLiquid	0.00%	0.035%	1 hour*

*HyperLiquid has 24 funding payments per day (hourly)

Order Fee Calculation

$fee = notional_value \times fee_rate$

$notional_value = quantity \times price$

Example:

quantity = 0.1 BTC
price = $50,000
fee_rate = 0.0004 (0.04% taker)

notional = 0.1 × 50,000 = $5,000
fee = 5,000 × 0.0004 = $2.00

Round-Trip Cost

For entering and exiting a position:

$round_trip_cost = 2 \times (fee_{long} + fee_{short}) + slippage$

Example (limit orders):

Binance maker: 0.02%
HyperLiquid maker: 0.00%

Entry cost: (0.0002 + 0.0000) = 0.0002 = 0.02%
Exit cost: (0.0002 + 0.0000) = 0.0002 = 0.02%
Total: 0.0004 = 0.04% = 4 basis points

On $10,000 position: $10,000 × 0.0004 = $4.00

Breakeven Spread

Minimum spread needed to be profitable:

$breakeven_spread = \frac{round_trip_cost}{holding_periods}$

Example (3 funding periods = 24 hours):

round_trip_cost = 0.0004 (4 bps)
holding_periods = 3

breakeven = 0.0004 / 3 = 0.000133 = 1.33 bps per period

Profit & Loss

Funding Payment Calculation

$funding_payment = position_size \times funding_rate$

For long position:

Positive rate: pay funding
Negative rate: receive funding

For short position:

Positive rate: receive funding
Negative rate: pay funding

Arbitrage Profit Per Period

$profit = position_size \times (rate_{short} - rate_{long})$

Example:

position_size = $10,000
Binance rate (long): +0.0001
HyperLiquid rate (short): +0.0006

profit = 10,000 × (0.0006 - 0.0001)
       = 10,000 × 0.0005
       = $5.00 per funding period

Net Profit After Fees

$net_profit = gross_profit - entry_fees - exit_fees$

Example:

gross_profit = $5.00 per period × 3 periods = $15.00
entry_fees = $4.00
exit_fees = $4.00

net_profit = 15.00 - 4.00 - 4.00 = $7.00 per day
monthly = $7.00 × 30 = $210.00

Return on Capital

$ROC = \frac{net_profit}{margin_used}$

$margin_used = \frac{position_size}{leverage}$

Example:

position_size = $10,000
leverage = 10x
margin_used = 10,000 / 10 = $1,000

daily_profit = $7.00
daily_ROC = 7.00 / 1,000 = 0.007 = 0.7%
annual_ROC = 0.007 × 365 = 2.555 = 255.5%

Risk Metrics

Maximum Position Size

$max_size = min (config_max, \frac{balance \times utilization}{margin_rate})$

Where:

$\text{config_max} = $10,000$ (default per-symbol cap)
$\text{utilization} = 0.50$ (50% of balance usage)
$\text{margin_rate} = \frac{1}{\text{leverage}}$

Example:

balance = $5,000
utilization = 0.50
leverage = 10x
margin_rate = 0.1

balance_limit = (5,000 × 0.50) / 0.1 = 25,000
max_size = min(10,000, 25,000) = $10,000

Margin Requirement

$margin_required = \frac{position_size}{leverage} \times 1.2$

The 1.2 factor provides a 20% safety buffer.

Example:

position_size = $10,000
leverage = 10x

margin = (10,000 / 10) × 1.2 = 1,000 × 1.2 = $1,200

Liquidation Distance

$liq_distance = \frac{margin - maintenance_margin}{position_size}$

Example:

margin = $1,000
maintenance = $100 (10% of margin)
position_size = $10,000

liq_distance = (1,000 - 100) / 10,000 = 0.09 = 9%

Position liquidates at 9% adverse price move

Sharpe Ratio

$Sharpe = \frac{mean_return - risk_free_rate}{std_deviation}$

Calculated over historical trades for strategy performance evaluation.

Maximum Drawdown

$max_drawdown = max (\frac{peak_equity - current_equity}{peak_equity})$

Tracks the largest peak-to-trough decline in account equity.

Configuration Reference

Default Values

Parameter	Default	Description
`min_spread_threshold`	0.0004 (4 bps)	Minimum spread to qualify
`min_ev_bps`	0.0005 (5 bps)	Minimum expected value
`max_position_usd`	$10,000	Maximum position size
`position_size_percentage`	0.30 (30%)	Percentage of balance
`leverage`	10x	Default leverage
`kelly_cap`	0.25	Quarter-Kelly
`max_exchange_utilization`	0.50 (50%)	Max balance usage
`staleness_haircut_bps`	0.0003 (3 bps)	Penalty for stale data

Exchange Fee Rates

Exchange	Maker	Taker
Binance Futures (VIP 0)	0.02%	0.04%
Bybit (Standard)	0.01%	0.06%
HyperLiquid	0.00%	0.035%

Example Configuration

[trading]
# Spread thresholds
min_funding_spread = 0.0004        # 4 basis points
min_ev_bps = 0.0005                # 5 basis points

# Position sizing
position_size_percent = 0.30       # 30% of balance
max_position_usd = 10000           # $10,000 cap
leverage = 10                      # 10x leverage

# Kelly criterion settings
use_kelly_criterion = true
kelly_fraction = 0.25              # Quarter-Kelly
max_exchange_utilization = 0.50    # 50% max usage

[risk]
# Slippage
max_slippage_bps = 50              # 0.5% maximum
estimated_slippage = 0.001         # 0.1% estimate

Quick Reference Card

Profit Estimation

Spread (bps)	Position Size	Per Period	Daily	Monthly
4	$10,000	$4.00	$12.00	$360
5	$10,000	$5.00	$15.00	$450
10	$10,000	$10.00	$30.00	$900
20	$10,000	$20.00	$60.00	$1,800

APY by Spread

Spread (bps)	APY (Pre-Fee)	APY (Post-Fee, ~4bps cost)
4	43.8%	0%
5	54.75%	10.95%
10	109.5%	65.7%
20	219%	175.2%

Minimum Capital Requirements

Position Target	Leverage	Margin Needed	Recommended Buffer
$5,000	10x	$500	$750
$10,000	10x	$1,000	$1,500
$20,000	10x	$2,000	$3,000
$50,000	10x	$5,000	$7,500

Monitoring Guide

Monitor your Wealth trading bot's performance with built-in dashboards and AI-powered queries.

Overview

The bot provides multiple monitoring options:

🤖 AI Queries - Ask questions in natural language via Grafana MCP
📊 Dashboards - Visual monitoring in Grafana
📈 Metrics - Real-time performance data via OpenTelemetry
🔔 Alerts - Automated notifications for important events
❤️ Health Endpoint - Quick status checks

Quick Health Check

CLI Commands

# Overall health status
curl http://localhost:9090/health | jq

# Check positions
wealth positions

# Check balances
wealth balance

# View configuration
wealth config

Health Endpoint Response

{
  "status": "healthy",
  "timestamp": "2024-01-15T10:30:00Z",
  "uptime_seconds": 3600,
  "components": {
    "websockets": {
      "binance": { "connected": true, "last_update": "2024-01-15T10:29:58Z", "error_count": 0 },
      "bybit": { "connected": true, "last_update": "2024-01-15T10:29:59Z", "error_count": 0 }
    },
    "exchanges": {
      "binance_api": { "reachable": true, "consecutive_failures": 0 },
      "bybit_api": { "reachable": true, "consecutive_failures": 0 }
    },
    "strategy": {
      "active_positions": 2,
      "fresh_funding_rates": 15
    }
  }
}

AI-Powered Monitoring

Ask your AI assistant (GitHub Copilot, Claude, etc.) natural language questions:

Examples:

"What's my current P&L?"
"Show funding rates for BTCUSDT"
"Are there any firing alerts?"
"Which exchange has the best execution performance?"
"What's the average position holding time?"

No need to write queries or navigate dashboards - AI queries metrics automatically.

Setup (5 minutes)

Start services:
```
docker compose up -d
```
Generate Grafana token at http://localhost:3000
Add to .env:
```
GRAFANA_SERVICE_ACCOUNT_TOKEN=glsa_...
```
Restart MCP:
```
docker compose restart grafana-mcp
```

Full guide: Grafana MCP Setup Guide

Grafana Dashboards

Accessing Dashboards

Open Grafana: http://localhost:3000
Default credentials: admin / admin
Navigate to Dashboards → Wealth Trading Bot

Key Dashboard Panels

Panel	Description
P&L Overview	Total profit/loss across all exchanges
Active Positions	Current open positions with entry prices
Funding Rate Spread	Spread between exchanges
Execution Success Rate	Order fill rate and latency
Balance by Exchange	USDT balance per exchange
WebSocket Status	Connection health for each exchange

Setting Up Grafana Cloud

For cloud monitoring (recommended for production):

Create free Grafana Cloud account
Configure OTLP export
Import dashboard templates

See Grafana Cloud Setup for full instructions.

Key Metrics

Trading Performance

Metric	Description	Good Value
Win Rate	% of profitable trades	> 60%
Average P&L	Mean profit per trade	> 0
Fill Rate	% of orders filled	> 95%
Execution Latency	Order placement time	< 100ms

System Health

Metric	Description	Good Value
WebSocket Uptime	Connection stability	> 99%
API Success Rate	Exchange API health	> 99%
Memory Usage	RAM consumption	< 500MB
Error Rate	Errors per minute	< 1

View in Terminal

# Metrics info endpoint
curl http://localhost:9090/metrics | jq

Alerts

Built-in Alert Types

Alert	Severity	Trigger
Low Balance	Critical	Account balance < $1,000
High Error Rate	Warning	API errors > 10%
Position Stuck	Warning	Position open > 24 hours
High Slippage	Warning	Slippage > 50 bps
Connection Failure	Critical	WebSocket disconnected

Configuring Alerts in Grafana

Go to Alerting → Alert rules
Create new alert rule
Set condition (e.g., wealth_balance_total < 1000)
Add notification channel (email, Slack, Discord)

Example Alert Rule

# Low Balance Alert
condition: wealth_balance_total{exchange="binance"} < 1000
for: 5m
severity: critical

Log Viewing

Real-time Logs

# Follow bot logs
wealth run 2>&1 | tee -a bot.log

# Or with Docker
docker compose logs -f wealth

Log Levels

Level	Example
INFO	`Bot running
WARN	`Rate limit approaching`
ERROR	`Order placement failed`

Searching Logs in Grafana (Loki)

# All errors
{service="wealth-bot"} |= "ERROR"

# Order execution logs
{service="wealth-bot"} |= "arbitrage"

# WebSocket issues
{service="wealth-bot"} |= "WebSocket"

Docker Compose Monitoring Stack

The included compose.yml provides:

Grafana - Dashboards and visualization
OpenTelemetry Collector - Metrics aggregation
Prometheus - Metrics storage
Loki - Log aggregation
Tempo - Distributed tracing

Starting the Stack

# Start all services
docker compose up -d

# Check service status
docker compose ps

# View logs
docker compose logs -f

Accessing Services

Service	URL	Credentials
Grafana	http://localhost:3000	admin / admin
Prometheus	http://localhost:9090	-
Bot Health	http://localhost:9090/health	-

Recommended Monitoring Setup

Development / Paper Trading

Local Grafana (Docker Compose)
Terminal logs
Health endpoint checks

Production / Live Trading

Grafana Cloud (free tier available)
Alert notifications (email/Slack)
AI-powered monitoring (MCP)
24/7 uptime monitoring

Quick Reference

# Health check
curl http://localhost:9090/health | jq

# View positions
wealth positions

# View balances  
wealth balance

# Start monitoring stack
docker compose up -d

# View Grafana
open http://localhost:3000

# Follow logs
docker compose logs -f wealth

Grafana Cloud Setup Guide

Quick 15-Minute Setup for Fully Managed Observability

Overview

This guide shows you how to migrate from self-hosted monitoring (Docker Compose) to Grafana Cloud - a fully managed observability platform with free tier (14-day trial, then free tier available).

Benefits:

✅ Zero infrastructure - No Prometheus, Grafana, Loki, or Tempo containers to manage
✅ Automatic scaling - Handles any volume of metrics/logs/traces
✅ High availability - 99.9% uptime SLA
✅ Global CDN - Fast dashboard loading worldwide
✅ Built-in alerting - Email, Slack, PagerDuty integrations
✅ Long-term retention - Metrics for 13 months (free tier)

Quick Start (15 Minutes)

TL;DR - Fast track to Grafana Cloud:

Create account → https://grafana.com/auth/sign-up/create-user
Get OTLP endpoint → Your stack → Send Data → OpenTelemetry → Copy endpoint URL

Configure collector (credentials already in otel-collector-grafana-cloud.yaml):

# Verify endpoint in otel-collector-grafana-cloud.yaml matches your region
# File already configured with prod-eu-central-0 credentials

Deploy:

docker compose up -d

# Update .env to point bot to local collector
OTLP_ENDPOINT=http://localhost:4317

# Run bot
export CREDENTIALS_PASSPHRASE="your_passphrase"
wealth run

View → https://YOUR-STACK.grafana.net → Explore → Query: wealth_funding_rate

Important: Do NOT send telemetry directly from the bot to Grafana Cloud's managed OTLP gateway. Always use a local OTLP Collector as a proxy (see Architecture and Troubleshooting for why).

For detailed instructions, continue reading below.

Prerequisites

✅ Wealth trading bot installed (see Getting Started)
✅ Docker and Docker Compose installed
✅ Credit card (for Grafana Cloud trial - free tier available after)

Step 1: Create Grafana Cloud Account

Go to https://grafana.com/auth/sign-up/create-user
Click "Start for free"
Fill in your details:
- Email address
- Company name (can use "Personal" or your name)
- Password
Verify your email

1.2 Create a Stack

After email verification:

You'll be prompted to create a stack (your isolated Grafana Cloud instance)
Choose:
- Stack name: wealth-trading-bot (or any name you prefer)
- Region: Choose closest to you (e.g., us-east-1, eu-west-1)
- Plan: Start with Free Trial (14 days, then free tier)
Click "Launch Stack"

1.3 Access Your Stack

Your stack will be created with a URL like:

https://YOUR-STACK-NAME.grafana.net

Save this URL - you'll need it later.

Step 2: Get Your Cloud Credentials

Note: The repository includes pre-configured credentials for prod-eu-central-0 region in otel-collector-grafana-cloud.yaml. If you're using a different region or want to use your own credentials, follow these steps.

2.1 Find Your OTLP Endpoint

In Grafana Cloud UI, go to your stack homepage
Click "Send Data" or "Connections"
Search for "OpenTelemetry" or "OTLP"
You'll see an endpoint like:
- https://otlp-gateway-YOUR-REGION.grafana.net/otlp

Example regions:

prod-eu-central-0 (Europe - Frankfurt)
prod-us-east-0 (US East - Virginia)
prod-us-central-0 (US Central)

2.2 Create an Access Token

Go to "Administration" → "Access Policies"
Click "Create access policy"
Set:
- Name: wealth-bot-telemetry
- Realms: Your stack
- Scopes:
  - metrics:write
  - logs:write
  - traces:write
Click "Create"
Click "Add token" → Copy the token (starts with glc_)

2.3 Get Your Instance ID

Your instance ID (username) is visible in the OTLP configuration:

Usually a 6-7 digit number (e.g., 1446931)
Found in: "Send Data" → "OpenTelemetry" → Look for "Instance ID" or "Username"

2.4 Update Collector Configuration

Edit otel-collector-grafana-cloud.yaml:

exporters:
  otlphttp/grafana_cloud:
    endpoint: "https://otlp-gateway-YOUR-REGION.grafana.net/otlp"  # Update region
    auth:
      authenticator: basicauth/grafana_cloud

extensions:
  basicauth/grafana_cloud:
    client_auth:
      username: "YOUR_INSTANCE_ID"      # Update with your instance ID
      password: "YOUR_API_TOKEN"         # Update with your token (glc_...)

Security Note: These credentials are in the Docker Compose config file, not .env, as they're only used by the OTLP Collector container, not the bot itself.

Step 3: Configure Bot to Use Local Collector

The bot should send telemetry to the local OTLP Collector, not directly to Grafana Cloud.

3.1 Update Environment Variables

In your .env file:

# Point bot to LOCAL collector (NOT Grafana Cloud directly)
OTLP_ENDPOINT=http://localhost:4317

# Optional: Set service metadata
ENVIRONMENT=production
SERVICE_NAME=wealth-bot

Critical: Do NOT set OTLP_ENDPOINT to Grafana Cloud's managed gateway URL. Direct connections from the Rust SDK have compatibility issues. See Troubleshooting for details.

Step 4: Start the OTLP Collector

3.1 Collector Configuration Overview

The repository includes a pre-configured OTLP Collector at otel-collector-grafana-cloud.yaml. This collector:

Receives OTLP data from the bot (gRPC on port 4317, HTTP on 4318)
Processes telemetry (batching, resource detection, filtering)
Exports to Grafana Cloud via OTLP/HTTP endpoint
Generates additional metrics from traces (span metrics, service graphs)

Key Features:

Uses grafanacloud connector for automatic span metrics generation
Drops unnecessary resource attributes to reduce cardinality
Adds deployment environment and service version to metrics
Health check endpoint on port 13133

Configuration is already complete - you only need to update credentials if using a different region (see Step 2).

Step 4: Start the OTLP Collector

4.1 Start the Collector

The Docker Compose configuration is already in your repository at compose.yml (Grafana Cloud is now the default).

# Start the OTLP Collector
docker compose up -d

# Verify it's healthy
curl http://localhost:13133
# Expected: {"status":"Server available","upSince":"..."}

# Check logs
docker compose logs otel-collector
# Look for: "Everything is ready. Begin running and processing data."

What this deploys:

Single container: OpenTelemetry Collector (otel/opentelemetry-collector-contrib:latest)
Ports exposed:
- 4317 - OTLP gRPC (bot connects here)
- 4318 - OTLP HTTP (alternative)
- 13133 - Health check endpoint
Credentials: Read from otel-collector-grafana-cloud.yaml (hardcoded, not from .env)

Step 5: Configure and Run the Bot

5.1 Update Bot Configuration

In your .env file:

# Point to LOCAL collector (NOT Grafana Cloud)
OTLP_ENDPOINT=http://localhost:4317

# Optional: Service metadata
ENVIRONMENT=production
SERVICE_NAME=wealth-bot

Critical: The bot should connect to localhost:4317 (local collector), NOT to Grafana Cloud's managed gateway. The collector handles the connection to Grafana Cloud.

5.2 Verify .env is Gitignored

# Check if .env is gitignored
grep "\.env" .gitignore

# If not, add it
echo ".env" >> .gitignore

Step 6: Test the Connection

6.1 Verify Collector is Running

# Check collector health
curl http://localhost:13133
# Expected: {"status":"Server available","upSince":"..."}

# Check collector logs
docker compose logs otel-collector | tail -20
# Look for: "Everything is ready. Begin running and processing data."

# Check for export errors
docker compose logs otel-collector | grep -i error
# Should be empty or only show benign warnings

6.2 Run the Bot

# Set required credentials
export CREDENTIALS_PASSPHRASE="your_passphrase"

# Ensure OTLP_ENDPOINT points to local collector
echo $OTLP_ENDPOINT
# Should output: http://localhost:4317

# Run the bot
wealth run

6.3 Verify Bot is Exporting

# Check bot health endpoint
curl http://localhost:9090/health | jq
# Look for: "otlp_status": "connected" or similar

# Watch bot logs for OTLP connection
# Should see: "OpenTelemetry OTLP export enabled to: http://localhost:4317"
# Should NOT see: "BatchLogProcessor.ExportError" or "Timeout expired"

6.4 Verify Data in Grafana Cloud

Open your Grafana Cloud instance: https://YOUR-STACK.grafana.net
Go to "Explore"
Select data source: Prometheus (or "grafanacloud-YOUR-STACK-prom")
Run query:
```
{__name__=~"wealth.*"}
```
You should see metrics appearing within 10-30 seconds

If no data appears, see Troubleshooting or OTLP Export Issues.

Step 7: Import Dashboards

7.1 Import Pre-built Dashboard

Your repository includes a pre-built dashboard at grafana/grafana-dashboard.json:

In Grafana Cloud UI, go to "Dashboards" → "Import"
Click "Upload JSON file"
Select grafana/grafana-dashboard.json from your repository
Configure:
- Name: Keep default or rename (e.g., "Wealth Trading Bot")
- Folder: Select or create a folder
- Prometheus: Select your Grafana Cloud Prometheus data source
Click "Import"

7.2 Verify Dashboard Data

After import:

Dashboard should populate with data within 30 seconds
Check panels for:
- Funding Rates by symbol and exchange
- Account Balances by exchange
- Trade Execution Metrics (success rate, errors)
- WebSocket Connection Status

7.3 Dashboard Variables (if needed)

If variables need adjustment:

Open imported dashboard
Click ⚙️ (Settings) → "Variables"
Update $pair variable if needed:
- Query: label_values(wealth_funding_rate, symbol)
- Data source: grafanacloud-YOUR-STACK-prom
Save dashboard

Migration Checklist

Use this checklist to track your migration progress:

Pre-Migration

Current self-hosted setup working (docker compose ps)
Bot exporting metrics (curl http://localhost:9090/health)
Optional: Export existing dashboards for backup

Grafana Cloud Setup

Create Grafana Cloud account
Create stack (save URL)
Get Prometheus endpoint + credentials
Get Loki endpoint (can reuse same credentials)
Get Tempo endpoint (can reuse same credentials)

Configuration

Copy .env.example to .env
Uncomment and fill GRAFANA_CLOUD_* variables
Set GRAFANA_CLOUD_USERNAME and GRAFANA_CLOUD_API_TOKEN
Verify .env is in .gitignore

Deployment

Stop old stack: docker compose down
Start Cloud stack: docker compose up -d
Verify collector health: curl http://localhost:13133
Check collector logs for errors
Run bot: wealth run
Verify bot health: curl http://localhost:9090/health

Verification

Open Grafana Cloud: https://YOUR-STACK.grafana.net
Query wealth_funding_rate in Explore
Check metrics appearing (10-30 seconds)
Verify logs in Loki
Check traces in Tempo (if enabled)

Dashboard & Alerts

Import grafana/grafana-dashboard.json
Update dashboard data source to Grafana Cloud Prometheus
Create alert rules (low balance, WebSocket down, etc.)
Set up notification channels (email, Slack)
Test notifications

Post-Migration

Monitor for 24 hours
Check usage: Administration → Usage insights
Optional: Clean up old Docker images
Update team documentation

Time estimate: ~45 minutes total (~15 minutes active work)

Step 8: Set Up Alerts

8.1 Create Alert Rules

Grafana Cloud has built-in alerting:

In dashboard, click any panel → "Edit"
Click "Alert" tab → "Create alert rule from this panel"
Configure:
- Name: Low Balance Alert
- Query: sum(wealth_account_balance{type="total"}) < 1000
- Threshold: Below 1000
- Evaluation interval: 1m
Click "Save"

8.2 Notification Channels

Set up notifications:

Go to "Alerting" → "Contact points"
Click "Add contact point"
Choose:
- Email (free)
- Slack (requires webhook)
- PagerDuty (requires integration key)
Test notification → Save

8.3 Recommended Alerts

Alert	Query	Threshold	Severity
Low Balance	`sum(wealth_account_balance{type="total"}) < 1000`	< $1,000	Critical
High Error Rate	`rate(wealth_order_errors_total[5m]) > 0.1`	> 10%	Warning
WebSocket Down	`wealth_websocket_status == 0`	== 0	Critical
Low Win Rate	`wealth_trades_win_rate < 40`	< 40%	Warning

Architecture Comparison

Before (Self-Hosted)

┌─────────────────────────────────────────────────────┐
│ Docker Compose (5 containers)                       │
├─────────────────────────────────────────────────────┤
│ Wealth Bot → OTLP Collector                         │
│              ↓                                       │
│              ├→ Prometheus (metrics)                │
│              ├→ Tempo (traces)                      │
│              └→ Loki (logs)                         │
│                 ↓                                    │
│              Grafana (dashboards)                   │
└─────────────────────────────────────────────────────┘

Resource Usage:

CPU: ~500-800 MB RAM (all containers)
Disk: ~1-5 GB (retention depends on volume)
Maintenance: Manual updates, backups, scaling

After (Grafana Cloud)

┌─────────────────────────────────────────────────────────────┐
│ Wealth Bot (Rust)                                           │
│    ↓ OTLP gRPC (localhost:4317)                            │
│ OTLP Collector (Docker)                                     │
│    ↓ OTLP/HTTP + Basic Auth                                │
│ Grafana Cloud                                               │
│   ├→ Prometheus (metrics)                                  │
│   ├→ Loki (logs)                                           │
│   └→ Tempo (traces)                                        │
└─────────────────────────────────────────────────────────────┘

Architecture Benefits:

✅ Local Collector as Proxy: Handles batching, retries, and Grafana Cloud-specific requirements
✅ No Direct Bot → Cloud Connection: Avoids SDK compatibility issues (see Why Use a Collector)
✅ Single Point of Auth: Credentials managed in one place (collector config)
✅ Debugging Layer: Collector logs show exactly what's sent to Grafana Cloud

Resource Usage:

CPU: ~100-200 MB RAM (OTLP Collector only)
Disk: Minimal (no local storage)
Maintenance: Zero (fully managed)
Network: ~5-10 MB/hour (compressed telemetry)

Why This Architecture?

According to Grafana's best practices:

"We advise that you send directly to an OTLP endpoint in testing or small scale development scenarios only. Use a collector for production."

Direct SDK → Grafana Cloud connections are unreliable due to:

Rate limiting on managed endpoints
SDK-specific compatibility issues
Missing batching/compression optimizations
No retry buffer for network failures

See OTLP Troubleshooting for detailed explanation.

Cost Optimization

Free Tier Limits (Grafana Cloud)

Service	Free Tier Quota	Overage Cost
Metrics	10k series, 13 months retention	$0.30/1k series/month
Logs	50 GB ingestion, 2 weeks retention	$0.50/GB
Traces	50 GB ingestion, 2 weeks retention	$0.50/GB
Dashboards	10 users, unlimited dashboards	Free

Current Bot Usage (Estimated)

Based on your metrics:

Metrics: ~500 series (well within free tier)
Logs: ~1-2 GB/month (within free tier)
Traces: ~500 MB/month (within free tier)

Expected Cost: $0/month (free tier) 🎉

Tips to Stay Within Free Tier

Reduce log volume:

# In .env, set log level to INFO (not DEBUG)
RUST_LOG=info

Sample traces (if you enable tracing heavily):

# In otel-collector-grafana-cloud.yaml
processors:
  probabilistic_sampler:
    sampling_percentage: 10  # Only send 10% of traces

Monitor usage:
- Go to "Administration" → "Usage insights"
- Check monthly consumption

Troubleshooting

Issue: "Connection refused" to Grafana Cloud

Symptom: OTLP Collector logs show connection errors

Solutions:

Check endpoints in .env:

grep GRAFANA_CLOUD .env
# Verify URLs match your region (e.g., us-east-1, eu-west-1)

Test connectivity:

# Test Prometheus endpoint
curl -u "$GRAFANA_CLOUD_USERNAME:$GRAFANA_CLOUD_API_TOKEN" \
  "$GRAFANA_CLOUD_PROMETHEUS_ENDPOINT"

# Expected: 401 (auth works), 404 (endpoint works), or 200 (success)
# Not expected: Connection refused, timeout

Check firewall:

# Ensure outbound HTTPS (443) is allowed
telnet prometheus-us-east-1.grafana.net 443

Issue: "Unauthorized" (401) from Grafana Cloud

Symptom: OTLP Collector logs show 401 errors

Solutions:

Verify credentials:

echo $GRAFANA_CLOUD_API_TOKEN
# Should start with "glc_"

Regenerate token:
- Go to Grafana Cloud → "Administration" → "Access Policies"
- Delete old token
- Create new token with metrics:write, logs:write, traces:write scopes
- Update .env
Restart collector:
```
docker compose restart otel-collector
```

Issue: No metrics in Grafana Cloud

Symptom: Dashboard shows "No data"

Solutions:

Check bot is exporting:

curl http://localhost:9090/metrics
# Should show: "backend": "OpenTelemetry OTLP"

Check collector is receiving:

docker compose logs otel-collector | grep "Metric"
# Should show: "Metrics" ... "sent": true

Check data source in Grafana:
- Go to "Connections" → "Data sources"
- Click on Prometheus data source
- Scroll down → "Save & test"
- Should show: "Data source is working"

Query directly:

# Query Grafana Cloud API
curl -u "$GRAFANA_CLOUD_USERNAME:$GRAFANA_CLOUD_API_TOKEN" \
  "https://prometheus-us-east-1.grafana.net/api/v1/query?query=wealth_funding_rate"

Issue: High data usage

Symptom: Approaching free tier limits

Solutions:

Check current usage:
- Grafana Cloud → "Administration" → "Usage insights"

Reduce metric cardinality:

# In otel-collector-grafana-cloud.yaml
processors:
  filter:
    metrics:
      exclude:
        match_type: strict
        metric_names:
          - wealth_websocket_message_latency_milliseconds

Aggregate metrics:

processors:
  metricstransform:
    transforms:
      - include: wealth_.*
        match_type: regexp
        action: aggregate_labels
        aggregation_type: sum
        label_set: [exchange]

Reduce log verbosity:

# In .env
RUST_LOG=warn  # Only warnings and errors

Migration Checklist

Before switching to Grafana Cloud:

Create Grafana Cloud account
Get Prometheus, Loki, and Tempo credentials
Create otel-collector-grafana-cloud.yaml
Grafana Cloud is now default (compose.yml)
Self-hosted stack available at compose.stack.yml
Update .env with Grafana Cloud credentials
Test connection: docker compose up -d
Verify metrics in Grafana Cloud
Import dashboards
Set up alerts
Configure notification channels
Update documentation (if team project)
Stop old local stack (if running): docker compose -f compose.stack.yml down

Next Steps

After successful migration:

Explore Grafana Cloud features:
- Pre-built dashboards
- SLO (Service Level Objectives) tracking
- Incident management
- OnCall rotation
Enable advanced features:
- Grafana Machine Learning - Anomaly detection
- Grafana Asserts - Automated root cause analysis
- Grafana Faro - Frontend observability (if you add a web UI)
Integrate with CI/CD:
- Use Grafana Cloud API for automated dashboard updates
- Add monitoring checks to GitHub Actions
Join Grafana Community:
- Slack: https://slack.grafana.com
- Forum: https://community.grafana.com

Monitoring Guide - All available metrics
Grafana MCP Setup - AI-powered monitoring (works with Cloud too!)
Deployment Guide - Production deployment
Configuration Guide - Bot configuration

Support

If you encounter issues:

Check collector logs: docker compose logs otel-collector
Verify credentials: grep GRAFANA_CLOUD .env
Test Grafana Cloud API: See troubleshooting section above
Consult Grafana Cloud docs: https://grafana.com/docs/grafana-cloud/

Grafana Cloud Support:

Free tier: Community support (Slack, forum)
Paid plans: Email and chat support

Still stuck? Open an issue in the project repo with:

OTLP Collector logs
Bot health check output: curl http://localhost:9090/health
Grafana Cloud stack URL (redact credentials)

Log Collection Guide

This guide covers log configuration, formats, and querying. For the complete observability setup including OpenTelemetry, metrics, and Grafana dashboards, see Monitoring Guide.

Quick Start

Enable OTLP Export (Recommended)

# Point to OpenTelemetry Collector (local or Grafana Cloud)
export OTLP_ENDPOINT=http://localhost:4317

wealth run

Logs, metrics, and traces are exported via OTLP gRPC. For complete OpenTelemetry setup, see Monitoring Guide - OpenTelemetry Metrics.

Local-Only Mode

If OTLP_ENDPOINT is not set, logs output to stdout/stderr:

# Optional: Enable file logging
export WEALTH__OBSERVABILITY__LOG_FILE=/tmp/wealth.log

wealth run

Log Query Examples

Basic Queries

# All logs from wealth bot
{job="wealth-bot"}

# Filter by log level
{job="wealth-bot"} |= "ERROR"
{job="wealth-bot"} |= "WARN"
{job="wealth-bot"} |= "INFO"

# Filter by level label (if parsed)
{job="wealth-bot", level="ERROR"}
{job="wealth-bot", level="INFO"}

# Filter by module
{job="wealth-bot"} |= "wealth::strategy"
{job="wealth-bot"} |= "wealth::execution"
{job="wealth-bot"} |= "wealth::market_data"

# Search for specific text
{job="wealth-bot"} |= "WebSocket"
{job="wealth-bot"} |= "arbitrage"
{job="wealth-bot"} |= "position"

Advanced Queries

# Errors in the last hour
{job="wealth-bot", level="ERROR"} [1h]

# Rate of errors per minute
rate({job="wealth-bot", level="ERROR"}[5m])

# Count logs by level
sum by (level) (count_over_time({job="wealth-bot"}[1h]))

# Logs containing correlation_id
{job="wealth-bot"} |~ "correlation_id=\\w+"

# WebSocket connection issues
{job="wealth-bot"} |~ "WebSocket.*error|WebSocket.*failed|WebSocket.*timeout"

# Order execution logs
{job="wealth-bot"} |= "Executing arbitrage" or |= "Order placed"

# Strategy-related logs
{job="wealth-bot", module=~"wealth::strategy.*"}

Event-Based Queries (Recommended)

All key log messages include a structured event field for precise filtering:

# Parse JSON and filter by event type
{service="wealth-bot"} | json | event="opportunity_detected"
{service="wealth-bot"} | json | event="arbitrage_executed"
{service="wealth-bot"} | json | event="position_close_succeeded"

# Trade skipped events (all use _skipped suffix)
{service="wealth-bot"} | json | event="quantity_validation_failed_skipped"
{service="wealth-bot"} | json | event="precision_mismatch_skipped"
{service="wealth-bot"} | json | event="unhedged_positions_skipped"
{service="wealth-bot"} | json | event="insufficient_balance_skipped"

# All skipped events
{service="wealth-bot"} | json | event=~".*_skipped"

# All position lifecycle events
{service="wealth-bot"} | json | event=~"position_.*"

# WebSocket and connection events
{service="wealth-bot"} | json | event=~"websocket_.*"

# Circuit breaker activity
{service="wealth-bot"} | json | event=~"circuit_breaker_.*"

# Error events requiring attention
{service="wealth-bot"} | json | event="unhedged_position_detected"
{service="wealth-bot"} | json | event="size_discrepancy_detected"

# Count opportunities vs executions
sum by (event) (count_over_time({service="wealth-bot"} | json | event=~"opportunity_detected|arbitrage_executed" [1h]))

See Loki JSON Parsing Guide for the complete list of 150+ event types.

Pattern Matching

# Extract values from logs using regex
{job="wealth-bot"} 
  | regexp "correlation_id=(?P<cid>\\w+)" 
  | line_format "{{.cid}}: {{.message}}"

# Parse structured data
{job="wealth-bot"} 
  | pattern `<_> <level> <module>: <message>`
  | level = "ERROR"

Creating a Logs Dashboard

1. Log Volume Panel

Query:

sum(rate({job="wealth-bot"}[1m])) by (level)

Visualization: Time series graph showing log rate by level

2. Error Rate Panel

Query:

sum(rate({job="wealth-bot", level="ERROR"}[5m]))

Visualization: Stat panel with alert threshold at > 0

3. Recent Errors Table

Query:

{job="wealth-bot", level="ERROR"}

Visualization: Logs panel (table view) Options: Show time, level, and message columns

4. Log Level Distribution

Query:

sum by (level) (count_over_time({job="wealth-bot"}[1h]))

Visualization: Pie chart

5. Module Activity

Query:

sum by (module) (count_over_time({job="wealth-bot"}[1h]))

Visualization: Bar chart

Log File Rotation

To prevent log files from growing too large:

Using logrotate (Linux)

Create /etc/logrotate.d/wealth:

/tmp/wealth*.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    create 0640 thiras thiras
    postrotate
        # Send SIGHUP to reload (if bot supports it)
        # pkill -HUP -f wealth || true
    endscript
}

Test configuration:

sudo logrotate -d /etc/logrotate.d/wealth
sudo logrotate -f /etc/logrotate.d/wealth

Using truncate

Simple script to truncate logs periodically:

#!/bin/bash
# truncate-logs.sh

LOG_FILE="/tmp/wealth.log"
MAX_SIZE_MB=100

if [ -f "$LOG_FILE" ]; then
    SIZE=$(du -m "$LOG_FILE" | cut -f1)
    if [ "$SIZE" -gt "$MAX_SIZE_MB" ]; then
        echo "Truncating $LOG_FILE (${SIZE}MB > ${MAX_SIZE_MB}MB)"
        > "$LOG_FILE"
    fi
fi

Add to crontab:

# Run every hour
0 * * * * /path/to/truncate-logs.sh

Alerting on Logs

Create Alert Rule in Grafana

Go to Alerting → Alert rules → New alert rule

Set query:

sum(rate({job="wealth-bot", level="ERROR"}[5m])) > 0

Set evaluation interval: 1m
Set condition: Alert when query result > 0
Add notification channel (email, Slack, etc.)

Common Alert Rules

High Error Rate:

sum(rate({job="wealth-bot", level="ERROR"}[5m])) > 0.1

WebSocket Connection Failures:

sum(count_over_time({job="wealth-bot"} |= "WebSocket" |= "failed" [5m])) > 3

No Logs Received (Bot Down):

absent_over_time({job="wealth-bot"}[5m]) == 1

Troubleshooting

Pull Model (Promtail)

No logs appearing in Loki

Check Promtail is running:

docker compose ps promtail
docker compose logs promtail

Verify log file exists and is readable:

ls -lah /tmp/wealth.log
tail -f /tmp/wealth.log

Check Promtail positions file:

docker compose exec promtail cat /tmp/positions.yaml

Test Loki directly:

curl -s http://localhost:3100/loki/api/v1/label/job/values

Logs not parsing correctly

Check log format matches regex:

# Example log line
echo "2025-11-10T01:23:45.123456Z  INFO wealth::strategy: Message" | \
  grep -oP '^\S+\s+\w+\s+[\w:]+:\s+.*$'

View Promtail debug logs:

docker compose logs promtail | grep -i error

Push Model (Loki Direct)

No logs appearing in Loki via push

Check Loki is running:

docker compose ps loki
docker compose logs loki

# Check Loki health
curl http://localhost:3100/ready

Verify Loki endpoint is reachable:

# Test HTTP endpoint (note: the /loki/api/v1/push path is added by the library)
curl -v http://localhost:3100/ready

Check bot is configured correctly:

# Verify environment variable is set
echo $OTLP_ENDPOINT

# Should see startup message when running bot:
# "OpenTelemetry initialized with endpoint: http://localhost:4317"

Check Loki logs for errors:

docker compose logs loki | grep -i error
docker compose logs loki | grep -i "push"

Test OpenTelemetry Collector health:

# Check OTLP receiver is responding
curl http://localhost:13133/

# Check metrics being exported to Prometheus
curl http://localhost:8889/metrics | grep wealth

Push connection timeouts

Check network connectivity:

# Test OTLP gRPC endpoint
telnet localhost 4317

# Or check if port is listening
nc -zv localhost 4317

Check Docker network:

docker network inspect wealth_monitoring

Check OpenTelemetry Collector configuration:

# View collector logs for errors
docker compose logs otel-collector

# Verify collector config (in compose.yml)
docker compose config | grep -A 20 otel-collector

Logs delayed or missing

Check OTLP export is working:
- OpenTelemetry batches logs before sending
- Default batch timeout is 10 seconds
- Check bot logs for OTLP export errors

Monitor OpenTelemetry Collector:

# Check collector is receiving telemetry
docker compose logs otel-collector | grep -i "logs"

# Check collector metrics
curl http://localhost:8888/metrics | grep otelcol_receiver

Verify labels are correct:

# Check available labels in Loki
curl http://localhost:3100/loki/api/v1/labels

# Check values for 'service' label
curl http://localhost:3100/loki/api/v1/label/service/values

General Issues

Performance issues

Check Loki disk usage:
```
docker compose exec loki df -h /loki
```
Limit log retention in Loki config:
- Edit Loki config to set retention period
- Default: unlimited (until disk full)

Advanced: JSON Logging

For better log parsing and indexing, JSON logging is supported. This is configured automatically when using OTLP export.

Update Promtail Config

In compose.yml, update the pipeline_stages:

pipeline_stages:
  - json:
      expressions:
        timestamp: timestamp
        level: level
        message: message
        module: target
        span: span
        correlation_id: fields.correlation_id
  - labels:
      level:
      module:
  - timestamp:
      source: timestamp
      format: RFC3339Nano

Log Retention

Loki stores logs with automatic compaction. Configure retention in compose.yml:

loki:
  command: 
    - -config.file=/etc/loki/local-config.yaml
    - -config.expand-env=true
  environment:
    - LOKI_RETENTION_PERIOD=30d

Or create custom Loki config with retention limits.

Best Practices

Use Loki direct push for production - Lower latency, simpler setup than OTLP
Keep file logging for debugging - Hybrid mode provides redundancy
Use structured logging - Include correlation_id, operation, etc.
Set appropriate log levels - Use DEBUG for development, INFO for production
Create dashboards - Visualize key metrics from logs
Set up alerts - Get notified of critical errors
Index important fields - Add labels for common filters (level, module)
Monitor Loki performance - Check ingestion rate and query latency
Configure log retention - Balance storage costs with retention needs
Use correlation IDs - Automatically included in logs for tracing

Comparison: Pull vs Push

Aspect	Pull (Promtail)	Push (Loki Direct)
Setup Complexity	Simple	Simpler (no Promtail needed)
Latency	5-10 seconds	< 1 second
Disk I/O	Required (log files)	Optional
Network Efficiency	Lower (file polling)	Higher (batched HTTP)
Reliability	File-based buffering	In-memory buffering
Scalability	One agent per host	Direct to Loki
Dependencies	Promtail service	None (built into bot)
Production Ready	✓	✓✓ (recommended)

Migration Path: Pull → Push

Phase 1: Enable OpenTelemetry OTLP export

# Keep existing file logging if desired
export WEALTH_LOG_FILE=/tmp/wealth.log

# Add OTLP endpoint
export OTLP_ENDPOINT=http://localhost:4317

wealth run

Phase 2: Verify OTLP export in Grafana
- Check logs appear in Loki via Grafana Explore
- Verify metrics in Prometheus
- Check traces in Tempo
- Confirm correlation between logs/metrics/traces

Phase 3: Disable file logging (optional)

# Remove file logging for OTLP-only mode
unset WEALTH_LOG_FILE

# Keep OTLP export
export OTLP_ENDPOINT=http://localhost:4317

wealth run

Phase 4: Production deployment

# Ensure all observability services are running
docker compose up -d

# Configure bot for OTLP
export OTLP_ENDPOINT=http://localhost:4317
export OTEL_RESOURCE_ATTRIBUTES="service.name=wealth-bot,deployment.environment=production"

wealth run

Monitoring Guide - Complete observability with OpenTelemetry, metrics, and dashboards
Grafana Cloud Setup - Production Grafana setup
Loki JSON Parsing - Complete event types reference
Troubleshooting - Common issues

External Resources

Configuration Hot-Reload

Overview

The Wealth trading bot supports hot-reload for configuration changes, allowing you to update trading parameters without restarting the bot. The system automatically detects changes to config.toml, validates them, and applies safe changes immediately while rejecting unsafe changes that require a full restart.

Features

Automatic Detection: File system monitoring detects config.toml changes within 500ms
Validation: New configuration is validated before applying
Safe/Unsafe Classification: Changes are classified as safe (hot-reloadable) or unsafe (requires restart)
Atomic Updates: All safe changes apply atomically with automatic rollback on failure
Observability: Reload events are tracked via OpenTelemetry metrics and structured logs
Zero Disruption: Active trades continue unaffected during reload
Transaction Semantics: Rollback on validation failure ensures config consistency

Safe vs Unsafe Changes

Safe Changes (Hot-Reloadable) ✅

These changes can be applied at runtime without restarting the bot:

Trading Parameters ([trading] section):

min_funding_spread = 0.0005          # Minimum funding rate spread
min_expected_value = 0.0006          # Minimum EV threshold
staleness_penalty = 0.0003           # Penalty for stale data
max_concurrent_positions = 5         # Max number of positions
position_size_percent = 0.30         # Position size as % of balance
max_position_usd = 10000            # Maximum position size (USD)
max_notional_per_symbol = 10000     # Max notional per symbol
use_kelly_criterion = true           # Enable Kelly Criterion sizing
kelly_fraction = 0.25                # Kelly criterion fraction
max_exchange_utilization = 0.50      # Max balance utilization per exchange
target_profit_percent = 0.05         # Target profit %
update_interval_secs = 60            # Position update interval
max_hedge_attempts = 5               # Max hedging retry attempts
hedge_tolerance_percent = 0.001      # Hedge tolerance

Risk Management ([risk] section):

max_slippage_bps = 50               # Maximum slippage (basis points)
min_slippage_bps = 10               # Minimum slippage
slippage_volatility_multiplier = 1.5 # Volatility multiplier
market_order_fallback_enabled = true # Enable market order fallback
limit_order_timeout_secs = 5        # Limit order timeout
trailing_stops_enabled = true       # Enable trailing stops
trailing_stop_activation = 0.03     # Trailing stop activation threshold
trailing_stop_distance = 0.40       # Trailing stop distance
trailing_stop_min_lock = 0.02       # Minimum profit lock

# Fee estimates (nested under [risk.fees])
[risk.fees]
estimated_slippage = 0.001

Observability ([observability] section):

metrics_port = 9090                 # Metrics server port (won't restart server)
otlp_endpoint = "http://localhost:4317"  # OpenTelemetry endpoint
service_name = "wealth-bot"         # Service name for telemetry
environment = "production"          # Environment label

Licensing ([licensing] section):

license_key = "..."                 # License key (validated on next check)
account_id = "..."                  # Account ID

Unsafe Changes (Require Restart) ⚠️

These changes require a full bot restart to apply safely:

Execution Mode ([execution] section):

mode = "live"                       # paper → live or live → paper

Reason: Switching between paper and live trading requires reinitializing execution clients and ensuring clean state.

Trading Instruments ([[instruments]] sections):

[[instruments]]
exchange = "binance"
symbol = "BTCUSDT"
base_asset = "BTC"
quote_asset = "USDT"

# Adding or removing instruments requires restart

Reason: Adding/removing instruments requires WebSocket reconnection and market data stream initialization.

Leverage Settings ([leverage] section):

default = 3                         # Default leverage

[leverage.overrides]
"binance:BTCUSDT" = 5              # Per-symbol overrides

Reason: Leverage changes require exchange API calls to update margin mode and may affect position calculations.

Resilience Settings ([execution.resilience] section):

[execution.resilience]
circuit_breaker_failures = 5
circuit_breaker_timeout_secs = 60
max_retries = 3
retry_initial_backoff_ms = 100
websocket_max_reconnects = 100

Reason: Changing resilience settings affects circuit breaker and retry state machines that require clean initialization.

Usage

Editing Configuration

Edit config.toml with your preferred editor:
```
vim config.toml
# or
nano config.toml
```
Save the file - The bot will automatically detect the change within 500ms

Check logs for reload status:

INFO wealth::config::watcher: Configuration file changed, reloading... config_path="config.toml"
INFO wealth::config::reload_handler: Processing config reload safe_changes=3 unsafe_changes=0 requires_restart=false
INFO wealth::config::reload_handler: Configuration reloaded successfully changes_applied=3 duration_ms=12

Safe Change Example

Before (config.toml):

[trading]
min_funding_spread = 0.0004
max_position_usd = 10000

After (edit and save):

[trading]
min_funding_spread = 0.0006    # Increased spread threshold
max_position_usd = 15000       # Increased position limit

Result:

Config file change detected within ~500ms (debounce)
Validation and application: ~10-50ms
Strategy effect: On next evaluation cycle (typically within 60s)
No restart required
Metrics recorded: wealth_config_reloads_total{status="success"}

Effect Timing: Configuration changes are applied in two phases:

Shared Config Update (~500ms): Config validated and updated after debounce
Strategy Effect (0-60s): Strategy reads config on next evaluation cycle

Unsafe Change Example

Before:

[execution]
mode = "paper"

After (edit and save):

[execution]
mode = "live"    # Attempting to switch to live trading

Result:

WARN wealth::config::reload_handler: Configuration contains unsafe changes - restart required unsafe_changes=[ExecutionModeChanged { old: Paper, new: Live }]

Changes NOT applied
Current config remains unchanged
Bot continues running with old config
Manual restart required to apply changes

Monitoring & Observability

Metrics

The following OpenTelemetry metrics track reload operations:

wealth_config_reloads_total (Counter)

Labels: status = success, restart_required, no_changes, error
Tracks total reload attempts and their outcomes

wealth_config_reload_duration_seconds (Histogram)

Tracks time taken to process reload (validation + application)
Typical values: 10-50ms for safe changes

wealth_config_reload_errors_total (Counter)

Labels: error_type = load_failed, validation_failed, apply_failed
Tracks reload failures by error type

Logs

Hot-reload events generate structured logs with full context:

File Change Detected:

INFO wealth::config::watcher: Configuration file changed, reloading...
  config_path="config.toml"

Reload Processing:

INFO wealth::config::reload_handler: Processing config reload
  safe_changes=3
  unsafe_changes=0
  requires_restart=false

Success:

INFO wealth::config::reload_handler: Configuration reloaded successfully
  changes_applied=3
  duration_ms=12
  changes=["MinFundingSpreadChanged { old: 0.0004, new: 0.0006 }", "MaxPositionChanged { old: 10000, new: 15000 }", "KellyFractionChanged { old: 0.25, new: 0.30 }"]

Restart Required:

WARN wealth::config::reload_handler: Configuration contains unsafe changes - restart required
  unsafe_changes=["ExecutionModeChanged { old: Paper, new: Live }"]

Validation Failure:

ERROR wealth::config::watcher: Failed to load new config, keeping current config
  error="TOML parse error: invalid value for field 'min_funding_spread'"

Grafana Dashboards

Query examples for monitoring hot-reload in Grafana:

Reload Success Rate:

rate(wealth_config_reloads_total{status="success"}[5m])
/ rate(wealth_config_reloads_total[5m])

Reload Latency (p95):

histogram_quantile(0.95,
  rate(wealth_config_reload_duration_seconds_bucket[5m])
)

Rejected Reloads (requires restart):

increase(wealth_config_reloads_total{status="restart_required"}[1h])

Best Practices

1. Test Changes in Paper Mode First

# Start in paper mode
export WEALTH__EXECUTION__MODE=paper
wealth run

# Edit config.toml with safe changes
vim config.toml

# Verify logs show successful reload
# Then test with live trading

2. Make Incremental Changes

# Good: Change one parameter at a time
[trading]
min_funding_spread = 0.0005  # Changed from 0.0004

# Avoid: Changing many parameters simultaneously
# (harder to debug if something goes wrong)

3. Monitor Metrics After Reload

# Check metrics endpoint
curl http://localhost:9090/metrics | grep config_reload

# Verify success
wealth_config_reloads_total{status="success"} 1

4. Keep Backup Configurations

# Before making changes
cp config.toml config.toml.backup

# If reload fails, restore quickly
mv config.toml.backup config.toml

5. Use Environment Variables for Sensitive Changes

# Avoid putting live API keys in config.toml
export WEALTH__EXECUTION__MODE=paper

# Edit config.toml safely
# Restart manually when ready for live trading

Troubleshooting

Reload Not Detected

Symptom: File changes don't trigger reload

Possible Causes:

Editor uses atomic write (creates temp file, then renames)
File watcher not initialized (check logs for "File watcher initialized")
Insufficient debounce time (rapid successive writes)

Solution:

Wait 1-2 seconds after saving
Check logs for file watcher errors
Verify file permissions on config.toml

Invalid Configuration

Symptom: Reload rejected with validation error

Log:

ERROR wealth::config::watcher: Failed to load new config, keeping current config
  error="missing field `min_funding_spread`"

Solution:

Fix validation error in config.toml
Refer to config.example.toml for correct format
Run validation manually: check logs during bot startup

Restart Required Warning

Symptom: Changes not applied, logs show "restart required"

Log:

WARN wealth::config::reload_handler: Configuration contains unsafe changes - restart required
  unsafe_changes=["ExecutionModeChanged { old: Paper, new: Live }"]

Solution:

Review Unsafe Changes section above

Restart bot to apply unsafe changes:

# Stop bot (Ctrl+C)
# Edit config.toml with unsafe changes
wealth run

Unexpected Behavior After Reload

Symptom: Bot behavior changed but not as expected

Possible Causes:

Multiple safe changes applied simultaneously
Cached values in strategy (future: notify strategy)
Timing of reload during trade execution

Solution:

Check logs for exact changes applied
Verify current config values via metrics/logs
Consider restarting for clean state

Implementation Details

Architecture

The hot-reload system uses a shared configuration architecture:

Data Flow:

1. User edits config.toml
   ↓
2. File system detects change
   ↓
3. Wait 500ms (debounce rapid edits)
   ↓
4. Load and validate new config
   ↓
5. Classify changes:
   ├─→ Safe changes → Apply immediately
   └─→ Unsafe changes → Reject (restart required)
   ↓
6. Strategy uses new config on next cycle (0-60s)

Key Features

Atomic Updates: Config changes apply all-or-nothing with automatic rollback on failure
Zero Downtime: Active trades continue unaffected during reload
Background Integration: Hot-reload watcher starts automatically with other tasks

Performance Impact

Minimal impact on trading performance:

Reload processing: 10-50ms for safe changes
No interruption to active trades
No additional latency for order execution

Limitations

Current Limitations (v0.33.0)

Strategy Effect Delay: Strategy reads config on each evaluation cycle
- Safe changes update shared config immediately (~500ms)
- Strategy effect occurs on next evaluation (0-60s delay)
- This is by design - avoids mid-evaluation config changes
Metrics Server Port: Changing metrics_port requires restart
- Port binding happens once at startup
- Change is logged but doesn't take effect until restart
Credentials: API credentials are not hot-reloadable
- Loaded once at startup from encrypted file
- Require full restart to change

Future Enhancements

Explicit Strategy Notification: Call strategy.apply_config_change() for immediate effect
Per-Component Reload: Selective component reinitialization for some unsafe changes
Configuration History: Track config change history for audit trail
Dynamic Instrument Updates: Add/remove instruments without restart

Security Considerations

File Permissions: Ensure config.toml has appropriate permissions (600 recommended)
Validation: All configs are validated before applying (invalid configs rejected)
Credentials: API keys should be in encrypted credentials.encrypted.json, not config.toml
Audit Trail: All reload events logged with timestamps and change details

Deployment Guide

This guide covers deployment strategies, production configurations, and operational best practices.

Pre-Deployment Checklist

Security

✅ API keys stored in environment variables (not hardcoded)
✅ Use IP whitelisting on Binance
✅ Enable withdrawal whitelist
✅ Use sub-account for trading
✅ Rotate API keys regularly

See: Security Guide for detailed credential management

Testing

✅ Test on testnet first
✅ Verify order placement/cancellation
✅ Test with small position sizes
✅ Monitor for 24 hours before scaling

See: Getting Started for testing procedures

Monitoring

✅ Set up log aggregation (Loki/Elasticsearch)
✅ Configure alerts for errors
✅ Monitor API response times
✅ Track position P&L
✅ Set up Grafana MCP for AI-powered monitoring

See: Monitoring Guide for comprehensive observability setup
See: Grafana MCP Setup for AI monitoring integration

Infrastructure

✅ Run on reliable server (not laptop)
✅ Use systemd for process management
✅ Configure automatic restarts
✅ Set up backup strategies

Production Deployment Options

Choose the deployment method that best fits your infrastructure:

graph LR
    A[Deployment Options] --> B[Systemd Service]
    A --> C[Docker Container]
    A --> D[Kubernetes Pod]
    
    B --> B1[Single Server<br/>Minimal Complexity<br/>systemctl management]
    C --> C1[Docker Compose<br/>Local/Cloud<br/>With monitoring stack]
    D --> D1[High Availability<br/>Auto-scaling<br/>Production scale]
    
    style B fill:#90caf9
    style C fill:#a5d6a7
    style D fill:#ce93d8

Systemd Service (Recommended for Single Server)

Best for single-server deployments with minimal complexity.

Service Configuration

Create /etc/systemd/system/wealth.service:

[Unit]
Description=Wealth Trading Bot
After=network.target

[Service]
Type=simple
User=trader
WorkingDirectory=/home/trader
Environment="CREDENTIALS_PASSPHRASE=your-secure-passphrase"
Environment="WEALTH__EXECUTION__MODE=live"
Environment="WEALTH__LICENSING__LICENSE_KEY=your-license-key"
ExecStart=/usr/local/bin/wealth run
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Enable and Start

# Reload systemd configuration
sudo systemctl daemon-reload

# Enable service to start on boot
sudo systemctl enable wealth

# Start the service
sudo systemctl start wealth

# Check service status
sudo systemctl status wealth

# View logs
sudo journalctl -u wealth -f

Management Commands

# Stop the service
sudo systemctl stop wealth

# Restart the service
sudo systemctl restart wealth

# Check service health
sudo systemctl is-active wealth

# View recent logs
sudo journalctl -u wealth -n 100

Docker Deployment

Best for containerized environments and cloud deployments.

Using Pre-built Image (Recommended)

# Pull the latest image
docker pull ghcr.io/thiras/wealth:latest

# Run with encrypted credentials
docker run -d --name wealth \
  -v $(pwd)/config.toml:/app/config.toml \
  -v $(pwd)/credentials.encrypted.json:/app/credentials.encrypted.json \
  -e CREDENTIALS_PASSPHRASE=your-passphrase \
  -e WEALTH__EXECUTION__MODE=live \
  -e WEALTH__LICENSING__LICENSE_KEY=your-license-key \
  ghcr.io/thiras/wealth:latest

Docker Compose (Recommended)

See compose.yml in the project root for a complete setup including OpenTelemetry Collector and Grafana.

# Download compose file
curl -LO https://raw.githubusercontent.com/thiras/wealth/main/compose.yml

# Start all services
docker compose up -d

# View logs
docker compose logs -f wealth

# Stop all services
docker compose down

Building Custom Image

For developers who need to build from source, see the Development Guide.

Kubernetes (Advanced)

Best for production-scale deployments requiring high availability and orchestration.

Deployment Manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: wealth-trading-bot
spec:
  replicas: 1
  selector:
    matchLabels:
      app: wealth
  template:
    metadata:
      labels:
        app: wealth
    spec:
      containers:
      - name: wealth
        image: wealth:latest
        env:
        - name: CREDENTIALS_PASSPHRASE
          valueFrom:
            secretKeyRef:
              name: wealth-secrets
              key: credentials-passphrase
        - name: WEALTH__LICENSING__LICENSE_KEY
          valueFrom:
            secretKeyRef:
              name: wealth-secrets
              key: license-key
        - name: WEALTH__EXECUTION__MODE
          value: "live"
        volumeMounts:
        - name: credentials
          mountPath: /app/credentials.encrypted.json
          subPath: credentials.encrypted.json
          readOnly: true
        - name: credentials
          mountPath: /app/.credentials.salt
          subPath: .credentials.salt
          readOnly: true
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"

Secrets Management

# Create secret for API credentials
kubectl create secret generic exchange-credentials \
  --from-literal=binance-api-key=xxx \
  --from-literal=binance-secret-key=yyy

# Apply deployment
kubectl apply -f deployment.yaml

# Check pod status
kubectl get pods -l app=wealth

# View logs
kubectl logs -l app=wealth -f

For complete Kubernetes manifests, check the /deploy/k8s/ directory (if available).

Deployment Best Practices

1. Environment Separation

Maintain separate configurations for different environments:

# Development
export WEALTH__EXECUTION__MODE=paper
# Configure testnet in config.toml or use encrypted credentials with testnet keys

# Staging
export WEALTH__EXECUTION__MODE=paper
# Use production API endpoints with paper mode for testing

# Production
export WEALTH__EXECUTION__MODE=live
export WEALTH__LICENSING__LICENSE_KEY=your-license-key
# Use encrypted credentials with production API keys

2. Graceful Shutdown

The bot implements graceful shutdown (6-8 seconds typical):

What happens:

Stop accepting - No new trades initiated
Close positions - All active positions closed safely
Cleanup - Statistics printed, connections closed

Trigger Options:

# Interactive mode
Ctrl+C

# Send graceful shutdown signal
kill -SIGTERM $(pidof wealth)

# Docker/Kubernetes (automatic on stop)
docker stop wealth  # Sends SIGTERM, waits 10s, then SIGKILL

Monitoring Shutdown:

Watch logs for "Shutdown signal received" messages
Verify "All positions closed successfully" message
Check final statistics in logs before exit

Configuration:

# Docker Compose
services:
  wealth:
    stop_grace_period: 10s

# Kubernetes
terminationGracePeriodSeconds: 10

3. Health Checks

Use the health endpoint for container orchestration:

# Check bot health
curl http://localhost:9090/health

# Kubernetes liveness probe
livenessProbe:
  httpGet:
    path: /health
    port: 9090
  initialDelaySeconds: 30
  periodSeconds: 10

4. Log Rotation

Configure log rotation to prevent disk space issues:

# systemd journal rotation
sudo journalctl --vacuum-time=7d
sudo journalctl --vacuum-size=500M

5. Backup Strategies

Export position data regularly: wealth export positions
Backup configuration files
Document API key rotation procedures

Configuration Guide - Environment variables and settings
Security Guide - Credential management and best practices
Monitoring Guide - Observability and alerting
Troubleshooting - Common deployment issues
Getting Started - Initial setup and testing

Next Steps

After deployment:

Monitor logs for the first 24 hours
Verify metrics are being collected
Test graceful shutdown and automatic restart
Set up alerting for critical errors
Document your deployment-specific configurations

For ongoing operations, see the Monitoring Guide.

Troubleshooting Guide

Common issues and solutions when running the Wealth trading bot.

Authentication Errors

Error: Binance API error (status=401, code=-2015): Invalid API-key

Causes:

Incorrect API key or secret
API key not enabled for futures trading
IP address not whitelisted
API key expired or revoked

Solutions:

Verify API key is correct:
```
wealth config show
```
Check IP whitelist on your exchange (Binance → API Management → Edit API)
Ensure API permissions include "Futures Trading"
Regenerate keys if unsure and update configuration

Prevention:

Use encrypted credentials instead of environment variables
Enable IP whitelisting after creating API keys
Test on testnet before live trading

Timestamp Errors

Error: Binance API error (status=400, code=-1021): Timestamp out of sync

Causes:

System clock drift
NTP service not running
Network latency issues

Solutions:

# Sync system clock with NTP
sudo ntpdate -s time.nist.gov

# Or use systemd-timesyncd
sudo systemctl restart systemd-timesyncd
sudo timedatectl status

# Install chrony (recommended for production)
sudo apt install chrony
sudo systemctl enable chrony
sudo systemctl start chrony

Verification:

timedatectl
# Should show: System clock synchronized: yes

Insufficient Balance

Error: Binance API error (status=400, code=-2019): Margin insufficient

Causes:

Insufficient USDT in futures wallet
Existing positions using available margin
Position size exceeds available balance

Solutions:

Check current balance:
```
wealth balance
```
Transfer funds to your futures wallet on the exchange

Reduce position size in config.toml:

[trading]
max_position_usd = 5000  # Lower this value

Close existing positions if needed:
```
wealth close --all
```

Rate Limiting

Error: Rate limit exceeded or 429 Too Many Requests

Causes:

Too many API requests in short time
Aggressive polling settings
Multiple bot instances sharing API keys

Solutions:

Wait and retry - Rate limits typically reset in 1-5 minutes
Check for multiple instances:
```
ps aux | grep wealth
```
Upgrade exchange tier for higher limits (Binance VIP, Bybit API tier)
Spread API keys across different bots if running multiple

WebSocket Issues

Error: WebSocket connection failed or WebSocket disconnected

Causes:

Unstable internet connection
Firewall blocking WebSocket connections
Exchange maintenance

Solutions:

Check internet connection:

ping fapi.binance.com
ping api.bybit.com

Allow WebSocket connections through firewall (port 443)
Check exchange status:
- Binance: https://www.binance.com/en/support/announcement
- Bybit: https://announcements.bybit.com/
Restart the bot - It auto-reconnects, but a restart ensures clean state:
```
# Stop with Ctrl+C, then restart
wealth run
```

Connection Issues

Error: Failed to connect to exchange or Network error

Causes:

DNS resolution failure
Network timeout
Exchange API down

Solutions:

Test connectivity:

curl -I https://fapi.binance.com
curl -I https://api.bybit.com
curl -I https://api.hyperliquid.xyz

Check DNS:
```
nslookup fapi.binance.com
```
Use alternative DNS (Google: 8.8.8.8, Cloudflare: 1.1.1.1)

Configuration Issues

Error: Configuration validation failed

Solutions:

Validate configuration:
```
wealth verify
```
Check config file syntax:
```
cat config.toml
```

Use example config as reference:

cp config.example.toml config.toml
# Then edit with your values

Credential Issues

Error: Failed to decrypt credentials

Causes:

Wrong passphrase
Missing credentials file
Corrupted credentials file

Solutions:

Verify passphrase is set correctly:
```
echo $CREDENTIALS_PASSPHRASE
```
Check credentials file exists:
```
ls -la credentials.encrypted.json
```

Re-create credentials if corrupted:

export CREDENTIALS_PASSPHRASE="your_passphrase"
wealth credentials create
wealth credentials add binance YOUR_KEY YOUR_SECRET
wealth credentials verify

Health Check

Quick diagnostic commands:

# Check bot health
curl http://localhost:9090/health | jq

# Check positions
wealth positions

# Check balances
wealth balance

# Verify configuration
wealth verify

# View current config
wealth config

Common Log Messages

Normal Operation

Initializing bot | mode=paper_trading - Starting in paper mode
WebSocket connected - Exchange connection established
Bot running | metrics_port=9090 - Bot is operational

Warnings (Usually OK)

Opportunity detected but spread too low - Normal filtering
Skipping execution: insufficient balance - Not enough funds for trade
Rate limit approaching - Consider reducing activity

Errors (Need Attention)

Authentication failed - Check API credentials
Order placement failed - Check balance and configuration
WebSocket disconnected - Network issue, will auto-reconnect

Getting Help

If you can't resolve an issue:

Check logs for detailed error messages:

wealth run --verbose 2>&1 | tee bot.log

Collect diagnostic info:

wealth --version
wealth verify
curl http://localhost:9090/health

Test in paper mode first:

export WEALTH__EXECUTION__MODE=paper
wealth run

Security & Credentials Guide

Guide to credential management, encryption, and security best practices

Overview

The bot implements comprehensive security measures to protect your API credentials:

AES-256-GCM encryption for stored credentials
PBKDF2 key derivation with 100,000 iterations
Automatic secret redaction in logs
Secure memory handling for sensitive data

Encrypted Credentials (Recommended)

How It Works

Your exchange API credentials are encrypted using:

Your passphrase → converted to encryption key via PBKDF2
AES-256-GCM → industry-standard authenticated encryption
Unique salt → stored separately for additional security

┌─────────────────┐     ┌─────────────────┐
│  Your Passphrase │────►│  PBKDF2 (100k)  │────► Encryption Key
└─────────────────┘     └─────────────────┘
                                │
┌─────────────────┐             ▼
│ API Credentials │────► AES-256-GCM ────► credentials.encrypted.json
└─────────────────┘

Setup

# 1. Set your passphrase
export CREDENTIALS_PASSPHRASE="your-secure-passphrase-here"

# 2. Create the encrypted credentials file
wealth credentials create

# 3. Add your exchange credentials (interactive mode - recommended)
wealth credentials add binance
wealth credentials add bybit
wealth credentials add hyperliquid

# Alternative: Add credentials with CLI args (less secure - appears in shell history)
# wealth credentials add binance --api-key YOUR_API_KEY --secret-key YOUR_SECRET_KEY

# 4. Verify everything works
wealth credentials verify

Security Note: Always prefer the interactive mode (without --api-key and --secret-key flags) as it prevents credentials from appearing in your shell history. The bot will prompt you securely for these values.

Files Created

File	Purpose	Security
`credentials.encrypted.json`	Encrypted credentials	AES-256-GCM encrypted
`.credentials.salt`	Encryption salt	Required for decryption

⚠️ Both files are required for decryption. Back them up securely!

Production Best Practices

File Permissions

# Set restrictive permissions
chmod 600 credentials.encrypted.json
chmod 600 .credentials.salt

# Verify
ls -la credentials.encrypted.json .credentials.salt
# Should show: -rw------- (owner read/write only)

Passphrase Management

DO NOT:

❌ Store passphrase in plain text files
❌ Commit passphrase to version control
❌ Share passphrase in chat/email
❌ Use weak passphrases (< 12 characters)
❌ Reuse passphrases across environments

DO:

✅ Use secrets management systems (AWS Secrets Manager, HashiCorp Vault)
✅ Use strong passphrases (12+ characters, mixed case, numbers, symbols)
✅ Rotate passphrases regularly
✅ Different passphrases for testnet/mainnet
✅ Audit access to encrypted files

Examples:

# AWS Secrets Manager
export CREDENTIALS_PASSPHRASE="$(aws secretsmanager get-secret-value \
    --secret-id prod/wealth/passphrase \
    --query SecretString \
    --output text)"

# HashiCorp Vault
export CREDENTIALS_PASSPHRASE="$(vault kv get \
    -field=passphrase \
    secret/wealth/prod)"

# Kubernetes Secret
export CREDENTIALS_PASSPHRASE="$(kubectl get secret wealth-passphrase \
    -o jsonpath='{.data.passphrase}' | base64 -d)"

Version Control

Add to .gitignore:

# Encrypted credentials
credentials.encrypted.json
.credentials.salt

# Environment files
.env
.env.local
.env.*.local

Backup Strategy

What to backup:

.credentials.salt - CRITICAL - Cannot decrypt without this
credentials.encrypted.json - Encrypted credentials
Passphrase - Store separately in secrets manager

Backup procedure:

# Backup to secure storage
aws s3 cp .credentials.salt s3://secure-backups/wealth/$(date +%Y%m%d)/
aws s3 cp credentials.encrypted.json s3://secure-backups/wealth/$(date +%Y%m%d)/

Docker Deployment

Docker Compose:

services:
  wealth:
    build: .
    environment:
      - CREDENTIALS_PASSPHRASE=${CREDENTIALS_PASSPHRASE}
    secrets:
      - credentials_passphrase
    volumes:
      - ./credentials.encrypted.json:/app/credentials.encrypted.json:ro
      - ./.credentials.salt:/app/.credentials.salt:ro

secrets:
  credentials_passphrase:
    external: true

Kubernetes Deployment

Create Secrets:

# Create from files
kubectl create secret generic wealth-credentials \
    --from-file=credentials.encrypted.json \
    --from-file=.credentials.salt

# Create passphrase secret
kubectl create secret generic wealth-passphrase \
    --from-literal=passphrase='your_secure_passphrase'

Deployment manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: wealth-bot
spec:
  template:
    spec:
      containers:
      - name: wealth
        image: wealth:latest
        env:
        - name: CREDENTIALS_PASSPHRASE
          valueFrom:
            secretKeyRef:
              name: wealth-passphrase
              key: passphrase
        volumeMounts:
        - name: credentials
          mountPath: /app/credentials.encrypted.json
          subPath: credentials.encrypted.json
          readOnly: true
        - name: credentials
          mountPath: /app/.credentials.salt
          subPath: .credentials.salt
          readOnly: true
      volumes:
      - name: credentials
        secret:
          secretName: wealth-credentials
          defaultMode: 0400

Security Audit Checklist

Before Production Deployment:

Credentials encrypted with strong passphrase (12+ chars)
Salt file exists and backed up (.credentials.salt)
File permissions set to 600
Encrypted files excluded from version control
Passphrase stored securely (not in plaintext files)
No hardcoded credentials
Verification test passed (wealth credentials verify)
Backup strategy implemented
Credential rotation procedure documented

Troubleshooting

"Failed to decrypt credentials" error:

Check passphrase is correct
Verify .credentials.salt file exists and hasn't been modified
Ensure credentials.encrypted.json hasn't been corrupted

"Salt file not found" error:

Verify .credentials.salt file exists in current directory
Check file permissions (should be readable)
Restore from backup if missing

Secrets still appearing in logs:

Verify using Secret<String> wrapper types
Check custom Debug implementation is present
Grep logs for credential patterns: grep -i "api_key" logs/
Should only show [REDACTED]

Licensing

Proprietary Software - Copyright © 2025 Ant Somers. All Rights Reserved.
See LICENSE for full terms.

Wealth Trading Bot requires a valid license for production use.

Overview

Development Mode: Licensing is optional (for testing)
Production Mode: Valid license required
Machine Binding: Each license can be activated on a limited number of machines
Automatic Validation: License is re-validated periodically during runtime

Getting a License

Contact: Visit wealth.thiras.net for licensing inquiries
Receive License Key: You'll receive a license key via secure channel
Configure: Set the license key environment variable

export WEALTH__LICENSING__LICENSE_KEY="YOUR-LICENSE-KEY-HERE"

Run the Bot: License activates automatically on first run

wealth run

License Commands

Check License Status

wealth license check

Shows:

License ID and status
Expiration date
Current machine activation

Show Machine Fingerprint

wealth license show-fingerprint

Use this when contacting support about licensing issues.

Validate License

wealth license validate

Tests license connectivity and configuration.

Deactivate License

wealth license deactivate --confirm

Use when transferring your license to a different machine.

Transferring Your License

Moving to a New Machine

On the old machine, deactivate:
```
wealth license deactivate --confirm
```

On the new machine, configure and run:

export WEALTH__LICENSING__LICENSE_KEY="your-license-key"
wealth run

The license will automatically activate on the new machine.

If You Can't Access the Old Machine

Contact support with your machine fingerprint:

wealth license show-fingerprint

They can remotely deactivate your old machine.

License Types

Type	Use Case	Features
Development	Testing & development	Lower machine limits, shorter duration
Production	Live trading	Higher machine limits, longer duration

Troubleshooting

"License key not configured"

Set your license key:

export WEALTH__LICENSING__LICENSE_KEY="your-license-key"

"Machine activation limit exceeded"

You've reached your activation limit. Either:

Deactivate an old machine: wealth license deactivate --confirm
Contact support to increase your limit

"License has expired"

Contact support to renew your license.

"Network error"

Ensure your firewall allows HTTPS connections to api.keygen.sh.

Security Best Practices

Never commit license keys to version control
Use environment variables for license configuration
Contact support if you suspect your key was compromised

FAQ

Q: Can I run the bot without a license for testing?
A: Yes, debug builds don't require a license.

Q: How many machines can I activate?
A: Depends on your license type. Check with sales.

Q: What happens if my license expires while running?
A: The bot will gracefully shut down within 6 hours, closing all positions safely.

Q: Can I transfer my license to a different machine?
A: Yes, deactivate on the old machine first, then activate on the new one.

Support

Sales & Licensing: Visit wealth.thiras.net
Technical Support: Open an issue on GitHub (include your machine fingerprint)