Configuration Hot-Reload

Overview

The Wealth trading bot supports hot-reload for configuration changes, allowing you to update trading parameters without restarting the bot. The system automatically detects changes to config.toml, validates them, and applies safe changes immediately while rejecting unsafe changes that require a full restart.

Hot-Reload Flow

flowchart LR
    subgraph Detection["File Detection"]
        EDIT[Edit config.toml] --> WATCH[File Watcher]
        WATCH -->|"~500ms"| DEBOUNCE[Debounce]
    end
    
    subgraph Processing["Reload Processing"]
        DEBOUNCE --> VALIDATE[Validate Config]
        VALIDATE -->|Invalid| REJECT[Reject Changes]
        VALIDATE -->|Valid| CLASSIFY[Classify Changes]
        
        CLASSIFY --> SAFE[Safe Changes]
        CLASSIFY --> UNSAFE[Unsafe Changes]
    end
    
    subgraph Apply["Application"]
        SAFE -->|"~10-50ms"| APPLY[Apply Atomically]
        UNSAFE --> LOG[Log Warning]
        LOG --> RESTART[Requires Restart]
        
        APPLY --> METRICS[Record Metrics]
        APPLY --> EFFECT[Strategy Effect<br/>0-60s]
    end
    
    style SAFE fill:#c8e6c9
    style UNSAFE fill:#ffcdd2
    style APPLY fill:#e3f2fd

Features

Automatic Detection: File system monitoring detects config.toml changes within 500ms
Validation: New configuration is validated before applying
Safe/Unsafe Classification: Changes are classified as safe (hot-reloadable) or unsafe (requires restart)
Atomic Updates: All safe changes apply atomically with automatic rollback on failure
Observability: Reload events are tracked via OpenTelemetry metrics and structured logs
Zero Disruption: Active trades continue unaffected during reload
Transaction Semantics: Rollback on validation failure ensures config consistency

Safe vs Unsafe Changes

flowchart TB
    subgraph Safe["✅ Safe Changes - Hot-Reloadable"]
        direction TB
        T1[trading.min_funding_spread]
        T2[trading.max_position_usd]
        T3[trading.position_size_percent]
        R1[risk.max_slippage_bps]
        R2[risk.trailing_stops_enabled]
        O1[observability.metrics_port]
        I1[instruments add/remove*]
    end
    
    subgraph Unsafe["⚠️ Unsafe Changes - Require Restart"]
        direction TB
        E1[execution.mode]
        L1[leverage.default]
        L2[leverage.overrides]
        RE1[execution.resilience.circuit_breaker_failures]
        RE2[execution.resilience.circuit_breaker_timeout_secs]
        RE3[execution.resilience.max_retries]
    end
    
    style Safe fill:#c8e6c9
    style Unsafe fill:#ffcdd2

Safe Changes (Hot-Reloadable) ✅

These changes can be applied at runtime without restarting the bot:

Trading Parameters ([trading] section):

min_funding_spread = 0.05          # Minimum funding rate spread
min_expected_value = 0.06          # Minimum EV threshold
staleness_penalty = 0.03           # Penalty for stale data
staleness_haircut_threshold_secs = 120  # Apply haircut when data > this age
max_concurrent_positions = 5         # Max number of positions
position_size_percent = 0.30         # Position size as % of balance
max_position_usd = 10000            # Maximum position size (USD)
max_notional_per_symbol = 10000     # Max notional per symbol
use_kelly_criterion = true           # Enable Kelly Criterion sizing
kelly_fraction = 0.25                # Kelly criterion fraction
max_exchange_utilization = 0.50      # Max balance utilization per exchange
target_profit_percent = 0.05         # Target profit %
update_interval_secs = 60            # Position update interval
max_hedge_attempts = 5               # Max hedging retry attempts
hedge_tolerance_percent = 0.001      # Hedge tolerance
fallback_price_usd = 100000         # Fallback price when WebSocket data unavailable

Risk Management ([risk] section):

max_slippage_bps = 50               # Maximum slippage (basis points)
min_slippage_bps = 10               # Minimum slippage
slippage_volatility_multiplier = 1.5 # Volatility multiplier
market_order_fallback_enabled = true # Enable market order fallback
limit_order_timeout_secs = 5        # Limit order timeout
trailing_stops_enabled = true       # Enable trailing stops
trailing_stop_activation = 0.03     # Trailing stop activation threshold
trailing_stop_distance = 0.40       # Trailing stop distance
trailing_stop_min_lock = 0.02       # Minimum profit lock

# Fee estimates (nested under [risk.fees])
[risk.fees]
estimated_slippage = 0.001

Observability ([observability] section):

metrics_port = 9090                 # Metrics server port (won't restart server)
otlp_endpoint = "http://localhost:4317"  # OpenTelemetry endpoint
service_name = "wealth-bot"         # Service name for telemetry
environment = "production"          # Environment label

Licensing ([licensing] section):

license_key = "..."                 # License key (validated on next check)
account_id = "..."                  # Account ID

Trading Instruments ([[instruments]] sections) (v0.31+):

[[instruments]]
exchange = "binance"
symbol = "BTCUSDT"
base_asset = "BTC"
quote_asset = "USDT"

# Adding new instruments is always allowed
# Removing instruments is blocked if active positions exist

When instruments are changed:

Added Instruments: Always allowed. Leverage is automatically set for new pairs and WebSocket streams restart to subscribe to new symbols.
Removed Instruments: Blocked if the pair has active positions. Enable force_remove_pairs to auto-cancel pending orders before removal.

[instrument_discovery]
force_remove_pairs = false    # Set to true to auto-cancel orders on pair removal

Unsafe Changes (Require Restart) ⚠️

These changes require a full bot restart to apply safely:

Execution Mode ([execution] section):

mode = "live"                       # paper → live or live → paper

Reason: Switching between paper and live trading requires reinitializing execution clients and ensuring clean state.

Leverage Settings ([leverage] section):

default = 3                         # Default leverage

[leverage.overrides]
"binance:BTCUSDT" = 5              # Per-symbol overrides

Reason: Leverage changes require exchange API calls to update margin mode and may affect position calculations.

Resilience Settings ([execution.resilience] section):

[execution.resilience]
circuit_breaker_failures = 5
circuit_breaker_timeout_secs = 60
max_retries = 3
retry_initial_backoff_ms = 100
websocket_max_reconnects = 100

Reason: Changing resilience settings affects circuit breaker and retry state machines that require clean initialization.

Usage

Editing Configuration

Edit config.toml with your preferred editor:
```
vim config.toml
# or
nano config.toml
```
Save the file - The bot will automatically detect the change within 500ms

Check logs for reload status:

INFO wealth::config::watcher: Configuration file changed, reloading... config_path="/etc/wealth/config.toml"
INFO wealth::config::reload_handler: Processing config reload safe_changes=3 unsafe_changes=0 requires_restart=false
INFO wealth::config::reload_handler: Configuration reloaded successfully changes_applied=3 duration_ms=12

Kubernetes / Helm Deployment

When running in Kubernetes with the official Helm chart, hot-reload works automatically with ConfigMap changes:

How it works:

The Helm chart mounts configuration as a ConfigMap at /etc/wealth/config.toml
The WEALTH_CONFIG environment variable points to this path
When hotReload.enabled: true (default), Kubernetes can update the ConfigMap in-place
The file watcher detects changes and applies them

Helm values for hot-reload:

# Hot-reload is enabled by default
hotReload:
  enabled: true  # Mount ConfigMap as directory (allows in-place updates)

config:
  enabled: true
  # Configuration values here...

Updating configuration:

# Option 1: Update via helm upgrade
helm upgrade wealth oci://ghcr.io/thiras/charts/wealth \
  --reuse-values \
  --set config.trading.min_funding_spread=0.06

# Option 2: Edit HelmChart CRD directly (K3s)
kubectl edit helmchart wealth -n kube-system
# Edit valuesContent, save - K3s will update the ConfigMap

# Option 3: Patch the ConfigMap directly
kubectl patch configmap wealth-config -n wealth --type merge \
  -p '{"data":{"config.toml":"..."}}'

Timing:

ConfigMap update → ~30-60 seconds for Kubernetes to propagate to pod
File watcher detection → ~500ms after file change
Total: ~30-90 seconds for config to take effect

Verify reload in logs:

kubectl logs -f deployment/wealth -n wealth | grep -E "config|reload"

Note: If hotReload.enabled: false, the chart adds a checksum/config annotation that triggers pod restart on ConfigMap changes instead of hot-reload.

Safe Change Example

Before (config.toml):

[trading]
min_funding_spread = 0.04
max_position_usd = 10000

After (edit and save):

[trading]
min_funding_spread = 0.0006    # Increased spread threshold
max_position_usd = 15000       # Increased position limit

Result:

Config file change detected within ~500ms (debounce)
Validation and application: ~10-50ms
Strategy effect: On next evaluation cycle (typically within 60s)
No restart required
Metrics recorded: wealth_config_reloads_total{status="success"}

Effect Timing: Configuration changes are applied in two phases:

Shared Config Update (~500ms): Config validated and updated after debounce
Strategy Effect (0-60s): Strategy reads config on next evaluation cycle

Instrument Hot-Reload Example (v0.31+)

Adding a New Trading Pair:

Before:

[[instruments]]
exchange = "binance"
symbol = "BTCUSDT"
base_asset = "BTC"
quote_asset = "USDT"

After (add ETHUSDT):

[[instruments]]
exchange = "binance"
symbol = "BTCUSDT"
base_asset = "BTC"
quote_asset = "USDT"

[[instruments]]
exchange = "binance"
symbol = "ETHUSDT"
base_asset = "ETH"
quote_asset = "USDT"

Result:

INFO wealth::app::task_manager: Processing instrument update callback added=["ETHUSDT"] removed=[]
INFO wealth::app::task_manager: Setting leverage for newly added instrument symbol="ETHUSDT" leverage=3
INFO wealth::strategy::market_data: Updating instruments via hot-reload symbols_count=2 symbols=["BTCUSDT", "ETHUSDT"]
INFO wealth::market_data: Updating active symbols for WebSocket streams old_count=1 new_count=2 added=["ETHUSDT"]
INFO wealth::market_data: WebSocket streams restarted successfully symbols_count=2

Removing a Trading Pair (with active position):

If you try to remove BTCUSDT while there's an active position:

WARN wealth::config::reload_handler: Instrument removal blocked due to active positions blocked_symbols=["BTCUSDT"]

The removal is blocked to prevent orphaned positions. Close the position first or enable force_remove_pairs to auto-cancel pending orders.

Unsafe Change Example

Before:

[execution]
mode = "paper"

After (edit and save):

[execution]
mode = "live"    # Attempting to switch to live trading

Result:

WARN wealth::config::reload_handler: Configuration contains unsafe changes - restart required unsafe_changes=[ExecutionModeChanged { old: Paper, new: Live }]

Changes NOT applied
Current config remains unchanged
Bot continues running with old config
Manual restart required to apply changes

Monitoring & Observability

Metrics

The following OpenTelemetry metrics track reload operations:

wealth_config_reloads_total (Counter)

Labels: status = success, restart_required, no_changes, error
Tracks total reload attempts and their outcomes

wealth_config_reload_duration_seconds (Histogram)

Tracks time taken to process reload (validation + application)
Typical values: 10-50ms for safe changes

wealth_config_reload_errors_total (Counter)

Labels: error_type = load_failed, validation_failed, apply_failed
Tracks reload failures by error type

Instrument Hot-Reload Metrics (v0.31+):

wealth_instruments_added_total (Counter)

Tracks total instruments added via hot-reload

wealth_instruments_removed_total (Counter)

Tracks total instruments removed via hot-reload

wealth_active_instruments (Gauge)

Current number of active trading instruments

Logs

Hot-reload events generate structured logs with full context:

File Change Detected:

INFO wealth::config::watcher: Configuration file changed, reloading...
  config_path="config.toml"

Reload Processing:

INFO wealth::config::reload_handler: Processing config reload
  safe_changes=3
  unsafe_changes=0
  requires_restart=false

Success:

INFO wealth::config::reload_handler: Configuration reloaded successfully
  changes_applied=3
  duration_ms=12
  changes=["MinFundingSpreadChanged { old: 0.04, new: 0.06 }", "MaxPositionChanged { old: 10000, new: 15000 }", "KellyFractionChanged { old: 0.25, new: 0.30 }"]

Restart Required:

WARN wealth::config::reload_handler: Configuration contains unsafe changes - restart required
  unsafe_changes=["ExecutionModeChanged { old: Paper, new: Live }"]

Validation Failure:

ERROR wealth::config::watcher: Failed to load new config, keeping current config
  error="TOML parse error: invalid value for field 'min_funding_spread'"

Grafana Dashboards

Query examples for monitoring hot-reload in Grafana:

Reload Success Rate:

rate(wealth_config_reloads_total{status="success"}[5m])
/ rate(wealth_config_reloads_total[5m])

Reload Latency (p95):

histogram_quantile(0.95,
  rate(wealth_config_reload_duration_seconds_bucket[5m])
)

Rejected Reloads (requires restart):

increase(wealth_config_reloads_total{status="restart_required"}[1h])

Best Practices

1. Test Changes in Paper Mode First

# Start in paper mode
export WEALTH__EXECUTION__MODE=paper
wealth run

# Edit config.toml with safe changes
vim config.toml

# Verify logs show successful reload
# Then test with live trading

2. Make Incremental Changes

# Good: Change one parameter at a time
[trading]
min_funding_spread = 0.05  # Changed from 0.04

# Avoid: Changing many parameters simultaneously
# (harder to debug if something goes wrong)

3. Monitor Metrics After Reload

# Check metrics endpoint
curl http://localhost:9090/metrics | grep config_reload

# Verify success
wealth_config_reloads_total{status="success"} 1

4. Keep Backup Configurations

# Before making changes
cp config.toml config.toml.backup

# If reload fails, restore quickly
mv config.toml.backup config.toml

5. Use Environment Variables for Sensitive Changes

# Avoid putting live API keys in config.toml
export WEALTH__EXECUTION__MODE=paper

# Edit config.toml safely
# Restart manually when ready for live trading

Troubleshooting

Reload Not Detected

Symptom: File changes don't trigger reload

Possible Causes:

Editor uses atomic write (creates temp file, then renames)
File watcher not initialized (check logs for "File watcher initialized")
Insufficient debounce time (rapid successive writes)

Solution:

Wait 1-2 seconds after saving
Check logs for file watcher errors
Verify file permissions on config.toml

Invalid Configuration

Symptom: Reload rejected with validation error

Log:

ERROR wealth::config::watcher: Failed to load new config, keeping current config
  error="missing field `min_funding_spread`"

Solution:

Fix validation error in config.toml
Refer to config.example.toml for correct format
Run validation manually: check logs during bot startup

Restart Required Warning

Symptom: Changes not applied, logs show "restart required"

Log:

WARN wealth::config::reload_handler: Configuration contains unsafe changes - restart required
  unsafe_changes=["ExecutionModeChanged { old: Paper, new: Live }"]

Solution:

Review Unsafe Changes section above

Restart bot to apply unsafe changes:

# Stop bot (Ctrl+C)
# Edit config.toml with unsafe changes
wealth run

Unexpected Behavior After Reload

Symptom: Bot behavior changed but not as expected

Possible Causes:

Multiple safe changes applied simultaneously
Cached values in strategy (future: notify strategy)
Timing of reload during trade execution

Solution:

Check logs for exact changes applied
Verify current config values via metrics/logs
Consider restarting for clean state

Implementation Details

Architecture

The hot-reload system uses a shared configuration architecture:

Data Flow:

1. User edits config.toml
   ↓
2. File system detects change
   ↓
3. Wait 500ms (debounce rapid edits)
   ↓
4. Load and validate new config
   ↓
5. Classify changes:
   ├─→ Safe changes → Apply immediately
   └─→ Unsafe changes → Reject (restart required)
   ↓
6. Strategy uses new config on next cycle (0-60s)

Key Features

Atomic Updates: Config changes apply all-or-nothing with automatic rollback on failure
Zero Downtime: Active trades continue unaffected during reload
Background Integration: Hot-reload watcher starts automatically with other tasks

Performance Impact

Minimal impact on trading performance:

Reload processing: 10-50ms for safe changes
No interruption to active trades
No additional latency for order execution

Limitations

Current Limitations

Strategy Effect Delay: Strategy reads config on each evaluation cycle
- Safe changes update shared config immediately (~500ms)
- Strategy effect occurs on next evaluation (0-60s delay)
- This is by design - avoids mid-evaluation config changes
Metrics Server Port: Changing metrics_port requires restart
- Port binding happens once at startup
- Change is logged but doesn't take effect until restart
Credentials: API credentials are not hot-reloadable
- Loaded once at startup from encrypted file
- Require full restart to change

Future Enhancements

Explicit Strategy Notification: Call strategy.apply_config_change() for immediate effect
Per-Component Reload: Selective component reinitialization for some unsafe changes
Configuration History: Track config change history for audit trail
Dynamic Instrument Updates: Add/remove instruments without restart

Security Considerations

File Permissions: Ensure config.toml has appropriate permissions (600 recommended)
Validation: All configs are validated before applying (invalid configs rejected)
Credentials: API keys should be in encrypted credentials.encrypted.json, not config.toml
Audit Trail: All reload events logged with timestamps and change details

Wealth Trading System - User Guide

Configuration Hot-Reload

Overview

Hot-Reload Flow

Features

Safe vs Unsafe Changes

Safe Changes (Hot-Reloadable) ✅

Unsafe Changes (Require Restart) ⚠️

Usage

Editing Configuration

Kubernetes / Helm Deployment

Safe Change Example

Instrument Hot-Reload Example (v0.31+)

Unsafe Change Example

Monitoring & Observability

Metrics

Logs

Grafana Dashboards

Best Practices

1. Test Changes in Paper Mode First

2. Make Incremental Changes

3. Monitor Metrics After Reload

4. Keep Backup Configurations

5. Use Environment Variables for Sensitive Changes

Troubleshooting

Reload Not Detected

Invalid Configuration

Restart Required Warning

Unexpected Behavior After Reload

Implementation Details

Architecture

Key Features

Performance Impact

Limitations

Current Limitations

Future Enhancements

Security Considerations

See Also