Troubleshooting Guide

Common issues and solutions when running the Wealth trading bot.

Troubleshooting Flow

flowchart TB
    START[Issue Detected] --> AUTH{Authentication<br/>Error?}
    
    AUTH -->|Yes| AUTHFIX[Check API Keys<br/>IP Whitelist<br/>Permissions]
    AUTH -->|No| TIME{Timestamp<br/>Error?}
    
    TIME -->|Yes| TIMEFIX[Sync System Clock<br/>Install NTP/Chrony]
    TIME -->|No| BAL{Balance<br/>Error?}
    
    BAL -->|Yes| BALFIX[Check Balance<br/>Transfer Funds<br/>Reduce Position Size]
    BAL -->|No| CONN{Connection<br/>Error?}
    
    CONN -->|Yes| CONNFIX[Check Internet<br/>Firewall Rules<br/>Exchange Status]
    CONN -->|No| CONFIG{Config<br/>Error?}
    
    CONFIG -->|Yes| CONFIGFIX[Run wealth verify<br/>Run wealth init<br/>Check TOML Syntax]
    CONFIG -->|No| HELP[Contact Support<br/>Check Logs]
    
    style START fill:#ffcdd2
    style AUTHFIX fill:#c8e6c9
    style TIMEFIX fill:#c8e6c9
    style BALFIX fill:#c8e6c9
    style CONNFIX fill:#c8e6c9
    style CONFIGFIX fill:#c8e6c9
    style HELP fill:#e3f2fd

Table of Contents

Authentication Errors

Error: Binance API error (status=401, code=-2015): Invalid API-key

Causes:

  • Incorrect API key or secret
  • API key not enabled for futures trading
  • IP address not whitelisted
  • API key expired or revoked

Solutions:

  1. Verify API key is correct:

    wealth config show

  2. Check IP whitelist on your exchange (Binance → API Management → Edit API)

  3. Ensure API permissions include "Futures Trading"

  4. Regenerate keys if unsure and update configuration

Prevention:

  • Use encrypted credentials instead of environment variables
  • Enable IP whitelisting after creating API keys
  • Test on testnet before live trading

Timestamp Errors

Error: Binance API error (status=400, code=-1021): Timestamp out of sync

Causes:

  • System clock drift
  • NTP service not running
  • Network latency issues

Solutions:

# Sync system clock with NTP
sudo ntpdate -s time.nist.gov

# Or use systemd-timesyncd
sudo systemctl restart systemd-timesyncd
sudo timedatectl status

# Install chrony (recommended for production)
sudo apt install chrony
sudo systemctl enable chrony
sudo systemctl start chrony

Verification:

timedatectl
# Should show: System clock synchronized: yes

Insufficient Balance

Error: Binance API error (status=400, code=-2019): Margin insufficient

Causes:

  • Insufficient USDT in futures wallet
  • Existing positions using available margin
  • Position size exceeds available balance

Solutions:

  1. Check current balance:

    wealth balance

  2. Transfer funds to your futures wallet on the exchange

  3. Reduce position size in config.toml:

    [trading]
max_position_usd = 5000  # Lower this value

  4. Close existing positions if needed:

    wealth close --all

Market Unavailable / Delisting Errors

Error: Market closed or unavailable or Bybit API error (code 30228) (delisting)

Causes:

  • Symbol is being delisted from the exchange
  • Market is temporarily closed for maintenance
  • Exchange has disabled trading for the symbol

What Happens:

  • The bot automatically marks the affected symbol as "unhealthy" on the specific exchange
  • Future attempts to trade this symbol on that exchange are blocked
  • Other exchanges remain unaffected (the unhealthy status is exchange-specific)

Symptoms in Logs:

event = "pair_marked_unhealthy"
symbol = "DAMUSDT"
exchange = "BybitPerpetualsUsd"
reason = "market_unavailable"

Solutions:

  1. Check exchange announcements for delisting notices

  2. Remove the symbol from your instruments configuration:

    [instruments]
symbols = ["BTCUSDT", "ETHUSDT"]  # Remove the delisted symbol

  3. Use dynamic discovery instead of manual instrument lists:

    [instrument_discovery]
enabled = true
min_volume_24h_usd = 10_000_000  # Filter low-volume symbols

  4. Monitor pair health in the TUI dashboard or logs:

    grep "pair_marked_unhealthy\|market_unavailable" ~/.wealth/logs/wealth.log

  5. Check rejection metrics in Grafana/Prometheus:

    # Count of opportunities skipped due to unhealthy pairs
wealth_opportunity_rejections_total{reason="pair_unhealthy"}

# Order errors due to market unavailable
wealth_order_errors_total{error_type="market_unavailable"}

Prevention:

  • Enable dynamic pair discovery to automatically filter out low-liquidity pairs
  • Set appropriate volume thresholds to avoid trading symbols near delisting
  • The bot will automatically recover when a symbol becomes available again
  • Monitor the pair_unhealthy rejection metric for early warning signs

Position Closing Issues

Error: reduce_only orders had zero fill but positions still exist or MANUAL INTERVENTION REQUIRED

Causes:

  • Position state mismatch between bot and exchange (e.g., partial liquidation)
  • ADL (Auto-Deleveraging) reduced position size externally
  • Partial or full liquidation occurred
  • Manual trades outside the bot modified positions
  • Network issues during position close
  • Exchange API returning stale data

Pre-Close Position Verification (v0.52+)

Before attempting to close positions, the bot now verifies actual position quantities from exchanges. This prevents close failures from quantity mismatch:

Detection Flow:

flowchart TB
    CLOSE[Close Position Triggered] --> QUERY[Query Actual Positions]
    QUERY --> COMPARE{Compare Stored<br/>vs Actual}
    
    COMPARE -->|Match| PROCEED[Close with Stored Qty]
    COMPARE -->|Drift >10%| ALERT[Log Drift Alert]
    COMPARE -->|Both Gone| CLEANUP[Remove from Tracking]
    
    ALERT --> ADJUST[Use Safe Quantity]
    ADJUST --> PROCEED2[Close with Adjusted Qty]
    
    style ALERT fill:#ffcdd2
    style CLEANUP fill:#c8e6c9
    style PROCEED fill:#c8e6c9
    style PROCEED2 fill:#fff9c4

Position Drift Events:

  • position_drift_alert - Significant drift detected (>10%)
  • long_position_missing - Long position completely gone (liquidated?)
  • short_position_missing - Short position completely gone (liquidated?)
  • positions_already_closed - Both sides already closed externally

Force Close Fallback

When reduce_only orders fail with zero fill, the bot automatically attempts a force_close_position fallback:

  • Fetches current position state directly from the exchange
  • Uses configurable slippage (hyperliquid_close_slippage, default 5%)
  • Retries the close with accurate position data

When Manual Intervention is Required:

If both reduce_only and force_close fallback fail, you'll see MANUAL INTERVENTION REQUIRED in the logs. This can happen due to:

  • Exchange API downtime
  • Network connectivity issues
  • Extreme market volatility

Solutions:

  1. Check current positions on exchanges:

    wealth positions

  2. Manually close positions on the exchange UI if needed

  3. Increase slippage for more reliable closes (default 5%):

    [execution.reconciliation]
hyperliquid_close_slippage = 0.10  # 10%

  4. Check metrics for fallback frequency:

    wealth_position_force_close_fallback_total

  5. Enable orphan position cleanup if bot is sole user of accounts:

    [execution.reconciliation]
auto_close_orphans = true

Stuck Executions & Auto-Shutdown (v0.53+)

Warning: Auto-shutdown triggered after N consecutive emergency close failures

What This Means:

Starting with v0.53, the bot automatically shuts down when emergency close operations repeatedly fail. This is a safety mechanism to prevent:

  • Unhedged positions accumulating
  • Liquidation risk from stuck executions
  • Cascading failures across exchanges

What Causes This:

  1. Exchange API outages preventing position closes
  2. Position limit reached on hedge exchange (e.g., Aster -5018 error)
  3. Circuit breaker blocking closes (mitigated in v0.53 with bypass)
  4. Network connectivity issues

Emergency Actions:

  1. Manually close positions immediately on exchange UIs before liquidation:

    • Binance Futures: futures.binance.com
    • HyperLiquid: app.hyperliquid.xyz
    • Aster: app.aster.finance

  2. Check for unhedged positions:

    wealth positions

  3. Review logs for root cause:

    grep "emergency_close\|PartialFill\|RollingBack" ~/.wealth/logs/wealth.log

Configuration Options:

[execution.reconciliation]
# How long before execution is considered stuck (default: 60s, was 300s pre-v0.53)
stuck_execution_threshold_secs = 60

# Auto-shutdown after this many consecutive emergency close failures
emergency_close_failure_threshold = 3  # Set to 0 to disable

What Changed in v0.53:

  • Per-exchange circuit breakers: Aster failures no longer block Binance closes
  • Circuit breaker bypass: Emergency closes bypass open circuit breakers
  • Faster stuck detection: 60s default (was 300s)
  • Auto-shutdown: Prevents runaway liquidation risk

Rate Limiting

Error: Rate limit exceeded or 429 Too Many Requests

Causes:

  • Too many API requests in short time
  • Aggressive polling settings
  • Multiple bot instances sharing API keys

Solutions:

  1. Wait and retry - Rate limits typically reset in 1-5 minutes

  2. Check for multiple instances:

    ps aux | grep wealth

  3. Upgrade exchange tier for higher limits (Binance VIP, Bybit API tier)

  4. Spread API keys across different bots if running multiple

WebSocket Issues

Error: WebSocket connection failed or WebSocket disconnected

Causes:

  • Unstable internet connection
  • Firewall blocking WebSocket connections
  • Exchange maintenance

Solutions:

  1. Check internet connection:

    ping fapi.binance.com
ping api.bybit.com

  2. Allow WebSocket connections through firewall (port 443)

  3. Check exchange status:

    • Binance: https://www.binance.com/en/support/announcement
    • Bybit: https://announcements.bybit.com/

  4. Restart the bot - It auto-reconnects, but a restart ensures clean state:

    # Stop with Ctrl+C, then restart
wealth run

Rate Limit Exceeded (v0.58.4+)

Error: WebSocket reconnection rate limit exceeded - entering cooldown

Causes:

  • Exchange experiencing outage or maintenance
  • Network connectivity issues causing rapid disconnections
  • DNS resolution failures

What Happens:

  1. After 10 failed reconnection attempts within 60 seconds, the bot enters a 5-minute cooldown
  2. A Critical severity alert is triggered
  3. The bot logs the cooldown duration and waits before retrying
  4. Other exchanges continue operating normally (per-exchange isolation)

Solutions:

  1. Wait for cooldown to expire - The bot will automatically retry after 5 minutes

  2. Check if the exchange is down:

    curl -I https://fstream.binance.com
curl -I https://stream.bybit.com

  3. Check your network: Persistent DNS or routing issues can cause rapid failures

  4. Monitor logs for the root cause:

    # Look for the underlying connection error
grep "websocket_connection_error" wealth.log | tail -20

Prevention: The rate limiter prevents "death spiral" scenarios where rapid reconnection attempts exhaust API limits and cause cascade failures across all exchanges.

Connection Issues

Error: Failed to connect to exchange or Network error

Causes:

  • DNS resolution failure
  • Network timeout
  • Exchange API down

Solutions:

  1. Test connectivity:

    curl -I https://fapi.binance.com
curl -I https://api.bybit.com
curl -I https://api.hyperliquid.xyz
curl -I https://fapi.asterdex.com

  2. Check DNS:

    nslookup fapi.binance.com

  3. Use alternative DNS (Google: 8.8.8.8, Cloudflare: 1.1.1.1)

Configuration Issues

Error: Configuration validation failed or Configuration file not found

Solutions:

  1. Initialize configuration (creates from embedded template):

    wealth init

  2. Validate existing configuration:

    wealth verify

  3. Check config file syntax:

    cat config.toml

  4. Reset to defaults (overwrites existing):

    wealth init --force

Credential Issues

Error: Failed to decrypt credentials

Causes:

  • Wrong passphrase
  • Missing credentials file
  • Corrupted credentials file

Solutions:

  1. Check passphrase is set correctly
  2. Verify files exist: ls -la credentials.encrypted.json .credentials.salt
  3. Re-create if corrupted: wealth credentials create && wealth credentials add binance

📖 For complete credential setup, see Security Guide.

Health Check

Quick diagnostic commands:

# Check bot health
curl http://localhost:9090/health | jq

# View positions and balances
wealth positions
wealth balance

# Verify configuration
wealth verify

📖 For detailed monitoring, see Monitoring Guide.

Common Log Messages

Normal Operation

  • Initializing bot | mode=paper_trading - Starting in paper mode
  • WebSocket connected - Exchange connection established
  • Bot running | metrics_port=9090 - Bot is operational

Warnings (Usually OK)

  • Opportunity detected but spread too low - Normal filtering
  • Skipping execution: insufficient balance - Not enough funds for trade
  • Rate limit approaching - Consider reducing activity

Errors (Need Attention)

  • Authentication failed - Check API credentials
  • Order placement failed - Check balance and configuration
  • WebSocket disconnected - Network issue, will auto-reconnect

Getting Help

If you can't resolve an issue:

  1. Check logs for detailed error messages:

    wealth run --verbose 2>&1 | tee bot.log

  2. Collect diagnostic info:

    wealth --version
wealth verify
curl http://localhost:9090/health

  3. Test in paper mode first:

    export WEALTH__EXECUTION__MODE=paper
wealth run

See Also