chia-blockchain: Harvester memory leakage [Bug]

What happened?

Harvester crashes after some days of work. It seems that a memory leakage occurs.

Resource-Exhaustion-Detector: Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: start_harvester.exe (5716) consumed 20,333,797,376 bytes

It started after enabling compressed plot farming.

Weakest PC info: WIN 10, 6 gb RAM, GTX 1660, 23GB free on C: after reboot

The problem persists on 3 different PCs

Current config:

CHIA_ALERTS_PUBKEY: ***
chia_ssl_ca:
  crt: config/ssl/ca/chia_ca.crt
  key: config/ssl/ca/chia_ca.key
daemon_max_message_size: 50000000
daemon_port: 55400
daemon_ssl:
  private_crt: config/ssl/daemon/private_daemon.crt
  private_key: config/ssl/daemon/private_daemon.key
farmer:
  full_node_peer:
    host: localhost
    port: 8444
  harvester_peer:
    host: localhost
    port: 8448
  logging: &id001
    log_filename: log/debug.log
    log_level: WARNING
    log_maxfilesrotation: 7
    log_stdout: false
    log_syslog: false
    log_syslog_host: localhost
    log_syslog_port: 514
  network_overrides: &id002
    config:
      mainnet:
        address_prefix: xch
        default_full_node_port: 8444
      testnet0:
        address_prefix: txch
      testnet1:
        address_prefix: txch
      testnet2:
        address_prefix: txch
      testnet3:
        address_prefix: txch
      testnet4:
        address_prefix: txch
      testnet7:
        address_prefix: txch
        default_full_node_port: 58444
    constants:
      mainnet:
        GENESIS_CHALLENGE: ***
        GENESIS_PRE_FARM_FARMER_PUZZLE_HASH: ***
        GENESIS_PRE_FARM_POOL_PUZZLE_HASH: ***
        NETWORK_TYPE: 0
      testnet0:
        GENESIS_CHALLENGE: ***
        GENESIS_PRE_FARM_FARMER_PUZZLE_HASH: ***
        GENESIS_PRE_FARM_POOL_PUZZLE_HASH: ***
        MIN_PLOT_SIZE: 18
        NETWORK_TYPE: 1
      testnet2:
        DIFFICULTY_CONSTANT_FACTOR: 10052721566054
        GENESIS_CHALLENGE: ***
        GENESIS_PRE_FARM_FARMER_PUZZLE_HASH: ***
        GENESIS_PRE_FARM_POOL_PUZZLE_HASH: ***
        MIN_PLOT_SIZE: 18
        NETWORK_TYPE: 1
      testnet3:
        DIFFICULTY_CONSTANT_FACTOR: 10052721566054
        GENESIS_CHALLENGE: ***
        GENESIS_PRE_FARM_FARMER_PUZZLE_HASH: ***
        GENESIS_PRE_FARM_POOL_PUZZLE_HASH: ***
        MEMPOOL_BLOCK_BUFFER: 10
        MIN_PLOT_SIZE: 18
        NETWORK_TYPE: 1
      testnet4:
        DIFFICULTY_CONSTANT_FACTOR: 10052721566054
        DIFFICULTY_STARTING: 30
        EPOCH_BLOCKS: 768
        GENESIS_CHALLENGE: ***
        GENESIS_PRE_FARM_FARMER_PUZZLE_HASH: ***
        GENESIS_PRE_FARM_POOL_PUZZLE_HASH: ***
        MEMPOOL_BLOCK_BUFFER: 10
        MIN_PLOT_SIZE: 18
        NETWORK_TYPE: 1
      testnet5:
        DIFFICULTY_CONSTANT_FACTOR: 10052721566054
        DIFFICULTY_STARTING: 30
        EPOCH_BLOCKS: 768
        GENESIS_CHALLENGE: ***
        GENESIS_PRE_FARM_FARMER_PUZZLE_HASH: ***
        GENESIS_PRE_FARM_POOL_PUZZLE_HASH: ***
        MEMPOOL_BLOCK_BUFFER: 10
        MIN_PLOT_SIZE: 18
        NETWORK_TYPE: 1
      testnet7:
        DIFFICULTY_CONSTANT_FACTOR: 10052721566054
        DIFFICULTY_STARTING: 30
        EPOCH_BLOCKS: 768
        GENESIS_CHALLENGE: ***
        GENESIS_PRE_FARM_FARMER_PUZZLE_HASH: ***
        GENESIS_PRE_FARM_POOL_PUZZLE_HASH: ***
        MEMPOOL_BLOCK_BUFFER: 50
        MIN_PLOT_SIZE: 18
        NETWORK_TYPE: 1
  pool_public_keys: !!set
    ***: null
  pool_share_threshold: 1000
  port: 8447
  rpc_port: 8559
  selected_network: mainnet
  ssl:
    private_crt: config/ssl/farmer/private_farmer.crt
    private_key: config/ssl/farmer/private_farmer.key
    public_crt: config/ssl/farmer/public_farmer.crt
    public_key: config/ssl/farmer/public_farmer.key
  start_rpc_server: true
  xch_target_address: ***
full_node:
  database_path: db/blockchain_v2_CHALLENGE.sqlite
  db_sync: auto
  dns_servers:
  - dns-introducer.chia.net
  enable_profiler: false
  enable_upnp: true
  exempt_peer_networks: []
  farmer_peer:
    host: localhost
    port: 8447
  introducer_peer:
    host: introducer.chia.net
    port: 8444
  log_sqlite_cmds: false
  logging: *id001
  max_inbound_farmer: 10
  max_inbound_timelord: 5
  max_inbound_wallet: 20
  network_overrides: *id002
  peer_connect_interval: 30
  peer_connect_timeout: 30
  peer_db_path: db/peer_table_node.sqlite
  port: 8444
  recent_peer_threshold: 6000
  rpc_port: 8555
  sanitize_weight_proof_only: false
  selected_network: mainnet
  send_uncompact_interval: 0
  short_sync_blocks_behind_threshold: 20
  simulator_database_path: sim_db/simulator_blockchain_v1_CHALLENGE.sqlite
  simulator_peer_db_path: sim_db/peer_table_node.sqlite
  ssl:
    private_crt: config/ssl/full_node/private_full_node.crt
    private_key: config/ssl/full_node/private_full_node.key
    public_crt: config/ssl/full_node/public_full_node.crt
    public_key: config/ssl/full_node/public_full_node.key
  start_rpc_server: true
  sync_blocks_behind_threshold: 300
  target_outbound_peer_count: 8
  target_peer_count: 80
  target_uncompact_proofs: 100
  timelord_peer:
    host: localhost
    port: 8446
  wallet_peer:
    host: localhost
    port: 8449
  weight_proof_timeout: 360
harvester:
  chia_ssl_ca:
    crt: config/ssl/ca/chia_ca.crt
    key: config/ssl/ca/chia_ca.key
  decompressor_thread_count: 1
  farmer_peer:
    host: localhost
    port: 8447
  logging: *id001
  network_overrides: *id002
  num_threads: 30
  parallel_decompressor_count: 1
  parallel_read: true
  plot_directories:
  - D:\
  - G:\
  - C:\
  - E:\
  plots_refresh_parameter:
    batch_size: 300
    batch_sleep_milliseconds: 1
    interval_seconds: 120
    retry_invalid_seconds: 1200
  port: 8448
  private_ssl_ca:
    crt: config/ssl/ca/private_ca.crt
    key: config/ssl/ca/private_ca.key
  rpc_port: 8560
  selected_network: mainnet
  ssl:
    private_crt: config/ssl/harvester/private_harvester.crt
    private_key: config/ssl/harvester/private_harvester.key
  start_rpc_server: true
inbound_rate_limit_percent: 100
introducer:
  host: localhost
  logging: *id001
  max_peers_to_send: 20
  network_overrides: *id002
  port: 8445
  recent_peer_threshold: 6000
  selected_network: mainnet
  ssl:
    public_crt: config/ssl/full_node/public_full_node.crt
    public_key: config/ssl/full_node/public_full_node.key
logging: *id001
min_mainnet_k_size: 32
network_overrides: *id002
outbound_rate_limit_percent: 30
ping_interval: 120
pool:
  logging: *id001
  network_overrides: *id002
  pool_list:
  - authentication_public_key: ***
    launcher_id: '***'
    owner_public_key: '***'
    p2_singleton_puzzle_hash: '***'
    payout_instructions: ***
    pool_url: https://pool.findchia.com
    target_puzzle_hash: '***'
  - authentication_public_key: ***
    launcher_id: '***'
    owner_public_key: '***'
    p2_singleton_puzzle_hash: '***'
    payout_instructions: ***
    pool_url: ''
    target_puzzle_hash: '***'
  selected_network: mainnet
  xch_target_address: ***
private_ssl_ca:
  crt: config/ssl/ca/private_ca.crt
  key: config/ssl/ca/private_ca.key
selected_network: mainnet
self_hostname: localhost
timelord:
  fast_algorithm: false
  full_node_peer:
    host: localhost
    port: 8444
  logging: *id001
  max_connection_time: 60
  network_overrides: *id002
  port: 8446
  sanitizer_mode: false
  selected_network: mainnet
  ssl:
    private_crt: config/ssl/timelord/private_timelord.crt
    private_key: config/ssl/timelord/private_timelord.key
    public_crt: config/ssl/timelord/public_timelord.crt
    public_key: config/ssl/timelord/public_timelord.key
  vdf_clients:
    ip:
    - localhost
    - localhost
    - 127.0.0.1
    ips_estimate:
    - 150000
  vdf_server:
    host: localhost
    port: 8000
timelord_launcher:
  host: localhost
  logging: *id001
  port: 8000
  process_count: 3
ui:
  daemon_host: localhost
  daemon_port: 55400
  daemon_ssl:
    private_crt: config/ssl/daemon/private_daemon.crt
    private_key: config/ssl/daemon/private_daemon.key
  logging: *id001
  network_overrides: *id002
  port: 8222
  rpc_port: 8555
  selected_network: mainnet
  ssh_filename: config/ssh_host_key
wallet:
  database_path: wallet/db/blockchain_wallet_v1_CHALLENGE_KEY.sqlite
  db_sync: auto
  enable_profiler: false
  full_node_peer:
    host: localhost
    port: 8444
  initial_num_public_keys: 100
  initial_num_public_keys_new_wallet: 5
  introducer_peer:
    host: introducer.chia.net
    port: 8444
  logging: *id001
  network_overrides: *id002
  num_sync_batches: 50
  peer_connect_interval: 60
  port: 8449
  recent_peer_threshold: 6000
  rpc_port: 9256
  selected_network: mainnet
  short_sync_blocks_behind_threshold: 20
  ssl:
    private_crt: config/ssl/wallet/private_wallet.crt
    private_key: config/ssl/wallet/private_wallet.key
    public_crt: config/ssl/wallet/public_wallet.crt
    public_key: config/ssl/wallet/public_wallet.key
  start_height_buffer: 100
  starting_height: 0
  target_peer_count: 5
  testing: false
  trusted_peers:
    trusted_node_1: config/ssl/full_node/public_full_node.crt
  wallet_peers_path: wallet/db/wallet_peers.sqlite

Version

2.1.1

What platform are you using?

Windows

What ui mode are you using?

GUI

Relevant log output

No response

About this issue

Original URL
State: open
Created 7 months ago
Comments: 17 (4 by maintainers)

Most upvoted comments

I’m experiencing this same issue, but only with my remote harvester. And I have a large footprint. It’s a Dell server with 2x 8 core 2.1Ghz CPUs, 384GB of RAM, with 40,708 C05 plots. It doesn’t seem to matter what I set the swapfile to. After less than a day, I get the virtual memory exhaustion somewhere in the System Logs and the node stops providing any plots to my main farmer. start_harvester.exe continues to run most of the time. It just does nothing. So, my powershell script that monitors it doesn’t trigger an alert.

I had previously run all of these plots of of my main harvester that also had about 37,000 more plots. It didn’t have virtual memory issues like this. It had rather high latency with the harvester, so that’s why I moved 1/2 to a remote harvester.

JamesKrolak on Dec 23, 2023

Same problem when using compressed plot farming on windows 10. When start farming, the commit mem increase(about 0.1G/min). And the mem will not be released if stop farming. Both GUI and CLI mode will cause this mem leak. I’m sure GPU plotting cause leak, and CPU plotting may also cause leak.

karlestira on Dec 14, 2023