lisk-sdk: Failed to generate blocks in v4.0.0-beta.1

In v4.0.0-beta.1 the node stopped to generate blocks with the following error. The chain on the node kept moving but stopped to generate blocks.

Error (no other errors in logs):

2023-06-08T02:09:33.225Z ERROR betanet engine 332042 [err=Execution context is already initialized with 6e0dd8df5d840a67ff8bfc7f291e8de2bf65ad1f86399a4d05ac7c8e9c41e643] Failed to generate a block
/home/lisk/lisk-core/bin/lisk-core: line 54: 332042 Killed                  "$NODE" "$DIR/run" "$@"

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 16 (7 by maintainers)

Most upvoted comments

Another update. From my syslogs. looks like 2 GB RAM without SWAP are not enough. I will add a SWAP of 4 GB and see if it works:

Jun 14 15:11:51 betanet kernel: [1903749.357609] node /home/lisk invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Jun 14 15:11:51 betanet kernel: [1903749.357615] CPU: 1 PID: 100948 Comm: node /home/lisk Not tainted 5.15.0-70-generic #77-Ubuntu
Jun 14 15:11:51 betanet kernel: [1903749.357618] Hardware name: Hetzner vServer/Standard PC (Q35 + ICH9, 2009), BIOS 20171111 11/11/2017
Jun 14 15:11:51 betanet kernel: [1903749.357619] Call Trace:
Jun 14 15:11:51 betanet kernel: [1903749.357621]  <TASK>
Jun 14 15:11:51 betanet kernel: [1903749.357623]  show_stack+0x52/0x5c
Jun 14 15:11:51 betanet kernel: [1903749.357628]  dump_stack_lvl+0x4a/0x63
Jun 14 15:11:51 betanet kernel: [1903749.357631]  dump_stack+0x10/0x16
Jun 14 15:11:51 betanet kernel: [1903749.357632]  dump_header+0x53/0x228
Jun 14 15:11:51 betanet kernel: [1903749.357635]  oom_kill_process.cold+0xb/0x10
Jun 14 15:11:51 betanet kernel: [1903749.357637]  out_of_memory+0x106/0x2e0
Jun 14 15:11:51 betanet kernel: [1903749.357641]  __alloc_pages_slowpath.constprop.0+0x9b7/0xa80
Jun 14 15:11:51 betanet kernel: [1903749.357644]  __alloc_pages+0x311/0x330
Jun 14 15:11:51 betanet kernel: [1903749.357646]  alloc_pages+0x9e/0x1e0
Jun 14 15:11:51 betanet kernel: [1903749.357648]  __page_cache_alloc+0x7e/0x90
Jun 14 15:11:51 betanet kernel: [1903749.357650]  pagecache_get_page+0x152/0x590
Jun 14 15:11:51 betanet kernel: [1903749.357652]  ? page_cache_ra_unbounded+0x166/0x210
Jun 14 15:11:51 betanet kernel: [1903749.357654]  filemap_fault+0x488/0xab0
Jun 14 15:11:51 betanet kernel: [1903749.357655]  ? filemap_map_pages+0x309/0x400
Jun 14 15:11:51 betanet kernel: [1903749.357657]  __do_fault+0x3c/0x120
Jun 14 15:11:51 betanet kernel: [1903749.357659]  do_read_fault+0xeb/0x160
Jun 14 15:11:51 betanet kernel: [1903749.357661]  do_fault+0xa0/0x2e0
Jun 14 15:11:51 betanet kernel: [1903749.357663]  handle_pte_fault+0x1cd/0x240
Jun 14 15:11:51 betanet kernel: [1903749.357665]  __handle_mm_fault+0x405/0x6f0
Jun 14 15:11:51 betanet kernel: [1903749.357667]  handle_mm_fault+0xd8/0x2c0
Jun 14 15:11:51 betanet kernel: [1903749.357669]  do_user_addr_fault+0x1c9/0x670
Jun 14 15:11:51 betanet kernel: [1903749.357672]  exc_page_fault+0x77/0x170
Jun 14 15:11:51 betanet kernel: [1903749.357674]  asm_exc_page_fault+0x27/0x30
Jun 14 15:11:51 betanet kernel: [1903749.357677] RIP: 0033:0x1d9b9f0
Jun 14 15:11:51 betanet kernel: [1903749.357684] Code: Unable to access opcode bytes at RIP 0x1d9b9c6.
Jun 14 15:11:51 betanet kernel: [1903749.357685] RSP: 002b:00007ffd6f51c5d0 EFLAGS: 00010293
Jun 14 15:11:51 betanet kernel: [1903749.357688] RAX: 00000000000000c1 RBX: 00007ffd6f51c700 RCX: 00007ffd6f51c8e0
Jun 14 15:11:51 betanet kernel: [1903749.357689] RDX: 00007ffd6f51c958 RSI: 000000000000005c RDI: 00007ffd6f51c7e0
Jun 14 15:11:51 betanet kernel: [1903749.357691] RBP: 00007ffd6f51c5e0 R08: 0000000000000061 R09: 0000000005d43278
Jun 14 15:11:51 betanet kernel: [1903749.357692] R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000005c
Jun 14 15:11:51 betanet kernel: [1903749.357693] R13: 00007ffd6f51c700 R14: 0000000000000003 R15: 000000007fffffff
Jun 14 15:11:51 betanet kernel: [1903749.357695]  </TASK>
Jun 14 15:11:51 betanet kernel: [1903749.357696] Mem-Info:
Jun 14 15:11:51 betanet kernel: [1903749.357697] active_anon:165 inactive_anon:444558 isolated_anon:0
Jun 14 15:11:51 betanet kernel: [1903749.357697]  active_file:23 inactive_file:4 isolated_file:0
Jun 14 15:11:51 betanet kernel: [1903749.357697]  unevictable:7681 dirty:0 writeback:0
Jun 14 15:11:51 betanet kernel: [1903749.357697]  slab_reclaimable:5924 slab_unreclaimable:8340
Jun 14 15:11:51 betanet kernel: [1903749.357697]  mapped:2287 shmem:980 pagetables:2443 bounce:0
Jun 14 15:11:51 betanet kernel: [1903749.357697]  kernel_misc_reclaimable:0
Jun 14 15:11:51 betanet kernel: [1903749.357697]  free:16693 free_pcp:624 free_cma:0
Jun 14 15:11:51 betanet kernel: [1903749.357701] Node 0 active_anon:660kB inactive_anon:1778232kB active_file:92kB inactive_file:16kB unevictable:30724kB isolated(anon):0kB isolated(file):0kB mapped:9148kB dirty:0kB writeback:0kB shmem:3920kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:3056kB pagetables:9772kB all_unreclaimable? no
Jun 14 15:11:51 betanet kernel: [1903749.357705] Node 0 DMA free:7812kB min:356kB low:444kB high:532kB reserved_highatomic:0KB active_anon:0kB inactive_anon:7192kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:12kB local_pcp:12kB free_cma:0kB
Jun 14 15:11:51 betanet kernel: [1903749.357709] lowmem_reserve[]: 0 1866 1866 1866 1866
Jun 14 15:11:51 betanet kernel: [1903749.357711] Node 0 DMA32 free:58960kB min:44696kB low:55868kB high:67040kB reserved_highatomic:14336KB active_anon:660kB inactive_anon:1771040kB active_file:324kB inactive_file:16kB unevictable:30724kB writepending:0kB present:2031472kB managed:1958616kB mlocked:27652kB bounce:0kB free_pcp:2484kB local_pcp:2220kB free_cma:0kB
Jun 14 15:11:51 betanet kernel: [1903749.357715] lowmem_reserve[]: 0 0 0 0 0
Jun 14 15:11:51 betanet kernel: [1903749.357717] Node 0 DMA: 5*4kB (M) 12*8kB (UME) 9*16kB (UME) 6*32kB (UME) 3*64kB (UE) 6*128kB (UME) 3*256kB (ME) 5*512kB (UME) 1*1024kB (U) 1*2048kB (M) 0*4096kB = 7812kB
Jun 14 15:11:51 betanet kernel: [1903749.357727] Node 0 DMA32: 380*4kB (UME) 864*8kB (ME) 518*16kB (UME) 318*32kB (UME) 169*64kB (UME) 60*128kB (UME) 23*256kB (UME) 5*512kB (ME) 5*1024kB (UM) 0*2048kB 0*4096kB = 58960kB
Jun 14 15:11:51 betanet kernel: [1903749.357739] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Jun 14 15:11:51 betanet kernel: [1903749.357741] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jun 14 15:11:51 betanet kernel: [1903749.357742] 3273 total pagecache pages
Jun 14 15:11:51 betanet kernel: [1903749.357742] 0 pages in swap cache
Jun 14 15:11:51 betanet kernel: [1903749.357743] Swap cache stats: add 0, delete 0, find 0/0
Jun 14 15:11:51 betanet kernel: [1903749.357744] Free swap  = 0kB
Jun 14 15:11:51 betanet kernel: [1903749.357745] Total swap = 0kB
Jun 14 15:11:51 betanet kernel: [1903749.357745] 511866 pages RAM
Jun 14 15:11:51 betanet kernel: [1903749.357746] 0 pages HighMem/MovableOnly
Jun 14 15:11:51 betanet kernel: [1903749.357746] 18372 pages reserved
Jun 14 15:11:51 betanet kernel: [1903749.357747] 0 pages hwpoisoned
Jun 14 15:11:51 betanet kernel: [1903749.357747] Tasks state (memory values in pages):
Jun 14 15:11:51 betanet kernel: [1903749.357748] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Jun 14 15:11:51 betanet kernel: [1903749.357752] [    435]     0   435    72337     6775   110592        0         -1000 multipathd
Jun 14 15:11:51 betanet kernel: [1903749.357755] [    738]   103   738     2221      804    57344        0          -900 dbus-daemon
Jun 14 15:11:51 betanet kernel: [1903749.357757] [    748]     0   748     8158     2662   110592        0             0 networkd-dispat
Jun 14 15:11:51 betanet kernel: [1903749.357759] [    752]     0   752     1723      458    53248        0             0 cron
Jun 14 15:11:51 betanet kernel: [1903749.357760] [    759]     0   759     3847      550    73728        0             0 systemd-logind
Jun 14 15:11:51 betanet kernel: [1903749.357762] [    765]     0   765      965      522    45056        0             0 atd
Jun 14 15:11:51 betanet kernel: [1903749.357763] [    793]     0   793     1449      166    49152        0             0 agetty
Jun 14 15:11:51 betanet kernel: [1903749.357765] [    795]     0   795    27436     2875   114688        0             0 unattended-upgr
Jun 14 15:11:51 betanet kernel: [1903749.357766] [    797]     0   797     1543      210    53248        0             0 agetty
Jun 14 15:11:51 betanet kernel: [1903749.357768] [   7193]   107  7193    55600      826    81920        0             0 rsyslogd
Jun 14 15:11:51 betanet kernel: [1903749.357770] [  11659]     0 11659    20699      691    61440        0             0 irqbalance
Jun 14 15:11:51 betanet kernel: [1903749.357771] [  11661]     0 11661    73909      891   159744        0             0 packagekitd
Jun 14 15:11:51 betanet kernel: [1903749.357773] [  11665]     0 11665    74406      746   577536        0          -250 systemd-journal
Jun 14 15:11:51 betanet kernel: [1903749.357775] [  11667]     0 11667    58622      451    86016        0             0 polkitd
Jun 14 15:11:51 betanet kernel: [1903749.357776] [  11678]   104 11678    22339      640    69632        0             0 systemd-timesyn
Jun 14 15:11:51 betanet kernel: [1903749.357777] [  11679]     0 11679     5562      477    61440        0         -1000 systemd-udevd
Jun 14 15:11:51 betanet kernel: [1903749.357779] [  11680]     0 11680    98116     1061   131072        0             0 udisksd
Jun 14 15:11:51 betanet kernel: [1903749.357780] [  11687]   101 11687     4030      529    69632        0             0 systemd-network
Jun 14 15:11:51 betanet kernel: [1903749.357782] [  11691]   102 11691     6317     1398    81920        0             0 systemd-resolve
Jun 14 15:11:51 betanet kernel: [1903749.357783] [  21044]     0 21044     3857     1081    61440        0         -1000 sshd
Jun 14 15:11:51 betanet kernel: [1903749.357785] [ 100286]  1000 100286     4258      972    73728        0             0 systemd
Jun 14 15:11:51 betanet kernel: [1903749.357786] [ 100287]  1000 100287    42553     1189   102400        0             0 (sd-pam)
Jun 14 15:11:51 betanet kernel: [1903749.357788] [ 100898]  1000 100898   227200    10107  1155072        0             0 PM2 v5.3.0: God
Jun 14 15:11:51 betanet kernel: [1903749.357789] [ 100948]  1000 100948   234828    14700  1556480        0             0 node /home/lisk
Jun 14 15:11:51 betanet kernel: [1903749.357791] [ 650875]  1000 650875     1842      512    53248        0             0 bash
Jun 14 15:11:51 betanet kernel: [1903749.357792] [ 650882]  1000 650882   616433   405762  5931008        0             0 node
Jun 14 15:11:51 betanet kernel: [1903749.357794] [ 652005]     0 652005     4228     1236    77824        0             0 sshd
Jun 14 15:11:51 betanet kernel: [1903749.357795] [ 652084]  1000 652084     4301      877    81920        0             0 sshd
Jun 14 15:11:51 betanet kernel: [1903749.357796] [ 652085]  1000 652085     3106     1812    61440        0             0 bash
Jun 14 15:11:51 betanet kernel: [1903749.357798] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-198.scope,task=node,pid=650882,uid=1000Jun 14 15:11:51 betanet kernel: [1903749.357854] Out of memory: Killed process 650882 (node) total-vm:2465732kB, anon-rss:1623048kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:5792kB oom_score_adj:0
Jun 14 15:11:51 betanet systemd[1]: session-198.scope: A process of this unit has been killed by the OOM killer.
+1
punkrock on Jun 14, 2023

@punkrock re-enabling generator will fix the issue seems bit strange , but it looks like somehow your node is disconnected from the network. It seems like somehow the node is getting banned or not being able to connect for syncing 🤔

I’m investigating further

+1
shuse2 on Jun 14, 2023

@sameersubudhi

I am failing to generate blocks again. Here the error logs (trace):

2023-06-12T03:42:11.199Z ERROR betanet engine 573312 [err=Event response for 'rpc-request' timed out trace=TimeoutError: Event response for 'rpc-request' timed out
    at SCClientSocket._handleEventAckTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:678:17)
    at Timeout._onTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:708:10)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-12T03:42:11.355Z ERROR betanet engine 573312 [err=Event response for 'rpc-request' timed out trace=TimeoutError: Event response for 'rpc-request' timed out
    at SCClientSocket._handleEventAckTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:678:17)
    at Timeout._onTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:708:10)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-12T03:42:11.535Z ERROR betanet engine 573312 [err=Event response for 'rpc-request' timed out trace=TimeoutError: Event response for 'rpc-request' timed out
    at SCClientSocket._handleEventAckTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:678:17)
    at Timeout._onTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:708:10)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-12T03:42:12.090Z ERROR betanet engine 573312 [err=Event response for 'rpc-request' timed out trace=TimeoutError: Event response for 'rpc-request' timed out
    at SCClientSocket._handleEventAckTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:678:17)
    at Timeout._onTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:708:10)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-12T03:42:12.815Z ERROR betanet engine 573312 [err=Event response for 'rpc-request' timed out trace=TimeoutError: Event response for 'rpc-request' timed out
    at SCClientSocket._handleEventAckTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:678:17)
    at Timeout._onTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:708:10)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-12T03:42:12.822Z ERROR betanet engine 573312 [err=Event response for 'rpc-request' timed out trace=TimeoutError: Event response for 'rpc-request' timed out
    at SCClientSocket._handleEventAckTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:678:17)
    at Timeout._onTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:708:10)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-12T03:42:12.822Z ERROR betanet engine 573312 [err=Event response for 'rpc-request' timed out trace=TimeoutError: Event response for 'rpc-request' timed out
    at SCClientSocket._handleEventAckTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:678:17)
    at Timeout._onTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:708:10)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-12T03:42:12.822Z ERROR betanet engine 573312 [err=Event response for 'rpc-request' timed out trace=TimeoutError: Event response for 'rpc-request' timed out
    at SCClientSocket._handleEventAckTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:678:17)
    at Timeout._onTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:708:10)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-12T03:42:12.823Z ERROR betanet engine 573312 [err=Event response for 'rpc-request' timed out trace=TimeoutError: Event response for 'rpc-request' timed out
    at SCClientSocket._handleEventAckTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:678:17)
    at Timeout._onTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:708:10)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-12T03:42:30.781Z ERROR betanet engine 573312 [peerId=45.32.203.143:7667 err=Apply penalty and abort synchronization mechanism with reason: Peer didn't return a common block trace=ApplyPenaltyAndAbortError: Apply penalty and abort synchronization mechanism with reason: Peer didn't return a common block
    at new SynchronizerError (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/synchronizer/errors.js:5:1)
    at new ApplyPenaltyAndAbortError (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/synchronizer/errors.js:35:9)
    at FastChainSwitchingMechanism._queryBlocks (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/synchronizer/fast_chain_switching_mechanism.js:61:19)
    at FastChainSwitchingMechanism.run (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/synchronizer/fast_chain_switching_mechanism.js:15:35)
    at runMicrotasks (<anonymous>)
    at runNextTicks (node:internal/process/task_queues:61:5)
    at processTimers (node:internal/timers:499:9)
    at async /home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/synchronizer/synchronizer.js:47:13
    at async Mutex.runExclusive (/home/lisk/lisk-core/node_modules/@liskhq/lisk-utils/dist-node/job_handlers/mutex.js:23:20)
    at async Synchronizer.run (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/synchronizer/synchronizer.js:33:9)] Fail to handle received block
2023-06-12T03:43:18.448Z ERROR betanet engine 573312 [err=Event response for 'rpc-request' timed out trace=TimeoutError: Event response for 'rpc-request' timed out
    at SCClientSocket._handleEventAckTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:678:17)
    at Timeout._onTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:708:10)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-12T03:43:18.771Z ERROR betanet engine 573312 [err=Event response for 'rpc-request' timed out trace=TimeoutError: Event response for 'rpc-request' timed out
    at SCClientSocket._handleEventAckTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:678:17)
    at Timeout._onTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:708:10)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-12T03:43:19.134Z ERROR betanet engine 573312 [err=Event response for 'rpc-request' timed out trace=TimeoutError: Event response for 'rpc-request' timed out
    at SCClientSocket._handleEventAckTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:678:17)
    at Timeout._onTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:708:10)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-12T03:43:19.258Z ERROR betanet engine 573312 [err=Event response for 'rpc-request' timed out trace=TimeoutError: Event response for 'rpc-request' timed out
    at SCClientSocket._handleEventAckTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:678:17)
    at Timeout._onTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:708:10)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-12T03:43:19.377Z ERROR betanet engine 573312 [err=Event response for 'rpc-request' timed out trace=TimeoutError: Event response for 'rpc-request' timed out
    at SCClientSocket._handleEventAckTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:678:17)
    at Timeout._onTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:708:10)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-12T03:43:19.426Z ERROR betanet engine 573312 [err=Event response for 'rpc-request' timed out trace=TimeoutError: Event response for 'rpc-request' timed out
    at SCClientSocket._handleEventAckTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:678:17)
    at Timeout._onTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:708:10)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-12T03:43:19.606Z ERROR betanet engine 573312 [err=Event response for 'rpc-request' timed out trace=TimeoutError: Event response for 'rpc-request' timed out
    at SCClientSocket._handleEventAckTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:678:17)
    at Timeout._onTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:708:10)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-12T08:34:22.246Z ERROR betanet engine 573312 [peerId=80.211.194.139:7667 err=List of connected peers is empty trace=Error: List of connected peers is empty
    at BlockSynchronizationMechanism._computeBestPeer (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/synchronizer/block_synchronization_mechanism.js:231:19)
    at BlockSynchronizationMechanism.run (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/synchronizer/block_synchronization_mechanism.js:28:31)
    at /home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/synchronizer/synchronizer.js:47:34
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Mutex.runExclusive (/home/lisk/lisk-core/node_modules/@liskhq/lisk-utils/dist-node/job_handlers/mutex.js:23:20)
    at async Synchronizer.run (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/synchronizer/synchronizer.js:33:9)
    at async Consensus._sync (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/consensus.js:584:13)
    at async /home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/consensus.js:310:17
    at async Mutex.runExclusive (/home/lisk/lisk-core/node_modules/@liskhq/lisk-utils/dist-node/job_handlers/mutex.js:23:20)] Fail to handle received block
2023-06-12T09:04:41.609Z ERROR betanet engine 573312 [err=Event response for 'rpc-request' timed out trace=TimeoutError: Event response for 'rpc-request' timed out
    at SCClientSocket._handleEventAckTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:678:17)
    at Timeout._onTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:708:10)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-12T09:07:41.608Z ERROR betanet engine 573312 [err=Event response for 'rpc-request' timed out trace=TimeoutError: Event response for 'rpc-request' timed out
    at SCClientSocket._handleEventAckTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:678:17)
    at Timeout._onTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:708:10)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
/home/lisk/lisk-core/bin/lisk-core: line 54: 573312 Killed                  "$NODE" "$DIR/run" "$@"
2023-06-12T13:31:50.548Z ERROR betanet engine 581650 [err=Incoming transaction fee is not sufficient to replace existing transaction trace=TransactionPoolError: Incoming transaction fee is not sufficient to replace existing transaction
    at TransactionPool.add (/home/lisk/lisk-core/node_modules/@liskhq/lisk-transaction-pool/dist-node/transaction_pool.js:127:24)
    at async NetworkEndpoint._receiveTransaction (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/generator/network_endpoint.js:162:27)
    at async NetworkEndpoint.handleEventPostTransactionsAnnouncement (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/generator/network_endpoint.js:132:21)] Failed to add transaction to pool.
2023-06-12T13:31:51.151Z ERROR betanet engine 581650 [err=Incoming transaction fee is not sufficient to replace existing transaction trace=TransactionPoolError: Incoming transaction fee is not sufficient to replace existing transaction
    at TransactionPool.add (/home/lisk/lisk-core/node_modules/@liskhq/lisk-transaction-pool/dist-node/transaction_pool.js:127:24)
    at async NetworkEndpoint._receiveTransaction (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/generator/network_endpoint.js:162:27)
    at async NetworkEndpoint.handleEventPostTransactionsAnnouncement (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/generator/network_endpoint.js:132:21)] Failed to add transaction to pool.
2023-06-12T13:34:23.619Z ERROR betanet engine 581650 [err=Incoming transaction fee is not sufficient to replace existing transaction trace=TransactionPoolError: Incoming transaction fee is not sufficient to replace existing transaction
    at TransactionPool.add (/home/lisk/lisk-core/node_modules/@liskhq/lisk-transaction-pool/dist-node/transaction_pool.js:127:24)
    at async NetworkEndpoint._receiveTransaction (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/generator/network_endpoint.js:162:27)
    at async NetworkEndpoint.handleEventPostTransactionsAnnouncement (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/generator/network_endpoint.js:132:21)] Failed to add transaction to pool.
2023-06-12T13:35:38.604Z ERROR betanet engine 581650 [err=Incoming transaction fee is not sufficient to replace existing transaction trace=TransactionPoolError: Incoming transaction fee is not sufficient to replace existing transaction
    at TransactionPool.add (/home/lisk/lisk-core/node_modules/@liskhq/lisk-transaction-pool/dist-node/transaction_pool.js:127:24)
    at async NetworkEndpoint._receiveTransaction (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/generator/network_endpoint.js:162:27)
    at async NetworkEndpoint.handleEventPostTransactionsAnnouncement (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/generator/network_endpoint.js:132:21)] Failed to add transaction to pool.
/home/lisk/lisk-core/bin/lisk-core: line 54: 581650 Killed                  "$NODE" "$DIR/run" "$@"
/home/lisk/lisk-core/bin/lisk-core: line 54: 586594 Killed                  "$NODE" "$DIR/run" "$@"
+1
punkrock on Jun 12, 2023

@punkrock Could you please run the node with trace logs and share them when this happens again?

+1
sameersubudhi on Jun 9, 2023

@sameersubudhi Ubuntu 22.04.2 LTS

+1
punkrock on Jun 8, 2023
Read more comments on GitHub
← lisk-sdk: error: Forever detected script was killed by signal: SIGKILL
lisk-sdk: Inconsistency in migrations →