jormungandr: Multiple "cannot process network headers, reason: deadline has elapsed, task: block" errors

Describe the bug Multiple errors

cannot process network headers, reason: deadline has elapsed, task: block

happening every minute/few minutes.

Example log output

--
Dec 20 11:16:54.665 INFO disconnecting client, node_id: 51cd865faa86387069afd6539137e7da4dc051b1d8a32f70, peer_addr: 77.168.127.192:3000, task: network
Dec 20 11:16:54.684 INFO failed to connect to peer, reason: Connection refused (os error 111), node_id: ff738164013735cc5cfa540644e90e7fc4cb8f3a69f8592e, peer_addr: 207.237.205.137:3100, task: network
Dec 20 11:16:54.887 ERRO cannot process network headers, reason: deadline has elapsed, task: block
Dec 20 11:16:54.894 INFO peer node ID differs from the expected a538c7ec3143d95e81d80e53b06cfcbaa99040db7d38f49f, node_id: 9776b8eccc91245744d039d3a82e97c6651162421722894e, peer_addr: 62.151.178.94:3000, task: network
Dec 20 11:16:54.963 INFO connecting to peer, node_id: 04f1782b84e0fda00143db199487fbaff8fdd7c23d1091a0, peer_addr: 173.249.43.58:3000, task: network
--
Dec 20 11:18:18.086 INFO disconnecting client, node_id: 72b5a4086612d12c063d77c743765485629a542d010a9006, peer_addr: 54.154.193.39:3000, task: network
Dec 20 11:18:18.220 INFO disconnecting client, node_id: 234588b3165ec62391137ceaeca3dc0910c5620a9cbd30fa, peer_addr: 50.39.165.123:3001, task: network
Dec 20 11:18:18.248 ERRO cannot process network headers, reason: deadline has elapsed, task: block
Dec 20 11:18:18.250 ERRO cannot process network headers, reason: deadline has elapsed, task: block
Dec 20 11:18:18.597 INFO received block announcement, hash: 23964dc4dfd6012a4747eccacfbddbf43fb8470d82a491a6bf0ac42f87f5a9a9, direction: in, stream: block_events, node_id: 408a43baf4c562338d15b9f72adb949b5c6cfc1ff8115744, peer_addr: 89.162.89.137:9009, task: network
Dec 20 11:18:18.597 INFO received block announcement from network, from_node_id: 408a43baf4c562338d15b9f72adb949b5c6cfc1ff8115744, date: 6.27946, parent: 1d42266650122f632fca534af3fa4bb4e5ec3a99cd1f81c60f18edf8685cf00d, hash: 23964dc4dfd6012a4747eccacfbddbf43fb8470d82a491a6bf0ac42f87f5a9a9, task: block
Dec 20 11:18:18.598 INFO receiving header stream from network, task: block
Dec 20 11:18:18.701 INFO failed to connect to peer, reason: No route to host (os error 113), node_id: b51dfd2fb84f2e477d693e166c688871a17b4e0d9c65a944, peer_addr: 46.183.114.176:3000, task: network
Dec 20 11:18:18.747 ERRO cannot process network headers, reason: deadline has elapsed, task: block
Dec 20 11:18:18.753 INFO peer node ID differs from the expected 2fd15a308ccc65e7a062eb68130181d2861e8af1515db882, node_id: 05b14771571b24e2d87e89649b8e5ed8cd9e5cca311a8435, peer_addr: 138.68.62.183:3000, task: network
Dec 20 11:18:18.764 INFO failed to connect to peer, reason: Connection refused (os error 111), node_id: a9427741277d0f6b200ff9fc5d5201cc1dd7e929aec6d377, peer_addr: 207.237.205.137:3100, task: network
--
Dec 20 11:20:18.191 INFO disconnecting client, node_id: 2a7a163ea3bcc2455b42d701d1818086513ce88f7acc645a, peer_addr: 178.128.110.133:3000, task: network
Dec 20 11:20:18.256 INFO peer node ID differs from the expected 2b3ef5b7134bb6f91c5ae02244122d68d1eee560ce100d61, node_id: f1da0885b10b0f73f7fc505b4839736ec0135039512fb1f1, peer_addr: 98.151.82.186:3000, task: network
Dec 20 11:20:18.607 ERRO cannot process network headers, reason: deadline has elapsed, task: block
Dec 20 11:20:19.234 INFO receiving block stream from network, task: block
Dec 20 11:20:19.443 INFO response stream failed, reason: not found, request: GetBlocks, node_id: f1da0885b10b0f73f7fc505b4839736ec0135039512fb1f1, peer_addr: 98.151.82.186:3000, task: network
--
Dec 20 11:23:08.861 INFO disconnecting client, node_id: f7948760c892c427ab65fd44d1da4084cbba28b3b0b4bd0f, peer_addr: 49.193.36.88:3000, task: network
Dec 20 11:23:08.879 INFO disconnecting client, node_id: 51433919d76c9d02bc80c0707a7a2503175e925854c31f64, peer_addr: 165.22.119.72:3000, task: network
Dec 20 11:23:11.015 ERRO cannot process network headers, reason: deadline has elapsed, task: block
Dec 20 11:23:12.367 INFO response stream failed, reason: the operation was aborted, request: PullHeaders, node_id: 22fb117f9f72f38b21bca5c0f069766c0d4327925d967791, peer_addr: 3.125.75.156:3000, task: network
Dec 20 11:23:13.809 INFO disconnecting client, node_id: 85941f4cd44f7bc4c58bc6fbd1538b48e27ed4b6c436bb5c, peer_addr: 95.179.219.33:3000, task: network
--
Dec 20 11:23:14.695 INFO disconnecting client, node_id: fe4967cfd001f1fe670d1be53d511978236dc2c3c7022b4a, peer_addr: 78.46.191.161:3000, task: network
Dec 20 11:23:15.662 INFO disconnecting client, node_id: b23dedf6ab2acb194d5d4e73bcfd88783ab051cf5aca0972, peer_addr: 89.162.89.137:9003, task: network
Dec 20 11:23:16.451 ERRO cannot process network headers, reason: deadline has elapsed, task: block
Dec 20 11:23:17.119 INFO response stream failed, reason: the operation was aborted, request: PullHeaders, node_id: 22fb117f9f72f38b21bca5c0f069766c0d4327925d967791, peer_addr: 3.125.75.156:3000, task: network
Dec 20 11:23:17.471 INFO failed to connect to peer, reason: Connection timed out (os error 110), node_id: e977c1693cd69c7cb3cdc84eb19f78723ea125ae97e6afa5, peer_addr: 14.203.162.172:3000, task: network
--
Dec 20 11:23:50.535 INFO disconnecting client, node_id: fe72da28b9fbac02297bf4a3ccddbd44142f8ce7b7ab9e2a, peer_addr: 14.200.26.39:3100, task: network
Dec 20 11:23:50.557 INFO disconnecting client, node_id: e2b74215830776151436879874d25ca3c9703cc1af8cca1c, peer_addr: 45.76.126.160:3000, task: network
Dec 20 11:23:51.801 ERRO cannot process network headers, reason: deadline has elapsed, task: block
Dec 20 11:23:52.667 INFO response stream failed, reason: the operation was aborted, request: PullHeaders, node_id: 22fb117f9f72f38b21bca5c0f069766c0d4327925d967791, peer_addr: 3.125.75.156:3000, task: network
Dec 20 11:23:52.782 INFO peer node ID differs from the expected 38060b25c67f21b587891b1ee5c57e017c0390e0660dc9e3, node_id: 83b6695a594a3d17b3d95d352e4b951cd76f694cba0720e3, peer_addr: 45.77.154.167:3000, task: network
--
Dec 20 11:24:34.288 INFO disconnecting client, node_id: 3e2d6ec5c486a004a0f1dec355e952992e5165c59dbf388f, peer_addr: 45.153.186.221:3100, task: network
Dec 20 11:24:35.239 INFO disconnecting client, node_id: 1690dcfa213a0ee91ee9b008b7d16938ab82ca742ad57371, peer_addr: 89.162.89.137:9004, task: network
Dec 20 11:24:35.948 ERRO cannot process network headers, reason: deadline has elapsed, task: block
Dec 20 11:24:36.452 INFO response stream failed, reason: the operation was aborted, request: PullHeaders, node_id: 22fb117f9f72f38b21bca5c0f069766c0d4327925d967791, peer_addr: 3.125.75.156:3000, task: network
Dec 20 11:24:37.592 INFO connecting to peer, node_id: 7af309f8a6aa3fcd1fbb1dc478b5ec0a4812af70cbf9b876, peer_addr: 3.226.160.142:3000, task: network

Mandatory Information

1. jcli --full-version output; - jcli 0.8.4 (HEAD-2227eb2, release, linux [x86_64]) - [rustc 1.40.0 (73528e339 2019-12-16)]
2. jormungandr --full-version output; - jormungandr 0.8.4 (HEAD-2227eb2, release, linux [x86_64]) - [rustc 1.40.0 (73528e339 2019-12-16)]

To Reproduce Steps to reproduce the behavior:

1. Config:

log:
  - output: stderr
    level: info
    format: plain
explorer:
  enabled: true
p2p:
  topics_of_interest:
    blocks: high
    messages: high
  trusted_peers:
    - address: "/ip4/52.9.132.248/tcp/3000"
      id: 671a9e7a5c739532668511bea823f0f5c5557c99b813456c
    - address: "/ip4/52.8.15.52/tcp/3000"
      id: 18bf81a75e5b15a49b843a66f61602e14d4261fb5595b5f5
    - address: "/ip4/13.114.196.228/tcp/3000"
      id: 7e1020c2e2107a849a8353876d047085f475c9bc646e42e9
    - address: "/ip4/13.112.181.42/tcp/3000"
      id: 52762c49a84699d43c96fdfe6de18079fb2512077d6aa5bc
    - address: "/ip4/3.125.75.156/tcp/3000"
      id: 22fb117f9f72f38b21bca5c0f069766c0d4327925d967791
    - address: "/ip4/52.28.91.178/tcp/3000"
      id: 23b3ca09c644fe8098f64c24d75d9f79c8e058642e63a28c
    - address: "/ip4/3.124.116.145/tcp/3000"
      id: 99cb10f53185fbef110472d45a36082905ee12df8a049b74
rest:
  cors:
    allowed_origins:
    - http://localhost:8000
    - http://127.0.0.1:8443
  listen: 127.0.0.1:8443
storage: "/home/lewis/jormungandr/storage/"

Expected behavior To see less errors.

Additional context

• I wasn’t seeing any of these errors on 0.8.3.
• The node was started on 0.8.4 at 2019-12-19 21:22:36.288 UTC
• The errors first started to appear at Dec 20 07:26:02.310 UTC
• Additional warning started to appear at Dec 19 22:31:47.590 UTC WARN blockchain is not moving up, the last block was 1834 seconds ago, task: stuck_notifier