solana: Dropped vote from mismatch hash

@atomlab It looks like there is still some dropped vote errors coming from your validator, perhaps?

Can you check your metrics chart for dropped votes and recent log files for messages like this?:

Jul 22 00:31:08 sol-testnet-01 solana-validator[7429]: [2020-07-22T00:31:08.183230894Z WARN solana_runtime::native_loader] Failed to load: DlOpen { desc: "/usr/local/bin/deps/libsolana_bpf_loader_program.so: cannot open shared object file: No such file or directory" }

Last five days my node have worked without any problem with delinquent and restarting (only once when my node was updating to 1.2.13 ).

Last two days grep logs. I don’t see any errors connected with deps libs.

root@sol-testnet-01:~# grep libsolana /var/log/syslog*
root@sol-testnet-01:~#
root@sol-testnet-01:~# grep "Failed to load" /var/log/syslog*
root@sol-testnet-01:~#
root@sol-testnet-01:~# grep "dropped vote Vote"  /var/log/syslog* | wc -l
354
root@sol-testnet-01:~#

I can share whole logs for last two or three days.

Files and libs

root@sol-testnet-01:~# tree /usr/local/bin/
/usr/local/bin/
├── deps
│   ├── libsolana_bpf_loader_program.rlib
│   ├── libsolana_bpf_loader_program.so
│   ├── libsolana_btc_spv_program.rlib
│   ├── libsolana_btc_spv_program.so
│   ├── libsolana_budget_program.rlib
│   ├── libsolana_budget_program.so
│   ├── libsolana_config_program.rlib
│   ├── libsolana_config_program.so
│   ├── libsolana_exchange_program.rlib
│   ├── libsolana_exchange_program.so
│   ├── libsolana_failure_program.rlib
│   ├── libsolana_failure_program.so
│   ├── libsolana_noop_program.rlib
│   ├── libsolana_noop_program.so
│   ├── libsolana_stake_program.rlib
│   ├── libsolana_stake_program.so
│   ├── libsolana_storage_program.rlib
│   ├── libsolana_storage_program.so
│   ├── libsolana_vote_program.rlib
│   └── libsolana_vote_program.so
├── loki-canary
├── node_exporter
├── promtail
├── solana
├── solana-bench-exchange
├── solana-bench-tps
├── solana-dos
├── solana_exporter
├── solana-faucet
├── solana-genesis
├── solana-gossip
├── solana-install
├── solana-install-init
├── solana-keygen
├── solana-ledger-tool
├── solana-log-analyzer
├── solana-net-shaper
├── solana-stake-accounts
├── solana-stake-monitor
├── solana-stake-o-matic
├── solana-sys-tuner
├── solana-tokens
├── solana-validator
└── solana-watchtower

1 directory, 45 files

root@sol-testnet-01:/opt# ldd /usr/local/bin/solana-validator
	linux-vdso.so.1 (0x00007ffda88bd000)
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007ffb43b1c000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ffb43918000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ffb436f9000)
	libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007ffb4346c000)
	libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007ffb42fa1000)
	libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007ffb42d83000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ffb42992000)
	/lib64/ld-linux-x86-64.so.2 (0x00007ffb45762000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ffb4278a000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ffb42572000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ffb421d4000)

Last three days

Hey @ryoqun, can you please investigate this a little.