vespa: Cannot start Vespa after Upgrade

After upgrading to the version 7.594.36 Vespa does not start

To Reproduce yum update vespa

Screenshots

[2022-06-23 16:36:50] container-clustercontroller Cluster 'stack': Storing new cluster state version in ZooKeeper: 12
[2022-06-23 16:36:50] container-clustercontroller Cluster 'stack': Master moratorium complete: timed out waiting for all nodes to report in
[2022-06-23 16:36:50] container-clustercontroller Cluster 'stack': Publishing cluster state version 12
[2022-06-23 16:36:50] distributor      Distributor node ready. Done initializing. Giving out of sequence metric event. Config id is stack/distributor/0
[2022-06-23 16:36:50] distributor      Transitioning from baseline state 'Down' to 'Up' (cluster state version 12)
[2022-06-23 16:36:50] container-clustercontroller Cluster event type SYSTEMSTATE @1656002210111: Reset 1 start timestamps as all available distributors have seen newest cluster state.
[2022-06-23 16:38:36] configserver     Session 58 prepared successfully.
[2022-06-23 16:38:37] configserver     Session activated: 58
[2022-06-23 16:38:37] configserver     Session 58 activated successfully using no host provisioner. Config generation 58. Based on session 57. File references: [file '7578094d437997ed']
[2022-06-23 16:38:37] searchnode       Filedistributorrpc config is changed
[2022-06-23 16:38:37] container-clustercontroller Switching to the latest deployed set of configurations and components. Application config generation: 58
[2022-06-23 16:38:37] container-clustercontroller Starting deconstruction of 2 components and 0 bundles from generation 57
[2022-06-23 16:38:38] container-clustercontroller Completed deconstruction in 0.030 seconds
[2022-06-23 16:38:38] searchnode       Add document database: doctypename(file), configid(stack/search/cluster.stack/file)
[2022-06-23 16:38:38] container        Switching to the latest deployed set of configurations and components. Application config generation: 58
[2022-06-23 16:38:38] container        Starting deconstruction of 2 components and 0 bundles from generation 57
[2022-06-23 16:38:38] container        Completed deconstruction in 0.013 seconds
[2022-06-23 16:38:38] searchnode       Add document database: doctypename(filter), configid(stack/search/cluster.stack/filter)
[2022-06-23 16:38:39] searchnode       vespa-proton-bin: /builddir/build/BUILD/vespa-7.594.36/searchcore/src/vespa/searchcore/proton/server/documentdb.cpp:433: void proton::DocumentDB::applySubDBConfig(const proton::DocumentDBConfig&, proton::DocumentDB::SerialNum, const proton::ReconfigParams&): Assertion `newDocType != nullptr' failed.
[2022-06-23 16:38:39] metricsproxy-container Switching to the latest deployed set of configurations and components. Application config generation: 58
[2022-06-23 16:38:39] metricsproxy-container Starting deconstruction of 2 components and 0 bundles from generation 57
[2022-06-23 16:38:39] metricsproxy-container Completed deconstruction in 0.002 seconds
[2022-06-23 16:38:40] config-sentinel  searchnode: incremented restart penalty to 254.000 seconds
[2022-06-23 16:38:40] config-sentinel  searchnode: incremented restart penalty to 510.000 seconds
[2022-06-23 16:38:40] config-sentinel  searchnode: will delay start by 506.543 seconds
[2022-06-23 16:47:06] searchnode       Filedistributorrpc config is changed
[2022-06-23 16:47:07] searchnode       Add document database: doctypename(file), configid(stack/search/cluster.stack/file)
[2022-06-23 16:47:07] searchnode       Add document database: doctypename(filter), configid(stack/search/cluster.stack/filter)
[2022-06-23 16:47:07] searchnode       vespa-proton-bin: /builddir/build/BUILD/vespa-7.594.36/searchcore/src/vespa/searchcore/proton/server/documentdb.cpp:433: void proton::DocumentDB::applySubDBConfig(const proton::DocumentDBConfig&, proton::DocumentDB::SerialNum, const proton::ReconfigParams&): Assertion `newDocType != nullptr' failed.
[2022-06-23 16:47:08] config-sentinel  searchnode: incremented restart penalty to 1022.000 seconds
[2022-06-23 16:47:08] config-sentinel  searchnode: incremented restart penalty to 1800.000 seconds
[2022-06-23 16:47:08] config-sentinel  searchnode: will delay start by 1798.416 seconds

Environment (please complete the following information):

  • OS: CentOS
  • AWS
  • Versions Stream 8

Vespa version 7.594.36

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 17 (9 by maintainers)

Commits related to this issue

Most upvoted comments

That worked. Thanks a million for the help. I think itโ€™s time to switch to a hosted version ๐Ÿ˜ƒ