weaviate: panic: runtime error: slice bounds out of range [:752185] with capacity 186278
Dear team
we updated to v1.19.8 and now we are having a runtime error similar to a previous issue #1897 that was fixed before. Can you please take a look at this?
Instance running in Azure, mounted file share.
Thanks Ahmed
`{“action”:“startup”,“level”:“debug”,“msg”:“created startup context, nothing done so far”,“startup_time_left”:“59m59.998647187s”,“time”:“2023-06-18T13:38:59Z”} {“action”:“startup”,“default_vectorizer_module”:“text2vec-openai”,“level”:“info”,“msg”:“the default vectorizer modules is set to "text2vec-openai", as a result all new schema classes without an explicit vectorizer setting, will use this vectorizer”,“time”:“2023-06-18T13:38:59Z”} {“action”:“startup”,“auto_schema_enabled”:true,“level”:“info”,“msg”:“auto schema enabled setting is set to "true"”,“time”:“2023-06-18T13:38:59Z”} {“action”:“startup”,“level”:“debug”,“msg”:“config loaded”,“startup_time_left”:“59m59.996590867s”,“time”:“2023-06-18T13:38:59Z”} {“action”:“startup”,“level”:“debug”,“msg”:“configured OIDC and anonymous access client”,“startup_time_left”:“59m59.996084962s”,“time”:“2023-06-18T13:38:59Z”} {“action”:“startup”,“level”:“debug”,“msg”:“initialized schema”,“startup_time_left”:“59m59.996023161s”,“time”:“2023-06-18T13:38:59Z”} {“action”:“startup”,“level”:“debug”,“msg”:“startup routine complete”,“time”:“2023-06-18T13:38:59Z”} {“action”:“startup”,“level”:“debug”,“msg”:“start registering modules”,“time”:“2023-06-18T13:38:59Z”} {“action”:“startup”,“level”:“debug”,“module”:“text2vec-openai”,“msg”:“enabled module”,“time”:“2023-06-18T13:38:59Z”} {“action”:“startup”,“level”:“debug”,“msg”:“completed registering modules”,“time”:“2023-06-18T13:38:59Z”} {“action”:“startup_cluster_schema_sync”,“level”:“debug”,“msg”:“Only node in the cluster at this point. No schema sync necessary.”,“time”:“2023-06-18T13:38:59Z”} {“action”:“cluster_api_startup”,“level”:“debug”,“msg”:“serving cluster api on port 7947”,“port”:7947,“time”:“2023-06-18T13:38:59Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/objects/segment-1687095509813461467.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:38:59Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/objects/segment-1687095509813461467.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:38:59Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_embedding_position/segment-1687095509929784915.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:38:59Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_url/segment-1687095509927246990.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:38:59Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property__id/segment-1687095509927166389.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:38:59Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_hierarchy_3/segment-1687095509927682894.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:38:59Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_hierarchy_4/segment-1687095509927831495.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:38:59Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_hierarchy_1/segment-1687095509927390891.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:38:59Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_hierarchy_2/segment-1687095509927051588.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:38:59Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_following_heading/segment-1687095509927095588.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:38:59Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_embedding_position/segment-1687095509929784915.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:38:59Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property__id/segment-1687095509927166389.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:38:59Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_url/segment-1687095509927246990.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:38:59Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_hierarchy_5/segment-1687095509928584303.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:38:59Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_hierarchy_3/segment-1687095509927682894.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:38:59Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_hierarchy_1/segment-1687095509927390891.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:38:59Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_hierarchy_4/segment-1687095509927831495.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:38:59Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_hierarchy_2/segment-1687095509927051588.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:38:59Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_following_heading/segment-1687095509927095588.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:38:59Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_text/segment-1687095509927390091.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:38:59Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_hierarchy_5/segment-1687095509928584303.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_text/segment-1687095509927390091.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_word_count/segment-1687095509928571103.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_included_tables/segment-1687095509926341681.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_word_count/segment-1687095509928571103.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_hierarchy_5_searchable/segment-1687095510174721432.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_hierarchy_4_searchable/segment-1687095510319438359.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_included_tables/segment-1687095509926341681.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_hierarchy_5_searchable/segment-1687095510174721432.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_hierarchy_4_searchable/segment-1687095510319438359.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_hierarchy_2_searchable/segment-1687095510155319540.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_hierarchy_3_searchable/segment-1687095510293821807.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_following_heading_searchable/segment-1687095510149380482.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_url_searchable/segment-1687095510197895160.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_hierarchy_2_searchable/segment-1687095510155319540.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_hierarchy_3_searchable/segment-1687095510293821807.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_following_heading_searchable/segment-1687095510149380482.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_text_searchable/segment-1687095510190636189.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_hierarchy_1_searchable/segment-1687095510193643618.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_url_searchable/segment-1687095510197895160.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_text_searchable/segment-1687095510190636189.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“RhB_v1”,“index”:“rhb_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_v1_5LlDN8n6lkAL_lsm/property_hierarchy_1_searchable/segment-1687095510193643618.wal”,“shard”:“5LlDN8n6lkAL”,“time”:“2023-06-18T13:39:00Z”} {“action”:“startup”,“level”:“debug”,“msg”:“shard=5LlDN8n6lkAL is ready”,“time”:“2023-06-18T13:39:00Z”} {“action”:“hnsw_vector_cache_prefill_level”,“count”:0,“hnsw_level”:0,“index_id”:“rhb_v1_5LlDN8n6lkAL”,“level”:“debug”,“msg”:“prefilled level in vector cache”,“time”:“2023-06-18T13:39:00Z”,“took”:129201} {“action”:“hnsw_vector_cache_prefill”,“count”:3000,“index_id”:“rhb_v1_5LlDN8n6lkAL”,“level”:“info”,“limit”:1000000000000,“msg”:“prefilled vector cache”,“time”:“2023-06-18T13:39:00Z”,“took”:516605} {“action”:“lsm_recover_from_active_wal”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/objects/segment-1687095510560160735.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/objects/segment-1687095510560160735.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_included_tables/segment-1687095407716312229.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_hierarchy_1/segment-1687095408333341178.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_word_count/segment-1687095408224397488.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_hierarchy_3/segment-1687095407075925598.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property__id/segment-1687095405195291749.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_hierarchy_4/segment-1687095408036418752.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_included_tables/segment-1687095407716312229.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_included_tables/segment-1687095484436953427.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_hierarchy_1/segment-1687095408333341178.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_hierarchy_1/segment-1687095484437440032.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_word_count/segment-1687095408224397488.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_word_count/segment-1687095484436447122.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_hierarchy_2/segment-1687095406852856358.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property__id/segment-1687095405195291749.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property__id/segment-1687095484436631524.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_hierarchy_3/segment-1687095407075925598.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_hierarchy_3/segment-1687095484439246449.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_hierarchy_4/segment-1687095408036418752.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_hierarchy_4/segment-1687095484436614424.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_embedding_position/segment-1687095407506238439.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_included_tables/segment-1687095484436953427.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_included_tables/segment-1687095490740984865.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_hierarchy_1/segment-1687095484437440032.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_hierarchy_1/segment-1687095490744365098.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_url/segment-1687095406659513504.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_following_heading/segment-1687095407827164343.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_hierarchy_3/segment-1687095484439246449.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_hierarchy_3/segment-1687095490741523470.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_included_tables/segment-1687095490740984865.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_included_tables/segment-1687095510676060178.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_hierarchy_4/segment-1687095484436614424.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_hierarchy_4/segment-1687095490740972265.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_hierarchy_3/segment-1687095490741523470.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_hierarchy_3/segment-1687095510678835906.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_included_tables/segment-1687095510676060178.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_hierarchy_1/segment-1687095490744365098.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_hierarchy_1/segment-1687095510678529403.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_hierarchy_4/segment-1687095490740972265.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_hierarchy_4/segment-1687095510683431851.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_hierarchy_3/segment-1687095510678835906.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_hierarchy_1/segment-1687095510678529403.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_url/segment-1687095406659513504.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“warning”,“msg”:“active write-ahead-log found. Did weaviate crash prior to this? Trying to recover…”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_url/segment-1687095484441622372.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} {“action”:“lsm_recover_from_active_wal_success”,“class”:“Rhb_playground_v1”,“index”:“rhb_playground_v1”,“level”:“info”,“msg”:“successfully recovered from write-ahead-log”,“path”:“/var/lib/weaviate/rhb_playground_v1_v9doXN1E0Yj2_lsm/property_hierarchy_4/segment-1687095510683431851.wal”,“shard”:“v9doXN1E0Yj2”,“time”:“2023-06-18T13:39:00Z”} panic: runtime error: slice bounds out of range [:752185] with capacity 186278
goroutine 1141 [running]: github.com/weaviate/weaviate/usecases/byte_operations.(*ByteOperations).CopyBytesFromBuffer(…) /go/src/github.com/weaviate/weaviate/usecases/byte_operations/byte_operations.go:57 github.com/weaviate/weaviate/adapters/repos/db/lsmkv/segmentindex.(*DiskTree).readNode(0xffffffffffffff9c?, {0x7fd45c548942?, 0x40f67f?, 0x0?}) /go/src/github.com/weaviate/weaviate/adapters/repos/db/lsmkv/segmentindex/disk_tree.go:105 +0x49b github.com/weaviate/weaviate/adapters/repos/db/lsmkv/segmentindex.(*DiskTree).AllKeys(0xc0000ff578) /go/src/github.com/weaviate/weaviate/adapters/repos/db/lsmkv/segmentindex/disk_tree.go:179 +0x9f github.com/weaviate/weaviate/adapters/repos/db/lsmkv.(*segment).computeAndStoreBloomFilter(0xc0004b4dd0, {0xc0005a1180, 0x64}) /go/src/github.com/weaviate/weaviate/adapters/repos/db/lsmkv/segment_bloom_filters.go:72 +0x3b github.com/weaviate/weaviate/adapters/repos/db/lsmkv.(*segment).initBloomFilter(0xc0004b4dd0) /go/src/github.com/weaviate/weaviate/adapters/repos/db/lsmkv/segment_bloom_filters.go:59 +0xc5 github.com/weaviate/weaviate/adapters/repos/db/lsmkv.newSegment({0xc0005a01c0, 0x61}, {0x1bd2330?, 0xc00391e850}, 0x0, 0x0?) /go/src/github.com/weaviate/weaviate/adapters/repos/db/lsmkv/segment.go:136 +0xa34 github.com/weaviate/weaviate/adapters/repos/db/lsmkv.newSegmentGroup({0xc003078370, 0x42}, {0x1bd2330, 0xc00391e850}, 0x0, 0x0, {0x17cb737, 0xa}, 0x0, {0x1bc0d80, …}) /go/src/github.com/weaviate/weaviate/adapters/repos/db/lsmkv/segment_group.go:118 +0x8c5 github.com/weaviate/weaviate/adapters/repos/db/lsmkv.NewBucket({0x1bbe888, 0xc000044078}, {0xc003078370, 0x42}, {0xc0000461c6, 0x11}, {0x1bd2330?, 0xc00391e850?}, 0x0, {0x1bc0d80, …}, …) /go/src/github.com/weaviate/weaviate/adapters/repos/db/lsmkv/bucket.go:119 +0x2e5 github.com/weaviate/weaviate/adapters/repos/db/lsmkv.(*Store).CreateOrLoadBucket(0xc003168500, {0x1bbe888, 0xc000044078}, {0xc00315ac80, 0xd}, {0xc0005be180, 0x3, 0x4}) /go/src/github.com/weaviate/weaviate/adapters/repos/db/lsmkv/store.go:124 +0x166 github.com/weaviate/weaviate/adapters/repos/db.(*Shard).createPropertyValueIndex(0xc0003823c0, {0x1bbe888, 0xc000044078}, 0xc00313a230) /go/src/github.com/weaviate/weaviate/adapters/repos/db/shard.go:472 +0x3d4 github.com/weaviate/weaviate/adapters/repos/db.(*Shard).createPropertyIndex.func1() /go/src/github.com/weaviate/weaviate/adapters/repos/db/shard.go:422 +0x58 golang.org/x/sync/errgroup.(*Group).Go.func1() /go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 +0x64 created by golang.org/x/sync/errgroup.(*Group).Go /go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:72 +0xa5 `
About this issue
- Original URL
- State: open
- Created 3 years ago
- Comments: 16 (1 by maintainers)
thanks @domenicozurlo , very helpful. I’ve ended up using a similar approach in our setup.
We have a script backup.sh which synchronize data periodically:
backup.sh
`#!/bin/sh
Script to synchronize data periodically
while true; do echo “Synchronize data from /var/lib/weaviate to /mnt/weaviate_data” rsync -av --delete /var/lib/weaviate/ /mnt/weaviate_data/ sleep 300 # backup every 5 minutes done`
And the entrypoint.sh for docker image:
`#!/bin/sh
Restore saved data at startup
if [ -d “/mnt/weaviate_data” ] && [ “$(ls -A /mnt/weaviate_data)” ]; then echo “Restoring data from /mnt/weaviate_data to /var/lib/weaviate” rsync -av /mnt/weaviate_data/ /var/lib/weaviate/ else echo “The /mnt/weaviate_data directory is empty or does not exist.” fi
Function to perform final backup before shutdown
On Azure app service SIGTERM is not triggered
shutdown_backup() { echo “Received SIGTERM, I perform the final backup…” rsync -av /var/lib/weaviate/ /mnt/weaviate_data/ echo “Final backup complete.” exit 0 }
Capture the SIGTERM signal to handle shutdown
trap ‘shutdown_backup’ SIGTERM
Start periodic backup in the background
./backup.sh &
Start Weaviate
weaviate --host 0.0.0.0 --port 80 --scheme http &
Wait for the Weaviate trial to finish
wait $!`
I found a solution using a different folder (for example /mnt/weaviate_data/). This folder is mounted with file share and I created a script which syncronize the /var/lib/weaviate/ folder with /mnt/weaviate_data/. At startup there is the restore of the data from /mnt/weaviate_data/
The only way currently is deploying via a kubernetes cluster, which is an overkill during development or for small applications. This is unfortunately excluding weaviate from various setups. I reached out to the team already but this appears to be of lower priority for them.
Try Azure AI search 😃