milvus: [Bug]: 2.3.2 downgrade to 2.2.14 fail exited with code 2

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: 2.3.2
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): Linux debian 5.10.0-23-amd64 #1 SMP Debian 5.10.179-1 (2023-05-12) x86_64 GNU/Linux
- CPU/Memory: 32/64G
- GPU: 
- Others:

Current Behavior

Milvus 2.2.14 milvus-standalone CPU device up is ok, after inserting some data try to upgrade to 2.3.2, the data directory uses the previous data directory(2.3.2 use 2.2.14 data directory)

Milvus 2.3.2 hit exited with code 134 #26717 standalone: container_name: milvus-standalone image: milvusdb/milvus:v2.3.2 command: [“milvus”, “run”, “standalone”] security_opt: - seccomp:unconfined

The configuration already has seccomp:unconfined but still cannot start

At this time, switch back to 2.2.14 and prompt exited with code 2

[lscpu]

Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 43 bits physical, 48 bits virtual CPU(s): 32 On-line CPU(s) list: 0-31 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 32 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 45 Model name: Intel® Xeon® CPU E5-2660 0 @ 2.20GHz Stepping: 7 CPU MHz: 2194.711 BogoMIPS: 4389.42 Hypervisor vendor: VMware Virtualization type: full L1d cache: 1 MiB L1i cache: 1 MiB L2 cache: 8 MiB L3 cache: 640 MiB NUMA node0 CPU(s): 0-15 NUMA node1 CPU(s): 16-31 Vulnerability Itlb multihit: KVM: Mitigation: VMX unsupported Vulnerability L1tf: Mitigation; PTE Inversion Vulnerability Mds: Mitigation; Clear CPU buffers; SMT Host state unknown Vulnerability Meltdown: Mitigation; PTI Vulnerability Mmio stale data: Unknown: No mitigations Vulnerability Retbleed: Mitigation; IBRS Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; IBRS, IBPB conditional, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected Flags:
fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc cpuid pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx hypervisor lahf_lm pti ssbd ibrs ibpb stibp tsc_adjust arat md_clear flush_l1d arch_capabilities

No AVX2

Expected Behavior

Milvus 2.2.14 CPU device how to upgrade to 2.3.2 or return to 2.2.14 without losing data

Steps To Reproduce

No response

Milvus Log

[Milvus 2.3.2]

milvus-standalone-2_3_2 | goroutine 5130 [chan receive, locked to thread]: milvus-standalone-2_3_2 | runtime.gopark(0x0?, 0xc003197f01?, 0x0?, 0x0?, 0x3c46bf6?) milvus-standalone-2_3_2 | /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc003197ed0 sp=0xc003197eb0 pc=0x17870d6 milvus-standalone-2_3_2 | runtime.chanrecv(0xc002edb500, 0xc003197fa8, 0x1) milvus-standalone-2_3_2 | /usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc003197f60 sp=0xc003197ed0 pc=0x174f5bd milvus-standalone-2_3_2 | runtime.chanrecv2(0xc002c83220?, 0xc0037f9590?) milvus-standalone-2_3_2 | /usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc003197f88 sp=0xc003197f60 pc=0x174f0f8 milvus-standalone-2_3_2 | github.com/panjf2000/ants/v2.(*goWorker).run.func1() milvus-standalone-2_3_2 | /go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/worker.go:63 +0x7b fp=0xc003197fe0 sp=0xc003197f88 pc=0x2c03c7b milvus-standalone-2_3_2 | runtime.goexit() milvus-standalone-2_3_2 | /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc003197fe8 sp=0xc003197fe0 pc=0x17bd9a1 milvus-standalone-2_3_2 | created by github.com/panjf2000/ants/v2.(*goWorker).run milvus-standalone-2_3_2 | /go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/worker.go:48 +0x65 milvus-standalone-2_3_2 | milvus-standalone-2_3_2 | rax 0x6b2e00d9f1891bdb milvus-standalone-2_3_2 | rbx 0x7ff35fc1d128 milvus-standalone-2_3_2 | rcx 0x38c48ded80000000 milvus-standalone-2_3_2 | rdx 0x0 milvus-standalone-2_3_2 | rdi 0x1 milvus-standalone-2_3_2 | rsi 0x653f1df3 milvus-standalone-2_3_2 | rbp 0x7ff277bbf208 milvus-standalone-2_3_2 | rsp 0x7ff277bbf150 milvus-standalone-2_3_2 | r8 0x7ff277bddec8 milvus-standalone-2_3_2 | r9 0x10 milvus-standalone-2_3_2 | r10 0xfffffffffffff7fe milvus-standalone-2_3_2 | r11 0x7ff49ae280b0 milvus-standalone-2_3_2 | r12 0x9ddfea08eb382d69 milvus-standalone-2_3_2 | r13 0x7ff4970ff8c0 milvus-standalone-2_3_2 | r14 0x12a05f200 milvus-standalone-2_3_2 | r15 0x198bb milvus-standalone-2_3_2 | rip 0x7ff496ae6e71 milvus-standalone-2_3_2 | rflags 0x10206 milvus-standalone-2_3_2 | cs 0x33 milvus-standalone-2_3_2 | fs 0x0 milvus-standalone-2_3_2 | gs 0x0 milvus-standalone-2_3_2 | milvus-standalone-2_3_2 | ----- milvus-standalone-2_3_2 | milvus-standalone-2_3_2 exited with code 134

[Milvus 2.2.14]

milvus-standalone-2.2.14 | panic: Invalid argument: Column families not opened: properties milvus-standalone-2.2.14 | milvus-standalone-2.2.14 | goroutine 1 [running]: milvus-standalone-2.2.14 | github.com/milvus-io/milvus/cmd/roles.(*MilvusRoles).Run(0xc00056de48, 0x1, {0x0, 0x0}) milvus-standalone-2.2.14 | /go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:245 +0xcf5 milvus-standalone-2.2.14 | github.com/milvus-io/milvus/cmd/milvus.(*run).execute(0xc001080060, {0xc000194060?, 0x3, 0x3}, 0xc0007e81e0) milvus-standalone-2.2.14 | /go/src/github.com/milvus-io/milvus/cmd/milvus/run.go:117 +0x6ae milvus-standalone-2.2.14 | github.com/milvus-io/milvus/cmd/milvus.RunMilvus({0xc000194060?, 0x3, 0x3}) milvus-standalone-2.2.14 | /go/src/github.com/milvus-io/milvus/cmd/milvus/milvus.go:60 +0x21e milvus-standalone-2.2.14 | main.main() milvus-standalone-2.2.14 | /go/src/github.com/milvus-io/milvus/cmd/main.go:26 +0x2e milvus-standalone-2.2.14 exited with code 2

Anything else?

No response

About this issue

  • Original URL
  • State: closed
  • Created 8 months ago
  • Comments: 22 (12 by maintainers)

Most upvoted comments

for 2.3.2, since we changed the log format, you can not downgrade to 2.2.14. remove the directory to /var/lib/milvus/rdb_data could solve the problem