milvus: [Bug]: 2.3.2 downgrade to 2.2.14 fail exited with code 2
Is there an existing issue for this?
- I have searched the existing issues
Environment
- Milvus version: 2.3.2
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): Linux debian 5.10.0-23-amd64 #1 SMP Debian 5.10.179-1 (2023-05-12) x86_64 GNU/Linux
- CPU/Memory: 32/64G
- GPU:
- Others:
Current Behavior
Milvus 2.2.14 milvus-standalone CPU device up is ok, after inserting some data try to upgrade to 2.3.2, the data directory uses the previous data directory(2.3.2 use 2.2.14 data directory)
Milvus 2.3.2 hit exited with code 134 #26717 standalone: container_name: milvus-standalone image: milvusdb/milvus:v2.3.2 command: [“milvus”, “run”, “standalone”] security_opt: - seccomp:unconfined
The configuration already has seccomp:unconfined but still cannot start
At this time, switch back to 2.2.14 and prompt exited with code 2
[lscpu]
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 43 bits physical, 48 bits virtual
CPU(s): 32
On-line CPU(s) list: 0-31
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 32
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 45
Model name: Intel® Xeon® CPU E5-2660 0 @ 2.20GHz
Stepping: 7
CPU MHz: 2194.711
BogoMIPS: 4389.42
Hypervisor vendor: VMware
Virtualization type: full
L1d cache: 1 MiB
L1i cache: 1 MiB
L2 cache: 8 MiB
L3 cache: 640 MiB
NUMA node0 CPU(s): 0-15
NUMA node1 CPU(s): 16-31
Vulnerability Itlb multihit: KVM: Mitigation: VMX unsupported
Vulnerability L1tf: Mitigation; PTE Inversion
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT Host state unknown
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Mmio stale data: Unknown: No mitigations
Vulnerability Retbleed: Mitigation; IBRS
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; IBRS, IBPB conditional, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Flags:
fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx rdtscp lm
constant_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc cpuid pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2
x2apic popcnt tsc_deadline_timer aes xsave avx hypervisor lahf_lm pti ssbd ibrs ibpb stibp tsc_adjust arat md_clear flush_l1d
arch_capabilities
No AVX2
Expected Behavior
Milvus 2.2.14 CPU device how to upgrade to 2.3.2 or return to 2.2.14 without losing data
Steps To Reproduce
No response
Milvus Log
[Milvus 2.3.2]
milvus-standalone-2_3_2 | goroutine 5130 [chan receive, locked to thread]: milvus-standalone-2_3_2 | runtime.gopark(0x0?, 0xc003197f01?, 0x0?, 0x0?, 0x3c46bf6?) milvus-standalone-2_3_2 | /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc003197ed0 sp=0xc003197eb0 pc=0x17870d6 milvus-standalone-2_3_2 | runtime.chanrecv(0xc002edb500, 0xc003197fa8, 0x1) milvus-standalone-2_3_2 | /usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc003197f60 sp=0xc003197ed0 pc=0x174f5bd milvus-standalone-2_3_2 | runtime.chanrecv2(0xc002c83220?, 0xc0037f9590?) milvus-standalone-2_3_2 | /usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc003197f88 sp=0xc003197f60 pc=0x174f0f8 milvus-standalone-2_3_2 | github.com/panjf2000/ants/v2.(*goWorker).run.func1() milvus-standalone-2_3_2 | /go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/worker.go:63 +0x7b fp=0xc003197fe0 sp=0xc003197f88 pc=0x2c03c7b milvus-standalone-2_3_2 | runtime.goexit() milvus-standalone-2_3_2 | /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc003197fe8 sp=0xc003197fe0 pc=0x17bd9a1 milvus-standalone-2_3_2 | created by github.com/panjf2000/ants/v2.(*goWorker).run milvus-standalone-2_3_2 | /go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/worker.go:48 +0x65 milvus-standalone-2_3_2 | milvus-standalone-2_3_2 | rax 0x6b2e00d9f1891bdb milvus-standalone-2_3_2 | rbx 0x7ff35fc1d128 milvus-standalone-2_3_2 | rcx 0x38c48ded80000000 milvus-standalone-2_3_2 | rdx 0x0 milvus-standalone-2_3_2 | rdi 0x1 milvus-standalone-2_3_2 | rsi 0x653f1df3 milvus-standalone-2_3_2 | rbp 0x7ff277bbf208 milvus-standalone-2_3_2 | rsp 0x7ff277bbf150 milvus-standalone-2_3_2 | r8 0x7ff277bddec8 milvus-standalone-2_3_2 | r9 0x10 milvus-standalone-2_3_2 | r10 0xfffffffffffff7fe milvus-standalone-2_3_2 | r11 0x7ff49ae280b0 milvus-standalone-2_3_2 | r12 0x9ddfea08eb382d69 milvus-standalone-2_3_2 | r13 0x7ff4970ff8c0 milvus-standalone-2_3_2 | r14 0x12a05f200 milvus-standalone-2_3_2 | r15 0x198bb milvus-standalone-2_3_2 | rip 0x7ff496ae6e71 milvus-standalone-2_3_2 | rflags 0x10206 milvus-standalone-2_3_2 | cs 0x33 milvus-standalone-2_3_2 | fs 0x0 milvus-standalone-2_3_2 | gs 0x0 milvus-standalone-2_3_2 | milvus-standalone-2_3_2 | ----- milvus-standalone-2_3_2 | milvus-standalone-2_3_2 exited with code 134
[Milvus 2.2.14]
milvus-standalone-2.2.14 | panic: Invalid argument: Column families not opened: properties milvus-standalone-2.2.14 | milvus-standalone-2.2.14 | goroutine 1 [running]: milvus-standalone-2.2.14 | github.com/milvus-io/milvus/cmd/roles.(*MilvusRoles).Run(0xc00056de48, 0x1, {0x0, 0x0}) milvus-standalone-2.2.14 | /go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:245 +0xcf5 milvus-standalone-2.2.14 | github.com/milvus-io/milvus/cmd/milvus.(*run).execute(0xc001080060, {0xc000194060?, 0x3, 0x3}, 0xc0007e81e0) milvus-standalone-2.2.14 | /go/src/github.com/milvus-io/milvus/cmd/milvus/run.go:117 +0x6ae milvus-standalone-2.2.14 | github.com/milvus-io/milvus/cmd/milvus.RunMilvus({0xc000194060?, 0x3, 0x3}) milvus-standalone-2.2.14 | /go/src/github.com/milvus-io/milvus/cmd/milvus/milvus.go:60 +0x21e milvus-standalone-2.2.14 | main.main() milvus-standalone-2.2.14 | /go/src/github.com/milvus-io/milvus/cmd/main.go:26 +0x2e milvus-standalone-2.2.14 exited with code 2
Anything else?
No response
About this issue
- Original URL
- State: closed
- Created 8 months ago
- Comments: 22 (12 by maintainers)
for 2.3.2, since we changed the log format, you can not downgrade to 2.2.14. remove the directory to /var/lib/milvus/rdb_data could solve the problem