SeekStorm: Panic at commit

When I put some documents and run index.commit().await, I get panics:

commit level 1 committed documents 89 89 mode append
thread 'main' panicked at core\src\panicking.rs:221:5:
unsafe precondition(s) violated: slice::from_raw_parts_mut requires the pointer to be aligned and non-null, and the total size of the slice not to exceed `isize::MAX`
stack backtrace:
   0: std::panicking::begin_panic_handler
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14\library/std\src\panicking.rs:662
   1: core::panicking::panic_nounwind_fmt::runtime
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14\library/core\src\panicking.rs:112
   2: core::panicking::panic_nounwind_fmt
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14\library/core\src\panicking.rs:122
   3: core::panicking::panic_nounwind
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14\library/core\src\panicking.rs:221
   4: core::slice::raw::from_raw_parts_mut::precondition_check
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14\library\core\src\ub_checks.rs:70
   5: core::slice::raw::from_raw_parts_mut<u16>
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14\library\core\src\ub_checks.rs:77
   6: seekstorm::utils::cast_byte_ushort_slice
             at C:\Users\dekak\.cargo\registry\src\index.crates.io-6f17d22bba15001f\seekstorm-0.12.3\src\seekstorm\utils.rs:156
   7: seekstorm::single::single_docid::async_fn$0
             at C:\Users\dekak\.cargo\registry\src\index.crates.io-6f17d22bba15001f\seekstorm-0.12.3\src\seekstorm\single.rs:164
   8: seekstorm::single::single_blockid::async_fn$0
             at C:\Users\dekak\.cargo\registry\src\index.crates.io-6f17d22bba15001f\seekstorm-0.12.3\src\seekstorm\single.rs:383
   9: seekstorm::search::impl$1::search::async_fn$0
             at C:\Users\dekak\.cargo\registry\src\index.crates.io-6f17d22bba15001f\seekstorm-0.12.3\src\seekstorm\search.rs:1854
  10: seekstorm::index::warmup::async_fn$0
             at C:\Users\dekak\.cargo\registry\src\index.crates.io-6f17d22bba15001f\seekstorm-0.12.3\src\seekstorm\index.rs:1998
  11: seekstorm::commit::impl$0::commit::async_fn$0
             at C:\Users\dekak\.cargo\registry\src\index.crates.io-6f17d22bba15001f\seekstorm-0.12.3\src\seekstorm\commit.rs:126
  12: fileserver::scan::async_fn$0
             at .\src\main.rs:282
  13: fileserver::main::async_block$0
             at .\src\main.rs:335

(...tokio runtime...)

It seems the error is somehow dependent on amount and content of indexed data inserted, but the relation is too complex for me to follow, probably because of some internal compression algorithms. Happens with both Text and StringSet field types. Problem appeared only in Mmap mode, for RAM everything works for now. Is there some limit to how much data may be inserted in single field or commit? Maybe some limitation of mmap on Windows 11?

stable-x86_64-pc-windows-msvc (default) rustc 1.82.0 (f6e511eec 2024-10-15)

Schema

        vec![
            SchemaField::new(
                "names".to_owned(),
                true,
                true,
                FieldType::StringSet, // TODO: perhaps TEXT, so the tokenizer is used
                false,
                5.0,
            ),
            SchemaField::new(
                "major_aspects".to_owned(),
                true,
                true,
                FieldType::StringSet,
                false,
                2.0,
            ),
            SchemaField::new(
                "minor_aspects".to_owned(),
                true,
                true,
                FieldType::Text,
                false,
                1.0,
            ),
            SchemaField::new("type".to_owned(), true, false, FieldType::String, true, 0.0),
            SchemaField::new(
                "content".to_owned(),
                true,
                false,
                FieldType::String,
                false,
                0.0,
            ),
        ]

Setup

    let meta = IndexMetaObject {
        id: 0,
        name: "Fileserver Database".to_string(),
        similarity: SimilarityType::Bm25f,
        tokenizer: TokenizerType::UnicodeAlphanumericFolded,
        access_type: AccessType::Mmap,
    };

    let index = create_index(
        DB_PATH.as_ref(),
        meta,
        &DatabaseEntry::schema(),
        true,
        &Vec::new(),
        11,
        false,
    )
    .unwrap();

On my computer it’s enough to put a single commit with document

{"names": Array [String("Concerto pour piano No.2 en Fa Mineur opus 21 - Maestoso")], "major_aspects": Array [String("Concerto pour piano No.2"), String("Fryderyk Chopin")], "type": String("piece"), "minor_aspects": String("Fryderyk Chopin Fryderyk Chopin Vladimir Pet
roschoff Hélène Gàl Philharmonisches Festspielorchester"), "content": String("{\"titles\":[\"Concerto pour piano No.2 en Fa Mineur opus 21 - Maestoso\"],\"collections\":[\"Concerto pour piano No.2\"],\"collection_display_authors\":[\"Fryderyk Chopin\"],\"collection_authors_sort\":[\"Chopin, Fryderyk\"],\"authors_display\":[\"Fryderyk Chopin\"],\"authors_sort\":[\"Chopin, Fryderyk\"],\"main_authors\":[\"Fryderyk Chopin\"],\"other_authors\":[\"Fryderyk Chopin\",\"Vladimir Petroschoff\",\"Hélène Gàl\",\"Philharmonisches Festspielorchester\"]}")}

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 22 (16 by maintainers)

Most upvoted comments

Sorry for the late reply, I was away from my computer due to the Spring Festival. After testing, the problem has been fixed. By the way, Happy Spring Festival

6900 refers to the number of indexed documents. I tried to debuging the SeekStorm server in vscode using codelldb, and it still did not become unresponsive when it reached 15000+.

This happens every time when I run the release version. The debug version (not debugging in IDE) also appears after 6900, but it will resume responding after a few seconds. I don’t know if it is because the program generated by debugging is not optimized by the compiler. #Also the error message and backtrace No error message,and idon’t know where is backtrace This should be a tricky question.

dataset: https://dumps.wikimedia.org/zhwiki/20250120/zhwiki-20250120-pages-articles-multistream.xml.bz2 Test code:https://www.mediafire.com/file/y0stsgjwanh4g3z/ConsoleApp1.7z/file