qdrant: on_disk payload is not decreasing RAM on system

Hi , I am trying out mnist dataset with rust client, and I have qdrant release code v1.8.1 and built from source with command

cargo build 
cargo run

I have configured on_disk_payload = true, but I have seen RAM is increasing with the number of rows in the dataset, I have started to insert 100 rows to 10k, 20k, 30k Ideally RAM shouldn’t increase as parameter is already configured

It looks like on_disk_payload itself is not working

About this issue

Original URL
State: open
Created 3 months ago
Comments: 18 (9 by maintainers)

Most upvoted comments

let dimension= 785
let rows=20000

let mut collection_name="col"


let config= QdrantClientConfig::from_url("localhost")
let client= QdrantClient::new(Some(config)).unwrap()


let x= client.create_collection(& CreateCollection{
collection_name: collection_name,
vectors_config: Some(VectorsConfig{
config:Some(Confi::Params(VectorParams{
                         size: dimension,
                          distance:cosine,
                          on_disk:Some(true),
                          .. Default:: default ()
})),
optimisers_config: Some(OptimizersConfigDiff{
memmap_threshold: Some(1)
   .. Default:: default (),
}),
quantization_config: Some(QuantizationConfig{
quantization: Some(Quantization:: Scalar (ScalarQuantization{
r#type: QuantizationType::Int8
always_ram: Some(false)
.. Default:: default ()
})),
shard_number:2(),
.. Default:: default ()
}).await

harshakhmk on Apr 2, 2024