NGT: QG Index Creation and Build Failure

Hello,

I had created a 113M embedding index (768 dim), with the following prf:

AccuracyTable
BatchSizeForCreation    200
BuildTimeLimit  0
DatabaseType    Memory
Dimension       768
DistanceType    Cosine
DynamicEdgeSizeBase     30
DynamicEdgeSizeRate     20
EdgeSizeForCreation     10
EdgeSizeForSearch       40
EdgeSizeLimitForCreation        5
EpsilonForCreation      0.1
GraphType       ANNG
IncomingEdge    80
IncrimentalEdgeSizeLimitForTruncation   0
IndexType       GraphAndTree
ObjectAlignment False
ObjectType      Float-4
OutgoingEdge    10
PathAdjustmentInterval  0
PrefetchOffset  1
PrefetchSize    3072
SeedSize        10
SeedType        None
ThreadPoolSize  32
TruncationThreadPoolSize        8

The search works fine on this index.

However, when I attempt to create and build a QG index, I run into multiple issues.

  1. Creating a QG index for this, by executing the following command, qbg create-qg -d 768 -D C -E 10 -S 40 -i t -o f -p 32 -N 384 -c 16 -C sqsu8 -B 2 -b 200 -M l -L s -e 0.1 -v /path_to_index

Issue: The generated /path_to_index/qg/prf file contains values that do not match the arguments I passed.

Here is the generated qg/prf file:

BatchSize       1000
CentroidCreationMode    1
DataSize        0
DataType        1
Dimension       768
DistanceType    1
GenuineDataType 1
GenuineDimension        768
GlobalCentroidLimit     1
GlobalRange     0
LocalCentroidCreationMode       1
LocalCentroidLimit      16
LocalClusterDataType    2
LocalCodebookState      1
LocalDivisionNo 384
LocalIDByteSize 2
LocalRange      0
LocalSampleCoefficient  100
MaxMagnitude    -1
QuantizerType   0
RefinementDataType      99
ScalarQuantizationClippingRate  0.01
ScalarQuantizationNoOfSamples   0
ScalarQuantizationOffset        0
ScalarQuantizationScale 0
SingleLocalCodebook     0
ThreadSize      24

Q. Firstly, the qg/prf values are not matching the passed argument values, Why are the values in qg/prf different from the ones I passed? Is there an internal default overriding them?

  1. QG Index Build Failure Post creation, I executed the command for building QG index, qbg build-qg -E 128 -v /path_to_index

which fails in creation (or gets stuck).

Observations:

  • CPU is getting 100% when is it processing the index objects (machine used is n2-highmem-128)
  • Intermediate logs:
append: Data loading time=2.255e-05 (sec) 0.02255 (msec)
# of objects=16
Index creation time=0.00132462 (sec) 1.32462 (msec)
qbg: loading the rotation...
QuantizationCodebook::buildIndex
QuantizationCodebook::buildIndex # of the centroids=1
load() done
codebook index size=1
  • Tail logs for this command:
# of processed objects=105000000, time=21.5226 (m), vm size=104.58 G/104.58 G
# of processed objects=106000000, time=21.7245 (m), vm size=104.58 G/104.58 G
# of processed objects=107000000, time=21.9246 (m), vm size=104.58 G/104.58 G
# of processed objects=108000000, time=22.1269 (m), vm size=104.58 G/104.58 G
# of processed objects=109000000, time=22.3308 (m), vm size=104.58 G/104.58 G
# of processed objects=110000000, time=22.5344 (m), vm size=104.58 G/104.58 G
# of processed objects=111000000, time=22.7377 (m), vm size=104.58 G/104.58 G
# of processed objects=112000000, time=22.941 (m), vm size=104.58 G/104.58 G
# of processed objects=113000000, time=23.1424 (m), vm size=104.58 G/104.58 G
cp: cannot stat '/path_to_index/qg/ws/hkc_3c': No such file or directory

After the cp error log, cpu and memory both become 0 and index build gets stuck (or failed).

Questions

  1. Why does the QG index creation (create-qg) generate a qg/prf file with mismatched values?
  • Are there any internal defaults that override command-line arguments?
  • Is there a way to verify which parameters were actually used?
  1. Why does the build-qg process fail after cp: cannot stat ā€˜/path_to_index/qg/ws/hkc_3c’?
  • What is this missing file, and why is it required?
  • Is it possible that a previous step failed, leading to missing files?
  • Is there any known issue related to QG index builds for very large datasets?
  1. How can I resolve this issue and successfully build the QG index for 113M embeddings?
  • Are there additional configuration settings I should check?
  • Are there memory/CPU constraints I should be aware of?

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 29 (16 by maintainers)

Most upvoted comments

Image

I tried re-running the search query (qbg search-qg index query.tsv), and did not come across that cached memory issue. Initially both RAM (in-memory), and cached memory increases, then all that cached memory transfers into in-memory RAM which reaches 94% (approx. 800GB) for a single search.

  1. qbg build-qg -p 1 -v [index]
optimizing...
DEBUG: Quantizer::open the inverted index [index]/qg/ivt
DEBUG: The inveverted index data size: 0
DEBUG: The first cluster size: 0
DEBUG: The inverted index file size: -1 bytes
optimize: # of objects=1000
optimize: updated # of objects=1000
optimize: # of clusters=16:0
the codebook index file is missing. this index must be QG.
optimize: # of vectors=1000/1000, # of matrices=1
optimize: time=144.055 (ms)
  1. qbg build-qg -p 2 -v [index]
building the inverted index...
GraphOptimizer::execute:  vm size=6.36 G:10.33 G
 delete all of objects
 vm size=6.36 G:10.33 G
GraphOptimizer: adjusting outgoing and incoming edges...
Optimizer::execute: Extract the graph data.
GraphReconstructor::reconstructGraph:10:120:9223372036854775807
GraphReconstructor: Warning. The edges are too few. 0:10 for 1
# of the nodes edges of which are in short = 1
Reconstruction time=5.1e-06:2.48e-07:1.63e-07
Optimizer::execute: Graph reconstruction time=2.2173e-05 (sec) 
GraphOptimizer: redusing shortcut edges...
GraphReconstructor::adjustPaths: graph preparing time=0.000171 (ms)
GraphReconstructor::adjustPaths: extracting removed edge candidates time=5.42781 (ms)
removeCandidateCount=0
Optimizer::execute: Path adjustment time=0.00545387 (sec) 
data/index/inhouse_v3_4_beit_2025_03_12/qg/ws/hkc_opt/sv-0->[index]/qg/local-0
append: Data loading time=3.3083e-05 (sec) 0.033083 (msec)
# of objects=16
Index creation time=0.00546047 (sec) 5.46047 (msec)
data/index/inhouse_v3_4_beit_2025_03_12/qg/ws/hkc_opt/sv-1->[index]/qg/local-1
append: Data loading time=1.807e-05 (sec) 0.01807 (msec)
# of objects=16
Index creation time=0.00449889 (sec) 4.49889 (msec)
data/index/inhouse_v3_4_beit_2025_03_12/qg/ws/hkc_opt/sv-2->[index]/qg/local-2
append: Data loading time=2.1837e-05 (sec) 0.021837 (msec)
# of objects=16
Index creation time=0.00438631 (sec) 4.38631 (msec)
.
.
.
.
.
qbg: loading the rotation...
DEBUG: Quantizer::open the inverted index [index]/qg/ivt
DEBUG: The inveverted index data size: 0
DEBUG: The first cluster size: 0
DEBUG: The inverted index file size: -1 bytes
QuantizationCodebook::buildIndex
QuantizationCodebook::buildIndex # of the centroids=1
DEBUG: Quantizer::open the inverted index [index]/qg/ivt
DEBUG: The inveverted index data size: 0
DEBUG: The first cluster size: 0
DEBUG: The inverted index file size: -1 bytes
loading the rotation...
QuantizationCodebook::buildIndex
QuantizationCodebook::buildIndex # of the centroids=1
codebook index size=1
# of processed objects=1000000, time=12.6164 (s), vm size=96.7 G/96.74 G
# of processed objects=2000000, time=25.1842 (s), vm size=96.7 G/96.74 G
# of processed objects=3000000, time=37.7162 (s), vm size=96.7 G/96.74 G
# of processed objects=4000000, time=50.2343 (s), vm size=96.7 G/96.74 G
.
.
.
.
.
.
# of processed objects=113000000, time=23.6545 (m), vm size=96.7 G/96.74 G
cp: cannot stat '[index]/qg/ws/hkc_3c': No such file or directory
DEBUG: save: The Inverted index size=2
DEBUG: save: The entry size=113676655
DEBUG: Quantizer::save the inverted index [index]/qg/ivt
DEBUG: The inveverted index data size: 2
DEBUG: The first cluster size: 113676655
DEBUG: The inverted index file size: 87758377680 bytes
DEBUG: Quantizer::open the inverted index [index]/qg/ivt
DEBUG: The inveverted index data size: 2
DEBUG: The first cluster size: 113676655
DEBUG: The inverted index file size: 87758377680 bytes
loading the rotation...
QuantizationCodebook::buildIndex
QuantizationCodebook::buildIndex # of the centroids=1
NGTQ index is completed.
  time=23.8782 (m)
  vmsize=99.85 G
  peak vmsize=191.58 G
saving...
NGTQ and NGTQBG indices are completed.
  vmsize=15.57 G
  peak vmsize=191.58 G
  1. qbg build-qg -p 3 -v [index]
building the quantized graph... 
DEBUG: Quantizer::open the inverted index [index]/qg/ivt
DEBUG: The inveverted index data size: 2
DEBUG: The first cluster size: 113676655
DEBUG: The inverted index file size: 87758377680 bytes
loading the rotation...
QuantizationCodebook::buildIndex
QuantizationCodebook::buildIndex # of the centroids=1
DEBUG: construct
DEBUG:  vmsize=102.63 G
DEBUG:  peak vmsize=102.63 G
DEBUG: extractInvertedIndexObject: The Inverted index size=2
DEBUG: extractInvertedIndexObject: The entry size=113676655
DEBUG: extractInvertedIndexObject: The last ID=2408073
DEBUG:  inverted index object size=2408074
DEBUG:  vmsize=105.65 G
DEBUG:  peak vmsize=105.65 G
DEBUG:  erased
DEBUG:  vmsize=23.92 G
DEBUG:  peak vmsize=105.65 G
DEBUG: graph repository size=113676656
DEBUG:  vmsize=23.92 G
DEBUG:  peak vmsize=105.65 G
DEBUG: resized
DEBUG:  vmsize=28.15 G
DEBUG:  peak vmsize=105.65 G
# of processed objects=1/113676655(0%)
  vmsize=28.15 G
  peak vmsize=105.65 G
Image

Phase-3 command failed again due to Segmentation fault (core dumped), not sure why that happened.

  1. qbg build-qg -p 1 -v [index]
optimizing...
optimize: # of objects=1000
optimize: updated # of objects=1000
optimize: # of clusters=16:0
the codebook index file is missing. this index must be QG.
optimize: # of vectors=1000/1000, # of matrices=1
optimize: time=136.683 (ms)
  1. qbg build-qg -p 2 -v [index]
starting few log lines...
building the inverted index...
GraphOptimizer::execute:  vm size=6.11 G:10.02 G
 delete all of objects
 vm size=6.11 G:10.02 G
GraphOptimizer: adjusting outgoing and incoming edges...
Optimizer::execute: Extract the graph data.
GraphReconstructor::reconstructGraph:10:120:9223372036854775807
GraphReconstructor: Warning. The edges are too few. 0:10 for 1
# of the nodes edges of which are in short = 1
Reconstruction time=5.534e-06:1.54e-07:1.9e-07
Optimizer::execute: Graph reconstruction time=3.693e-05 (sec) 
GraphOptimizer: redusing shortcut edges...
GraphReconstructor::adjustPaths: graph preparing time=0.00022 (ms)
GraphReconstructor::adjustPaths: extracting removed edge candidates time=4.53551 (ms)
removeCandidateCount=0
Optimizer::execute: Path adjustment time=0.0045632 (sec) 
data/index/inhouse_v3_4_beit_2025_03_12/qg/ws/hkc_opt/sv-0->data/index/inhouse_v3_4_beit_2025_03_12/qg/local-0
append: Data loading time=3.4526e-05 (sec) 0.034526 (msec)
# of objects=16


last few log lines...

# of processed objects=113000000, time=26.5813 (m), vm size=96.7 G/96.7 G
cp: cannot stat 'data/index/inhouse_v3_4_beit_2025_03_12/qg/ws/hkc_3c': No such file or directory
NGTQ index is completed.
  time=26.8049 (m)
  vmsize=99.85 G
  peak vmsize=191.58 G
saving...
NGTQ and NGTQBG indices are completed.
  vmsize=15.57 G
  peak vmsize=191.58 G
  1. qbg build-qg -p 3 -v [index]
building the quantized graph... 
construct
  vmsize=102.63 G
  peak vmsize=102.63 G
inverted index object size=2408074
  vmsize=105.65 G
  peak vmsize=105.65 G
erased
  vmsize=23.92 G
  peak vmsize=105.65 G
graph repository size=113676656
  vmsize=23.92 G
  peak vmsize=105.65 G
resized
  vmsize=28.15 G
  peak vmsize=105.65 G
# of processed objects=1/113676655(0%)
  vmsize=28.15 G
  peak vmsize=105.65 G


Segmentation fault (core dumped)

So the phase-3 command failed due to Segmentation fault, not sure why that happened.

Image

I can see a small spike in disk writes, and context switching also, but the memory usage is very low.

What could be the reason for this segmentation fault (core dumped) error?