PaCMAP: Could not broadcast input array from shape (0,) into shape (60,)

Running the following code:

embeddings_list = df['embedding'].tolist()
embeddings_array = np.array(embeddings_list)
print('embeddings_array.shape:', embeddings_array.shape)
print('embeddings_array[0][0] type:', type(embeddings_array[0][0]))

# Check for NaN and Inf values
nan_count = np.isnan(embeddings_array).sum()
inf_count = np.isinf(embeddings_array).sum()
print(f"Found {nan_count} NaN values and {inf_count} Inf values in embeddings.")

# Perform PaCMAP dimensionality reduction
reducer = pacmap.PaCMAP(
    n_components=2,
    n_neighbors=10,
    MN_ratio=0.5,
    FP_ratio=2.0
)
pacmap_result = reducer.fit_transform(embeddings_array, init="pca")

I get the following output:

embeddings_array.shape: (42695, 1024)
embeddings_array[0][0] type: <class 'numpy.float64'>
Found 0 NaN values and 0 Inf values in embeddings.

Traceback (most recent call last):
  File "/User/user/misc/visualize_embeddings.py", line 108, in <module>
    perform_pca()
  File "/User/user/misc/visualize_embeddings.py", line 52, in perform_pca
    pacmap_result = reducer.fit_transform(embeddings_array, init="pca")
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/User/user/.venv/lib/python3.12/site-packages/pacmap/pacmap.py", line 1003, in fit_transform
    self.fit(X, init, save_pairs)
  File "/User/user/.venv/lib/python3.12/site-packages/pacmap/pacmap.py", line 962, in fit
    self.sample_pairs(X, self.save_tree)
  File "/User/user/.venv/lib/python3.12/site-packages/pacmap/pacmap.py", line 1081, in sample_pairs
    self.pair_neighbors, self.pair_MN, self.pair_FP, self.tree = generate_pair(
                                                                 ^^^^^^^^^^^^^^
  File "/User/user/.venv/lib/python3.12/site-packages/pacmap/pacmap.py", line 504, in generate_pair
    nbrs[i, :] = nbrs_[1:]
    ~~~~^^^^^^
ValueError: could not broadcast input array from shape (0,) into shape (60,)

Am i doing everything correctly? Is the array formatted in the correct way? The embeddings have 1024 dimensions and there are 42695 embedded texts.

About this issue

Most upvoted comments

Mine is ARM.

Could you please try to create a simple environment using poetry or uv? Mamba/anaconda/whatever does a lot of magic under the hood, and are best avoided unless for research purposes (with poor software supply chain hygiene practices)

The test is done under uv for ARM machine w/ Sonoma, but we still cannot replicate this error.

Thanks for providing us with the detailed information. Unfortunately, we don’t have the specific setup the same as your hardware to replicate your error. Thanks for the help from @gauravrparikh, we have replicated your pip freeze setup in the MacOS system, but it works fine. Based on the error message that you have provided, my guess would be some issues with the ANNOY package that the NN tree structure is not successfully constructed, which finally led to this error.

I would leave this issue open till we have figured out what is exactly happening from your side. However, if you would like to use PaCMAP and its variants, you can directly use google colab to run your code. It is working and tested successfully for the code that I have provided previously. Again, thanks for your interest in using PaCMAP.

Mine is ARM.

Could you please try to create a simple environment using poetry or uv? Mamba/anaconda/whatever does a lot of magic under the hood, and are best avoided unless for research purposes (with poor software supply chain hygiene practices)