PaCMAP: Could not broadcast input array from shape (0,) into shape (60,)
Running the following code:
embeddings_list = df['embedding'].tolist()
embeddings_array = np.array(embeddings_list)
print('embeddings_array.shape:', embeddings_array.shape)
print('embeddings_array[0][0] type:', type(embeddings_array[0][0]))
# Check for NaN and Inf values
nan_count = np.isnan(embeddings_array).sum()
inf_count = np.isinf(embeddings_array).sum()
print(f"Found {nan_count} NaN values and {inf_count} Inf values in embeddings.")
# Perform PaCMAP dimensionality reduction
reducer = pacmap.PaCMAP(
n_components=2,
n_neighbors=10,
MN_ratio=0.5,
FP_ratio=2.0
)
pacmap_result = reducer.fit_transform(embeddings_array, init="pca")
I get the following output:
embeddings_array.shape: (42695, 1024)
embeddings_array[0][0] type: <class 'numpy.float64'>
Found 0 NaN values and 0 Inf values in embeddings.
Traceback (most recent call last):
File "/User/user/misc/visualize_embeddings.py", line 108, in <module>
perform_pca()
File "/User/user/misc/visualize_embeddings.py", line 52, in perform_pca
pacmap_result = reducer.fit_transform(embeddings_array, init="pca")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/User/user/.venv/lib/python3.12/site-packages/pacmap/pacmap.py", line 1003, in fit_transform
self.fit(X, init, save_pairs)
File "/User/user/.venv/lib/python3.12/site-packages/pacmap/pacmap.py", line 962, in fit
self.sample_pairs(X, self.save_tree)
File "/User/user/.venv/lib/python3.12/site-packages/pacmap/pacmap.py", line 1081, in sample_pairs
self.pair_neighbors, self.pair_MN, self.pair_FP, self.tree = generate_pair(
^^^^^^^^^^^^^^
File "/User/user/.venv/lib/python3.12/site-packages/pacmap/pacmap.py", line 504, in generate_pair
nbrs[i, :] = nbrs_[1:]
~~~~^^^^^^
ValueError: could not broadcast input array from shape (0,) into shape (60,)
Am i doing everything correctly? Is the array formatted in the correct way? The embeddings have 1024 dimensions and there are 42695 embedded texts.
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 20
The test is done under
uvfor ARM machine w/ Sonoma, but we still cannot replicate this error.Thanks for providing us with the detailed information. Unfortunately, we don’t have the specific setup the same as your hardware to replicate your error. Thanks for the help from @gauravrparikh, we have replicated your pip freeze setup in the MacOS system, but it works fine. Based on the error message that you have provided, my guess would be some issues with the ANNOY package that the NN tree structure is not successfully constructed, which finally led to this error.
I would leave this issue open till we have figured out what is exactly happening from your side. However, if you would like to use PaCMAP and its variants, you can directly use google colab to run your code. It is working and tested successfully for the code that I have provided previously. Again, thanks for your interest in using PaCMAP.
Mine is ARM.
Could you please try to create a simple environment using
poetryoruv? Mamba/anaconda/whatever does a lot of magic under the hood, and are best avoided unless for research purposes (with poor software supply chain hygiene practices)