graphvite: undefined symbol: _ZN3fLS13FLAGS_log_dirE

Hi,

When I was trying to install it from source I got error when executing python setup.py install

The message is as follow:

Traceback (most recent call last):
  File "setup.py", line 22, in <module>
    from graphvite import __version__, lib_path, lib_file
  File "/home/husong/graphvite/python/graphvite/__init__.py", line 36, in <module>
    lib = imp.load_dynamic("libgraphvite", lib_file)
ImportError: /home/husong/graphvite/python/graphvite/../../build/lib/libgraphvite.so: undefined symbol: _ZN3fLS13FLAGS_log_dirE

Any idea why this is happening? How can I get around this problem?

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 17

Most upvoted comments

The issue continued to persist for me as well (like #89), I solved as follows:

conda create -n graphvite python=3.7
conda activate graphvite
conda install -c pytorch faiss-gpu=1.6.3 cudatoolkit=10.1 # note this will also install openmp (cf. intel-openmp-2021.3.0)
conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 -c pytorch
pip install pybind11
git clone https://github.com/DeepGraphLearning/graphvite.git
cd graphvite
nano conda/requirements.txt

Update as follows (some requirements commented out):

# cmake
cmake >=3.12
gxx_linux-64 >=5.4
glog
gflags
#cudatoolkit >=9.2
#python
#pybind11

# make
#mkl >=2018

# run
#numpy >=1.11
pyyaml
conda-forge::easydict
six
future
imageio
psutil
scipy
matplotlib
#pytorch
#torchvision
nltk

Install remaining dependencies and then install gaphvite from conda:

conda install -y --file conda/requirements.txt # might take a bit of time to complete
conda install -c milagraph graphvite

Please check if you see the following files:

ls /path/to/envs/graphvite/lib/python3.7/site-packages/graphvite
libfaiss.so  libgraphvite.so

At this point, open python:

>>> import graphvite
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/netscratch/samin/dev/miniconda3/envs/graphvite/lib/python3.7/site-packages/graphvite/__init__.py", line 36, in <module>
    lib = imp.load_dynamic("libgraphvite", lib_file)
  File "/netscratch/samin/dev/miniconda3/envs/graphvite/lib/python3.7/imp.py", line 342, in load_dynamic
    return _load(spec)
ImportError: /netscratch/samin/dev/miniconda3/envs/graphvite/lib/python3.7/site-packages/graphvite/lib/libgraphvite.so: undefined symbol: _ZN6google10LogMessageC1EPKciiiMS0_FvvE

And check: echo $LD_LIBRARY_PATH

/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64

Find libglog:

whereis libglog
libglog: /usr/lib/x86_64-linux-gnu/libglog.so /usr/lib/x86_64-linux-gnu/libglog.a
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/x86_64-linux-gnu

Open python

>>> import graphvite
>>> import graphvite.application as gap

From command line:

graphvite baseline quick start

running baseline: demo/quick_start.yaml
downloading https://www.dropbox.com/s/cf21ouuzd563cqx/BlogCatalog-dataset.zip?dl=1 to BlogCatalog-dataset.zip
extracting BlogCatalog-dataset/data/edges.csv from BlogCatalog-dataset.zip to edges.csv
converting edges.csv to blogcatalog_graph.txt
splitting graph blogcatalog_graph.txt into blogcatalog_train.txt, blogcatalog_valid.txt, blogcatalog_test.txt
extracting BlogCatalog-dataset/data/group-edges.csv from BlogCatalog-dataset.zip to group-edges.csv
converting group-edges.csv to blogcatalog_label.txt
loading graph from /root/.graphvite/dataset/blogcatalog/blogcatalog_train.txt
0.00018755%
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Graph<uint32>
------------------ Graph -------------------
#vertex: 10308, #edge: 327429
as undirected: yes, normalization: no
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[time] GraphApplication.load: 0.0584445 s
[time] GraphApplication.build: 1.36667 s
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
GraphSolver<128, float32, uint32>
----------------- Resource -----------------
#worker: 1, #sampler: 7, #partition: 1
tied weights: no, episode size: 500
gpu memory limit: 23.5 GiB
gpu memory cost: 51.5 MiB
----------------- Sampling -----------------
augmentation step: 2, shuffle base: 2
random walk length: 40
random walk batch size: 100
#negative: 1, negative sample exponent: 0.75
----------------- Training -----------------
model: LINE
optimizer: SGD
learning rate: 0.025, lr schedule: linear
weight decay: 0.005
#epoch: 2000, batch size: 100000
resume: no
positive reuse: 1, negative weight: 5
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Batch id: 0 / 6548
loss = 0
Batch id: 1000 / 6548
loss = 0.387899
Batch id: 2000 / 6548
loss = 0.383534
Batch id: 3000 / 6548
loss = 0.37939
Batch id: 4000 / 6548
loss = 0.375876
Batch id: 5000 / 6548
loss = 0.372967
Batch id: 6000 / 6548
loss = 0.37103
[time] GraphApplication.train: 19.9742 s
------------- link prediction --------------
effective edges: 6646 / 6650
effective filter edges: 327429 / 327429
remaining edges: 6646 / 6646
AUC: 0.904191
[time] GraphApplication.evaluate: 26.4254 s
----------- node classification ------------
effective labels: 14472 / 14476
macro-F1@20%: 0.243524
micro-F1@20%: 0.392344
[time] GraphApplication.evaluate: 32.8746 s
save model to `line_blogcatalog.pkl`

I did not have to comment out any part of the code or change the RUNPATH. However, at this point, I only noted one issue, I cannot import faiss and graphvite at same time and I presume this conflict is due to libfaiss.so provided in both faiss-gpu and graphvite from their respective conda installations and depending on the order of import, the relevant dynamic file is linked:

Python 3.7.11 (default, Jul 27 2021, 14:32:16)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import graphvite
>>> import faiss
Traceback (most recent call last):
  File "/netscratch/samin/dev/miniconda3/envs/snomed_kge/lib/python3.7/site-packages/faiss/swigfaiss_avx2.py", line 14, in swig_import_helper
    return importlib.import_module(mname)
  File "/netscratch/samin/dev/miniconda3/envs/snomed_kge/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 670, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 583, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1043, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: /netscratch/samin/dev/miniconda3/envs/snomed_kge/lib/python3.7/site-packages/faiss/_swigfaiss.so: undefined symbol: _ZNK5faiss5Index6assignElPKfPll

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/netscratch/samin/dev/miniconda3/envs/snomed_kge/lib/python3.7/site-packages/faiss/loader.py", line 31, in <module>
    from .swigfaiss_avx2 import *
  File "/netscratch/samin/dev/miniconda3/envs/snomed_kge/lib/python3.7/site-packages/faiss/swigfaiss_avx2.py", line 17, in <module>
    _swigfaiss = swig_import_helper()
  File "/netscratch/samin/dev/miniconda3/envs/snomed_kge/lib/python3.7/site-packages/faiss/swigfaiss_avx2.py", line 16, in swig_import_helper
    return importlib.import_module('_swigfaiss')
  File "/netscratch/samin/dev/miniconda3/envs/snomed_kge/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named '_swigfaiss'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/netscratch/samin/dev/miniconda3/envs/snomed_kge/lib/python3.7/site-packages/faiss/swigfaiss.py", line 14, in swig_import_helper
    return importlib.import_module(mname)
  File "/netscratch/samin/dev/miniconda3/envs/snomed_kge/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 670, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 583, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1043, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: /netscratch/samin/dev/miniconda3/envs/snomed_kge/lib/python3.7/site-packages/faiss/_swigfaiss.so: undefined symbol: _ZNK5faiss5Index6assignElPKfPll

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/netscratch/samin/dev/miniconda3/envs/snomed_kge/lib/python3.7/site-packages/faiss/__init__.py", line 17, in <module>
    from .loader import *
  File "/netscratch/samin/dev/miniconda3/envs/snomed_kge/lib/python3.7/site-packages/faiss/loader.py", line 39, in <module>
    from .swigfaiss import *
  File "/netscratch/samin/dev/miniconda3/envs/snomed_kge/lib/python3.7/site-packages/faiss/swigfaiss.py", line 17, in <module>
    _swigfaiss = swig_import_helper()
  File "/netscratch/samin/dev/miniconda3/envs/snomed_kge/lib/python3.7/site-packages/faiss/swigfaiss.py", line 16, in swig_import_helper
    return importlib.import_module('_swigfaiss')
  File "/netscratch/samin/dev/miniconda3/envs/snomed_kge/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named '_swigfaiss'
>>>

and other way around:

Python 3.7.11 (default, Jul 27 2021, 14:32:16)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import faiss
>>> import graphvite
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/netscratch/samin/dev/miniconda3/envs/snomed_kge/lib/python3.7/site-packages/graphvite/__init__.py", line 36, in <module>
    lib = imp.load_dynamic("libgraphvite", lib_file)
  File "/netscratch/samin/dev/miniconda3/envs/snomed_kge/lib/python3.7/imp.py", line 342, in load_dynamic
    return _load(spec)
ImportError: /netscratch/samin/dev/miniconda3/envs/snomed_kge/lib/python3.7/site-packages/graphvite/lib/libgraphvite.so: undefined symbol: _ZN5faiss3gpu14GpuIndexFlatL2C1EPNS0_12GpuResourcesEiNS0_18GpuIndexFlatConfigE
>>>

suamin on Oct 11, 2021

This issue persists even after installing intel-openmp=2019.4. Can someone please consolidate these requirements and update in requirements file or add necessary steps user should run?

nm-narasimha on Jan 25, 2020