hail: Missing breeze function in Spark 2.1.0 : breeze.linalg.DenseVector$.canSetD()Lbreeze/generic/UFunc$InPlaceImpl2
Hi everyone, I’ve been trying to get Hail up and running on my laptop and our HPC cluster and I keep running into the same problem. The install goes fine, but when I run the tests it fails out on both my laptop and our cluster at the same point, here :
14:17:27.809 [ERROR] [system.err] hail: info: while writing: 14:17:27.809 [ERROR] [system.err] /tmp/testExportKT.tsv 14:17:27.810 [ERROR] [system.err] merge time: 7.677ms 14:17:28.591 [ERROR] [system.err] hail: info: Coerced sorted dataset 14:17:30.368 [ERROR] [system.err] .hail: info: Coerced sorted dataset 14:17:31.306 [ERROR] [system.err] … 14:17:31.904 [ERROR] [system.err] ================================================================== 14:17:31.905 [ERROR] [system.err] ERROR: test_dataset (hail.tests.ContextTests) 14:17:31.905 [ERROR] [system.err] ---------------------------------------------------------------------- 14:17:31.905 [ERROR] [system.err] Traceback (most recent call last): 14:17:31.905 [ERROR] [system.err] File “/scratch/PI/dpwall/computeEnvironments/hail/python/hail/tests.py”, line 181, in test_dataset 14:17:31.906 [ERROR] [system.err] sample2.grm(‘gcta-grm-bin’, ‘/tmp/sample2.grm’) 14:17:31.906 [ERROR] [system.err] File “/scratch/PI/dpwall/computeEnvironments/hail/python/hail/dataset.py”, line 1988, in grm 14:17:31.906 [ERROR] [system.err] self.hc._run_command(self, pargs) 14:17:31.906 [ERROR] [system.err] File “/scratch/PI/dpwall/computeEnvironments/hail/python/hail/context.py”, line 90, in _run_command 14:17:31.907 [ERROR] [system.err] raise_py4j_exception(e) 14:17:31.907 [ERROR] [system.err] File “/scratch/PI/dpwall/computeEnvironments/hail/python/hail/java.py”, line 87, in raise_py4j_exception 14:17:31.907 [ERROR] [system.err] raise FatalError(msg, e.java_exception) 14:17:31.908 [ERROR] [system.err] FatalError: NoSuchMethodError: breeze.linalg.DenseVector$.canSetD()Lbreeze/generic/UFunc$InPlaceImpl2; 14:17:31.908 [ERROR] [system.err] 14:17:31.908 [ERROR] [system.err] ---------------------------------------------------------------------- 14:17:31.908 [ERROR] [system.err] Ran 7 tests in 83.523s
I noticed that a previous issue #209 from early last year had the exact same issue in a different context where the function breeze.linalg.DenseVector$.canSetD()Lbreeze/generic/UFunc$InPlaceImpl2; didn’t exist. However, when I use spark 2.0.2, it is able to run almost all tests successfully. Perhaps a problem with the paths in spark 2.1.0?
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Comments: 15 (2 by maintainers)
Great, thank you so much for the work you’ve been doing, I think this is really going to help. Also, I’ve been spending a lot of time trying to figure out how to get hail working on given clusters given that a lot of money has already been spent by our lab and others on dedicated machines. It uses the Magpie package out of LNLL, but I put in some tweaks to make it run right on our HPC cluster which I imagine other people will encounter with theirs. Thanks again!
Alright, I’ll try to do that later this week. AWESOME work.
On Sun, Feb 26, 2017 at 11:15 AM, jbloom22 notifications@github.com wrote:
– Nathaniel T. Stockham Stanford University | Class of 2011 B.S. | Mathematical and Computational Science, Honors M.S. | Aeronautical and Astronautical Engineering Ph.D student | Computational Neuroscience stockham@stanford.edu