mpi4py: test_io.TestIOSelf failures on Fedora Rawhide i686

Seeing this on Fedora Rawhide i686:

+ mpiexec -np 1 python3 test/runtests.py -v --no-builddir
/builddir/build/BUILD/mpi4py-3.1.1/test/runtests.py:76: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
  from distutils.util import get_platform
[0@buildvm-x86-08.iad2.fedoraproject.org] Python 3.10 (/usr/bin/python3)
[0@buildvm-x86-08.iad2.fedoraproject.org] MPI 3.1 (Open MPI 4.1.1)
[0@buildvm-x86-08.iad2.fedoraproject.org] mpi4py 3.1.1 (/builddir/build/BUILDROOT/mpi4py-3.1.1-1.fc36.i386/usr/lib/python3.10/site-packages/openmpi/mpi4py)
--------------------------------------------------------------------------
The OSC pt2pt component does not support MPI_THREAD_MULTIPLE in this release.
Workarounds are to run on a single node, or to use a system with an RDMA
capable network such as Infiniband.
--------------------------------------------------------------------------
...
======================================================================
ERROR: testIReadIWrite (test_io.TestIOSelf)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mpi4py-3.1.1/test/test_io.py", line 124, in testIReadIWrite
    fh.Set_view(0, etype)
  File "mpi4py/MPI/File.pyx", line 215, in mpi4py.MPI.File.Set_view
mpi4py.MPI.Exception: MPI_ERR_ARG: invalid argument of some other kind
======================================================================
ERROR: testIReadIWriteAll (test_io.TestIOSelf)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mpi4py-3.1.1/test/test_io.py", line 302, in testIReadIWriteAll
    fh.Set_view(0, etype)
  File "mpi4py/MPI/File.pyx", line 215, in mpi4py.MPI.File.Set_view
mpi4py.MPI.Exception: MPI_ERR_ARG: invalid argument of some other kind
======================================================================
ERROR: testIReadIWriteAt (test_io.TestIOSelf)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mpi4py-3.1.1/test/test_io.py", line 75, in testIReadIWriteAt
    fh.Set_view(0, etype)
  File "mpi4py/MPI/File.pyx", line 215, in mpi4py.MPI.File.Set_view
mpi4py.MPI.Exception: MPI_ERR_ARG: invalid argument of some other kind
======================================================================
ERROR: testIReadIWriteAtAll (test_io.TestIOSelf)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mpi4py-3.1.1/test/test_io.py", line 227, in testIReadIWriteAtAll
    fh.Set_view(0, etype)
  File "mpi4py/MPI/File.pyx", line 215, in mpi4py.MPI.File.Set_view
mpi4py.MPI.Exception: MPI_ERR_ARG: invalid argument of some other kind
======================================================================
ERROR: testIReadIWriteShared (test_io.TestIOSelf)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mpi4py-3.1.1/test/test_io.py", line 176, in testIReadIWriteShared
    fh.Set_view(0, etype)
  File "mpi4py/MPI/File.pyx", line 215, in mpi4py.MPI.File.Set_view
mpi4py.MPI.Exception: MPI_ERR_ARG: invalid argument of some other kind
======================================================================
ERROR: testReadWrite (test_io.TestIOSelf)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mpi4py-3.1.1/test/test_io.py", line 97, in testReadWrite
    fh.Set_view(0, etype)
  File "mpi4py/MPI/File.pyx", line 215, in mpi4py.MPI.File.Set_view
mpi4py.MPI.Exception: MPI_ERR_ARG: invalid argument of some other kind
======================================================================
ERROR: testReadWriteAll (test_io.TestIOSelf)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mpi4py-3.1.1/test/test_io.py", line 276, in testReadWriteAll
    fh.Set_view(0, etype)
  File "mpi4py/MPI/File.pyx", line 215, in mpi4py.MPI.File.Set_view
mpi4py.MPI.Exception: MPI_ERR_ARG: invalid argument of some other kind
======================================================================
ERROR: testReadWriteAllBeginEnd (test_io.TestIOSelf)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mpi4py-3.1.1/test/test_io.py", line 329, in testReadWriteAllBeginEnd
    fh.Set_view(0, etype)
  File "mpi4py/MPI/File.pyx", line 215, in mpi4py.MPI.File.Set_view
mpi4py.MPI.Exception: MPI_ERR_ARG: invalid argument of some other kind
======================================================================
ERROR: testReadWriteAt (test_io.TestIOSelf)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mpi4py-3.1.1/test/test_io.py", line 53, in testReadWriteAt
    fh.Set_view(0, etype)
  File "mpi4py/MPI/File.pyx", line 215, in mpi4py.MPI.File.Set_view
mpi4py.MPI.Exception: MPI_ERR_ARG: invalid argument of some other kind
======================================================================
ERROR: testReadWriteAtAll (test_io.TestIOSelf)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mpi4py-3.1.1/test/test_io.py", line 203, in testReadWriteAtAll
    fh.Set_view(0, etype)
  File "mpi4py/MPI/File.pyx", line 215, in mpi4py.MPI.File.Set_view
mpi4py.MPI.Exception: MPI_ERR_ARG: invalid argument of some other kind
======================================================================
ERROR: testReadWriteAtAllBeginEnd (test_io.TestIOSelf)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mpi4py-3.1.1/test/test_io.py", line 252, in testReadWriteAtAllBeginEnd
    fh.Set_view(0, etype)
  File "mpi4py/MPI/File.pyx", line 215, in mpi4py.MPI.File.Set_view
mpi4py.MPI.Exception: MPI_ERR_ARG: invalid argument of some other kind
======================================================================
ERROR: testReadWriteOrdered (test_io.TestIOSelf)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mpi4py-3.1.1/test/test_io.py", line 355, in testReadWriteOrdered
    fh.Set_view(0, etype)
  File "mpi4py/MPI/File.pyx", line 215, in mpi4py.MPI.File.Set_view
mpi4py.MPI.Exception: MPI_ERR_ARG: invalid argument of some other kind
======================================================================
ERROR: testReadWriteOrderedBeginEnd (test_io.TestIOSelf)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mpi4py-3.1.1/test/test_io.py", line 379, in testReadWriteOrderedBeginEnd
    fh.Set_view(0, etype)
  File "mpi4py/MPI/File.pyx", line 215, in mpi4py.MPI.File.Set_view
mpi4py.MPI.Exception: MPI_ERR_ARG: invalid argument of some other kind
======================================================================
ERROR: testReadWriteShared (test_io.TestIOSelf)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mpi4py-3.1.1/test/test_io.py", line 151, in testReadWriteShared
    fh.Set_view(0, etype)
  File "mpi4py/MPI/File.pyx", line 215, in mpi4py.MPI.File.Set_view
mpi4py.MPI.Exception: MPI_ERR_ARG: invalid argument of some other kind
======================================================================
ERROR: testIReadIWrite (test_io.TestIOWorld)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mpi4py-3.1.1/test/test_io.py", line 124, in testIReadIWrite
    fh.Set_view(0, etype)
  File "mpi4py/MPI/File.pyx", line 215, in mpi4py.MPI.File.Set_view
mpi4py.MPI.Exception: MPI_ERR_ARG: invalid argument of some other kind
======================================================================
ERROR: testIReadIWriteAll (test_io.TestIOWorld)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mpi4py-3.1.1/test/test_io.py", line 302, in testIReadIWriteAll
    fh.Set_view(0, etype)
  File "mpi4py/MPI/File.pyx", line 215, in mpi4py.MPI.File.Set_view
mpi4py.MPI.Exception: MPI_ERR_ARG: invalid argument of some other kind
======================================================================
ERROR: testIReadIWriteAt (test_io.TestIOWorld)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mpi4py-3.1.1/test/test_io.py", line 75, in testIReadIWriteAt
    fh.Set_view(0, etype)
  File "mpi4py/MPI/File.pyx", line 215, in mpi4py.MPI.File.Set_view
mpi4py.MPI.Exception: MPI_ERR_ARG: invalid argument of some other kind
======================================================================
ERROR: testIReadIWriteAtAll (test_io.TestIOWorld)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mpi4py-3.1.1/test/test_io.py", line 227, in testIReadIWriteAtAll
    fh.Set_view(0, etype)
  File "mpi4py/MPI/File.pyx", line 215, in mpi4py.MPI.File.Set_view
mpi4py.MPI.Exception: MPI_ERR_ARG: invalid argument of some other kind
======================================================================
ERROR: testIReadIWriteShared (test_io.TestIOWorld)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mpi4py-3.1.1/test/test_io.py", line 176, in testIReadIWriteShared
    fh.Set_view(0, etype)
  File "mpi4py/MPI/File.pyx", line 215, in mpi4py.MPI.File.Set_view
mpi4py.MPI.Exception: MPI_ERR_ARG: invalid argument of some other kind
======================================================================
ERROR: testReadWrite (test_io.TestIOWorld)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mpi4py-3.1.1/test/test_io.py", line 97, in testReadWrite
    fh.Set_view(0, etype)
  File "mpi4py/MPI/File.pyx", line 215, in mpi4py.MPI.File.Set_view
mpi4py.MPI.Exception: MPI_ERR_ARG: invalid argument of some other kind
======================================================================
ERROR: testReadWriteAll (test_io.TestIOWorld)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mpi4py-3.1.1/test/test_io.py", line 276, in testReadWriteAll
    fh.Set_view(0, etype)
  File "mpi4py/MPI/File.pyx", line 215, in mpi4py.MPI.File.Set_view
mpi4py.MPI.Exception: MPI_ERR_ARG: invalid argument of some other kind
======================================================================
ERROR: testReadWriteAllBeginEnd (test_io.TestIOWorld)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mpi4py-3.1.1/test/test_io.py", line 329, in testReadWriteAllBeginEnd
    fh.Set_view(0, etype)
  File "mpi4py/MPI/File.pyx", line 215, in mpi4py.MPI.File.Set_view
mpi4py.MPI.Exception: MPI_ERR_ARG: invalid argument of some other kind
======================================================================
ERROR: testReadWriteAt (test_io.TestIOWorld)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mpi4py-3.1.1/test/test_io.py", line 53, in testReadWriteAt
    fh.Set_view(0, etype)
  File "mpi4py/MPI/File.pyx", line 215, in mpi4py.MPI.File.Set_view
mpi4py.MPI.Exception: MPI_ERR_ARG: invalid argument of some other kind
======================================================================
ERROR: testReadWriteAtAll (test_io.TestIOWorld)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mpi4py-3.1.1/test/test_io.py", line 203, in testReadWriteAtAll
    fh.Set_view(0, etype)
  File "mpi4py/MPI/File.pyx", line 215, in mpi4py.MPI.File.Set_view
mpi4py.MPI.Exception: MPI_ERR_ARG: invalid argument of some other kind
======================================================================
ERROR: testReadWriteAtAllBeginEnd (test_io.TestIOWorld)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mpi4py-3.1.1/test/test_io.py", line 252, in testReadWriteAtAllBeginEnd
    fh.Set_view(0, etype)
  File "mpi4py/MPI/File.pyx", line 215, in mpi4py.MPI.File.Set_view
mpi4py.MPI.Exception: MPI_ERR_ARG: invalid argument of some other kind
======================================================================
ERROR: testReadWriteOrdered (test_io.TestIOWorld)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mpi4py-3.1.1/test/test_io.py", line 355, in testReadWriteOrdered
    fh.Set_view(0, etype)
  File "mpi4py/MPI/File.pyx", line 215, in mpi4py.MPI.File.Set_view
mpi4py.MPI.Exception: MPI_ERR_ARG: invalid argument of some other kind
======================================================================
ERROR: testReadWriteOrderedBeginEnd (test_io.TestIOWorld)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mpi4py-3.1.1/test/test_io.py", line 379, in testReadWriteOrderedBeginEnd
    fh.Set_view(0, etype)
  File "mpi4py/MPI/File.pyx", line 215, in mpi4py.MPI.File.Set_view
mpi4py.MPI.Exception: MPI_ERR_ARG: invalid argument of some other kind
======================================================================
ERROR: testReadWriteShared (test_io.TestIOWorld)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mpi4py-3.1.1/test/test_io.py", line 151, in testReadWriteShared
    fh.Set_view(0, etype)
  File "mpi4py/MPI/File.pyx", line 215, in mpi4py.MPI.File.Set_view
mpi4py.MPI.Exception: MPI_ERR_ARG: invalid argument of some other kind
======================================================================
FAIL: testStruct4 (test_util_dtlib.TestUtilDTLib) (typecode='q')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mpi4py-3.1.1/test/test_util_dtlib.py", line 165, in testStruct4
    self.assertEqual(mt.extent, n*ex1)
AssertionError: 20 != 24
======================================================================
FAIL: testStruct4 (test_util_dtlib.TestUtilDTLib) (typecode='Q')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mpi4py-3.1.1/test/test_util_dtlib.py", line 165, in testStruct4
    self.assertEqual(mt.extent, n*ex1)
AssertionError: 20 != 24
======================================================================
FAIL: testStruct4 (test_util_dtlib.TestUtilDTLib) (typecode='d')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mpi4py-3.1.1/test/test_util_dtlib.py", line 165, in testStruct4
    self.assertEqual(mt.extent, n*ex1)
AssertionError: 20 != 24
======================================================================
FAIL: testStruct4 (test_util_dtlib.TestUtilDTLib) (typecode='g')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mpi4py-3.1.1/test/test_util_dtlib.py", line 165, in testStruct4
    self.assertEqual(mt.extent, n*ex1)
AssertionError: 28 != 36
======================================================================
FAIL: testStruct4 (test_util_dtlib.TestUtilDTLib) (typecode='D')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mpi4py-3.1.1/test/test_util_dtlib.py", line 165, in testStruct4
    self.assertEqual(mt.extent, n*ex1)
AssertionError: 36 != 40
======================================================================
FAIL: testStruct4 (test_util_dtlib.TestUtilDTLib) (typecode='G')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mpi4py-3.1.1/test/test_util_dtlib.py", line 165, in testStruct4
    self.assertEqual(mt.extent, n*ex1)
AssertionError: 52 != 60
======================================================================
FAIL: testStruct4 (test_util_dtlib.TestUtilDTLib) (typecode='i8')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mpi4py-3.1.1/test/test_util_dtlib.py", line 165, in testStruct4
    self.assertEqual(mt.extent, n*ex1)
AssertionError: 20 != 24
======================================================================
FAIL: testStruct4 (test_util_dtlib.TestUtilDTLib) (typecode='u8')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mpi4py-3.1.1/test/test_util_dtlib.py", line 165, in testStruct4
    self.assertEqual(mt.extent, n*ex1)
AssertionError: 20 != 24
======================================================================
FAIL: testStruct4 (test_util_dtlib.TestUtilDTLib) (typecode='f8')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/mpi4py-3.1.1/test/test_util_dtlib.py", line 165, in testStruct4
    self.assertEqual(mt.extent, n*ex1)
AssertionError: 20 != 24
----------------------------------------------------------------------
Ran 1385 tests in 41.098s
FAILED (failures=9, errors=28, skipped=191)

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 31 (29 by maintainers)

Most upvoted comments

yep, reported at https://github.com/open-mpi/ompi/issues/10287

+1
drew-parsons on Apr 17, 2022

I’ve applied patch https://github.com/mpi4py/mpi4py/commit/2d868d7794c930fc524cfe603eedd1cc4f46b56b to the debian mpi4py package 3.1.3-2. It allows i386 to pass CI tests (by skipping the test_io long double test with openmpi).

+1
drew-parsons on Apr 17, 2022

Just in case anyone put my words is doubt, I’m pasting below a trivial reproducer. The code below aborts on Debian 11 i386 with the system openmpi package. After commenting out the line using the MPI_LONG_DOUBLE datatype, the reproducer runs to completion and returns with success.

#include <mpi.h>

int main() {
  MPI_File fh;
  MPI_Init(NULL,NULL);
  MPI_File_set_errhandler(MPI_FILE_NULL,MPI_ERRORS_ARE_FATAL);
  MPI_File_open(MPI_COMM_WORLD, "/tmp/data", MPI_MODE_CREATE|MPI_MODE_WRONLY, MPI_INFO_NULL, &fh);
  MPI_File_set_size(fh, 0);
  MPI_File_set_view(fh, 0, MPI_FLOAT, MPI_FLOAT, "native", MPI_INFO_NULL);
  MPI_File_set_view(fh, 0, MPI_DOUBLE, MPI_DOUBLE, "native", MPI_INFO_NULL);
  MPI_File_set_view(fh, 0, MPI_LONG_DOUBLE, MPI_LONG_DOUBLE, "native", MPI_INFO_NULL);
  MPI_File_close(&fh);
  MPI_Finalize();
}
+1
dalcinl on Apr 15, 2022

@drew-parsons @opoplawski I finally figured it out. I had to fire up Debian 11 i386 on Genome Boxes to reproduce the failure.

As usual, the problem is not mpi4py or Python’s fault. Looks like Open MPI cannot handle long double [complex] datatypes in MPI I/O calls (or at least in MPI_File_set_view()) on i386.

I pushed a commit 2d868d7 to branch test/openmpi-io-i386-ldbl. I branched off the current master branch, but the patch should apply easily to sources from latest tarball release. My “fix” just disables using the long precision datatypes, as they are evidently broken in openmpi-4.x (and probably older releases), and I have absolutely zero interest in pursuing this issue upstream.

Hopping to move forward and close this issue for good…

+1
dalcinl on Apr 15, 2022

The disease seems to be metastasizing. Now test_io.TestIOSelf is segfaulting on amd64 with openmpi 4.1.3, https://ci.debian.net/data/autopkgtest/testing/amd64/m/mpi4py/20603810/log.gz

+1
drew-parsons on Apr 5, 2022

I suspect the MPI-IO failure here (test_io.TestIOSelf) is not related to the ROMIO segfault from https://github.com/open-mpi/ompi/issues/9432.

We’re now seeing the same error in Debian i386 tests (mpi4py 3.1.1-8 with openmpi 4.1.2~rc1-4)

https://ci.debian.net/data/autopkgtest/unstable/i386/m/mpi4py/15635903/log.gz https://ci.debian.net/data/autopkgtest/testing/i386/m/mpi4py/15626637/log.gz

+1
drew-parsons on Sep 30, 2021
Read more comments on GitHub
← mpi4py: MPICommExecutor & MPIPoolExecutor Freeze Indefinitely
lombok-intellij-plugin: Exception occurred on IntelliJ 2020.2 →