ray: [release/core] data_processing_tests/dask_on_ray_large_scale_test.py fails due to object store
Output (last 1800 lines, everything before that was lost):
(pid=3251, ip=172.31.51.160) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3251, ip=172.31.51.160) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3251, ip=172.31.51.160)
(pid=3251, ip=172.31.51.160) During handling of the above exception, another exception occurred:
(pid=3251, ip=172.31.51.160)
(pid=3251, ip=172.31.51.160) Traceback (most recent call last):
(pid=3251, ip=172.31.51.160) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3251, ip=172.31.51.160) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3251, ip=172.31.51.160) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3251, ip=172.31.51.160) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3251, ip=172.31.51.160) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3251, ip=172.31.51.160) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=3251, ip=172.31.51.160) Traceback (most recent call last):
(pid=3251, ip=172.31.51.160) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3251, ip=172.31.51.160) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3251, ip=172.31.51.160) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3251, ip=172.31.51.160) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3251, ip=172.31.51.160) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=6550, ip=172.31.24.215) Traceback (most recent call last):
(pid=6550, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=6550, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=6550, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=6550, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=6550, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=6550, ip=172.31.24.215)
(pid=6550, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=6550, ip=172.31.24.215)
(pid=6550, ip=172.31.24.215) Traceback (most recent call last):
(pid=6550, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=6550, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=6550, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=6550, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=6550, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=6550, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=6550, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=6550, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=6550, ip=172.31.24.215)
(pid=6550, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=6550, ip=172.31.24.215)
(pid=6550, ip=172.31.24.215) Traceback (most recent call last):
(pid=6550, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=6550, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=6550, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=6550, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=6550, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=6550, ip=172.31.24.215) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=6550, ip=172.31.24.215) Traceback (most recent call last):
(pid=6550, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=6550, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=6550, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=6550, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=6550, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3247, ip=172.31.27.86) Traceback (most recent call last):
(pid=3247, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=3247, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=3247, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=3247, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3247, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=3247, ip=172.31.27.86)
(pid=3247, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=3247, ip=172.31.27.86)
(pid=3247, ip=172.31.27.86) Traceback (most recent call last):
(pid=3247, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=3247, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=3247, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=3247, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3247, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3247, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3247, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3247, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3247, ip=172.31.27.86)
(pid=3247, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=3247, ip=172.31.27.86)
(pid=3247, ip=172.31.27.86) Traceback (most recent call last):
(pid=3247, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3247, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3247, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3247, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3247, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3247, ip=172.31.27.86) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=3247, ip=172.31.27.86) Traceback (most recent call last):
(pid=3247, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3247, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3247, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3247, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3247, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3746, ip=172.31.27.86) Traceback (most recent call last):
(pid=3746, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=3746, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=3746, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=3746, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3746, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=3746, ip=172.31.27.86)
(pid=3746, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=3746, ip=172.31.27.86)
(pid=3746, ip=172.31.27.86) Traceback (most recent call last):
(pid=3746, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=3746, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=3746, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=3746, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3746, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3746, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3746, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3746, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3746, ip=172.31.27.86)
(pid=3746, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=3746, ip=172.31.27.86)
(pid=3746, ip=172.31.27.86) Traceback (most recent call last):
(pid=3746, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3746, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3746, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3746, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3746, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3746, ip=172.31.27.86) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=3746, ip=172.31.27.86) Traceback (most recent call last):
(pid=3746, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3746, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3746, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3746, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3746, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3227, ip=172.31.27.86) Traceback (most recent call last):
(pid=3227, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=3227, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=3227, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=3227, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3227, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=3227, ip=172.31.27.86)
(pid=3227, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=3227, ip=172.31.27.86)
(pid=3227, ip=172.31.27.86) Traceback (most recent call last):
(pid=3227, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=3227, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=3227, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=3227, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3227, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3227, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3227, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3227, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3227, ip=172.31.27.86)
(pid=3227, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=3227, ip=172.31.27.86)
(pid=3227, ip=172.31.27.86) Traceback (most recent call last):
(pid=3227, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3227, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3227, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3227, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3227, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3227, ip=172.31.27.86) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=3227, ip=172.31.27.86) Traceback (most recent call last):
(pid=3227, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3227, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3227, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3227, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3227, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=4785, ip=172.31.27.86) Traceback (most recent call last):
(pid=4785, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=4785, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=4785, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=4785, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=4785, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=4785, ip=172.31.27.86)
(pid=4785, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=4785, ip=172.31.27.86)
(pid=4785, ip=172.31.27.86) Traceback (most recent call last):
(pid=4785, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=4785, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=4785, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=4785, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=4785, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=4785, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=4785, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=4785, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=4785, ip=172.31.27.86)
(pid=4785, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=4785, ip=172.31.27.86)
(pid=4785, ip=172.31.27.86) Traceback (most recent call last):
(pid=4785, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=4785, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=4785, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=4785, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=4785, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=4785, ip=172.31.27.86) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=4785, ip=172.31.27.86) Traceback (most recent call last):
(pid=4785, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=4785, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=4785, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=4785, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=4785, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=4777, ip=172.31.27.86) Traceback (most recent call last):
(pid=4777, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=4777, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=4777, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=4777, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=4777, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=4777, ip=172.31.27.86)
(pid=4777, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=4777, ip=172.31.27.86)
(pid=4777, ip=172.31.27.86) Traceback (most recent call last):
(pid=4777, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=4777, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=4777, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=4777, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=4777, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=4777, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=4777, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=4777, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=4777, ip=172.31.27.86)
(pid=4777, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=4777, ip=172.31.27.86)
(pid=4777, ip=172.31.27.86) Traceback (most recent call last):
(pid=4777, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=4777, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=4777, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=4777, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=4777, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=4777, ip=172.31.27.86) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=4777, ip=172.31.27.86) Traceback (most recent call last):
(pid=4777, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=4777, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=4777, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=4777, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=4777, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=6547, ip=172.31.24.215) Traceback (most recent call last):
(pid=6547, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=6547, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=6547, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=6547, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=6547, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=6547, ip=172.31.24.215)
(pid=6547, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=6547, ip=172.31.24.215)
(pid=6547, ip=172.31.24.215) Traceback (most recent call last):
(pid=6547, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=6547, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=6547, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=6547, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=6547, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=6547, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=6547, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=6547, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=6547, ip=172.31.24.215)
(pid=6547, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=6547, ip=172.31.24.215)
(pid=6547, ip=172.31.24.215) Traceback (most recent call last):
(pid=6547, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=6547, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=6547, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=6547, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=6547, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=6547, ip=172.31.24.215) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=6547, ip=172.31.24.215) Traceback (most recent call last):
(pid=6547, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=6547, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=6547, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=6547, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=6547, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3773, ip=172.31.27.86) Traceback (most recent call last):
(pid=3773, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=3773, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=3773, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=3773, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3773, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=3773, ip=172.31.27.86)
(pid=3773, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=3773, ip=172.31.27.86)
(pid=3773, ip=172.31.27.86) Traceback (most recent call last):
(pid=3773, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=3773, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=3773, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=3773, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3773, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3773, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3773, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3773, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3773, ip=172.31.27.86)
(pid=3773, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=3773, ip=172.31.27.86)
(pid=3773, ip=172.31.27.86) Traceback (most recent call last):
(pid=3773, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3773, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3773, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3773, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3773, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3773, ip=172.31.27.86) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=3773, ip=172.31.27.86) Traceback (most recent call last):
(pid=3773, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3773, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3773, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3773, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3773, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3222, ip=172.31.27.86) Traceback (most recent call last):
(pid=3222, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=3222, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=3222, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=3222, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3222, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=3222, ip=172.31.27.86)
(pid=3222, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=3222, ip=172.31.27.86)
(pid=3222, ip=172.31.27.86) Traceback (most recent call last):
(pid=3222, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=3222, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=3222, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=3222, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3222, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3222, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3222, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3222, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3222, ip=172.31.27.86)
(pid=3222, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=3222, ip=172.31.27.86)
(pid=3222, ip=172.31.27.86) Traceback (most recent call last):
(pid=3222, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3222, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3222, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3222, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3222, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3222, ip=172.31.27.86) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=3222, ip=172.31.27.86) Traceback (most recent call last):
(pid=3222, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3222, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3222, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3222, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3222, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=4779, ip=172.31.27.86) Traceback (most recent call last):
(pid=4779, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=4779, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=4779, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=4779, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=4779, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=4779, ip=172.31.27.86)
(pid=4779, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=4779, ip=172.31.27.86)
(pid=4779, ip=172.31.27.86) Traceback (most recent call last):
(pid=4779, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=4779, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=4779, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=4779, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=4779, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=4779, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=4779, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=4779, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=4779, ip=172.31.27.86)
(pid=4779, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=4779, ip=172.31.27.86)
(pid=4779, ip=172.31.27.86) Traceback (most recent call last):
(pid=4779, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=4779, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=4779, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=4779, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=4779, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=4779, ip=172.31.27.86) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=4779, ip=172.31.27.86) Traceback (most recent call last):
(pid=4779, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=4779, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=4779, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=4779, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=4779, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3764, ip=172.31.27.86) Traceback (most recent call last):
(pid=3764, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=3764, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=3764, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=3764, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3764, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=3764, ip=172.31.27.86)
(pid=3764, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=3764, ip=172.31.27.86)
(pid=3764, ip=172.31.27.86) Traceback (most recent call last):
(pid=3764, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=3764, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=3764, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=3764, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3764, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3764, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3764, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3764, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3764, ip=172.31.27.86)
(pid=3764, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=3764, ip=172.31.27.86)
(pid=3764, ip=172.31.27.86) Traceback (most recent call last):
(pid=3764, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3764, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3764, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3764, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3764, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3764, ip=172.31.27.86) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=3764, ip=172.31.27.86) Traceback (most recent call last):
(pid=3764, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3764, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3764, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3764, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3764, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=5358, ip=172.31.27.86) Traceback (most recent call last):
(pid=5358, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=5358, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=5358, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=5358, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=5358, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=5358, ip=172.31.27.86)
(pid=5358, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=5358, ip=172.31.27.86)
(pid=5358, ip=172.31.27.86) Traceback (most recent call last):
(pid=5358, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=5358, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=5358, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=5358, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=5358, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=5358, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=5358, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=5358, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=5358, ip=172.31.27.86)
(pid=5358, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=5358, ip=172.31.27.86)
(pid=5358, ip=172.31.27.86) Traceback (most recent call last):
(pid=5358, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=5358, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=5358, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=5358, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=5358, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=5358, ip=172.31.27.86) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=5358, ip=172.31.27.86) Traceback (most recent call last):
(pid=5358, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=5358, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=5358, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=5358, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=5358, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=4786, ip=172.31.27.86) Traceback (most recent call last):
(pid=4786, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=4786, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=4786, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=4786, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=4786, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=4786, ip=172.31.27.86)
(pid=4786, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=4786, ip=172.31.27.86)
(pid=4786, ip=172.31.27.86) Traceback (most recent call last):
(pid=4786, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=4786, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=4786, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=4786, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=4786, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=4786, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=4786, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=4786, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=4786, ip=172.31.27.86)
(pid=4786, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=4786, ip=172.31.27.86)
(pid=4786, ip=172.31.27.86) Traceback (most recent call last):
(pid=4786, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=4786, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=4786, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=4786, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=4786, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=4786, ip=172.31.27.86) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=4786, ip=172.31.27.86) Traceback (most recent call last):
(pid=4786, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=4786, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=4786, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=4786, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=4786, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3749, ip=172.31.27.86) Traceback (most recent call last):
(pid=3749, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=3749, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=3749, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=3749, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3749, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=3749, ip=172.31.27.86)
(pid=3749, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=3749, ip=172.31.27.86)
(pid=3749, ip=172.31.27.86) Traceback (most recent call last):
(pid=3749, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=3749, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=3749, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=3749, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3749, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3749, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3749, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3749, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3749, ip=172.31.27.86)
(pid=3749, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=3749, ip=172.31.27.86)
(pid=3749, ip=172.31.27.86) Traceback (most recent call last):
(pid=3749, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3749, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3749, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3749, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3749, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3749, ip=172.31.27.86) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=3749, ip=172.31.27.86) Traceback (most recent call last):
(pid=3749, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3749, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3749, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3749, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3749, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3761, ip=172.31.27.86) Traceback (most recent call last):
(pid=3761, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=3761, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=3761, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=3761, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3761, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=3761, ip=172.31.27.86)
(pid=3761, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=3761, ip=172.31.27.86)
(pid=3761, ip=172.31.27.86) Traceback (most recent call last):
(pid=3761, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=3761, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=3761, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=3761, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3761, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3761, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3761, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3761, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3761, ip=172.31.27.86)
(pid=3761, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=3761, ip=172.31.27.86)
(pid=3761, ip=172.31.27.86) Traceback (most recent call last):
(pid=3761, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3761, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3761, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3761, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3761, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3761, ip=172.31.27.86) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=3761, ip=172.31.27.86) Traceback (most recent call last):
(pid=3761, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3761, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3761, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3761, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3761, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=4778, ip=172.31.27.86) Traceback (most recent call last):
(pid=4778, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=4778, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=4778, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=4778, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=4778, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=4778, ip=172.31.27.86)
(pid=4778, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=4778, ip=172.31.27.86)
(pid=4778, ip=172.31.27.86) Traceback (most recent call last):
(pid=4778, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=4778, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=4778, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=4778, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=4778, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=4778, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=4778, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=4778, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=4778, ip=172.31.27.86)
(pid=4778, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=4778, ip=172.31.27.86)
(pid=4778, ip=172.31.27.86) Traceback (most recent call last):
(pid=4778, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=4778, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=4778, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=4778, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=4778, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=4778, ip=172.31.27.86) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=4778, ip=172.31.27.86) Traceback (most recent call last):
(pid=4778, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=4778, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=4778, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=4778, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=4778, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=5038, ip=172.31.27.86) Traceback (most recent call last):
(pid=5038, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=5038, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=5038, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=5038, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=5038, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=5038, ip=172.31.27.86)
(pid=5038, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=5038, ip=172.31.27.86)
(pid=5038, ip=172.31.27.86) Traceback (most recent call last):
(pid=5038, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=5038, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=5038, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=5038, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=5038, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=5038, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=5038, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=5038, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=5038, ip=172.31.27.86)
(pid=5038, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=5038, ip=172.31.27.86)
(pid=5038, ip=172.31.27.86) Traceback (most recent call last):
(pid=5038, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=5038, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=5038, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=5038, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=5038, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=5038, ip=172.31.27.86) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=5038, ip=172.31.27.86) Traceback (most recent call last):
(pid=5038, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=5038, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=5038, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=5038, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=5038, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3223, ip=172.31.27.86) Traceback (most recent call last):
(pid=3223, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=3223, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=3223, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=3223, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3223, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=3223, ip=172.31.27.86)
(pid=3223, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=3223, ip=172.31.27.86)
(pid=3223, ip=172.31.27.86) Traceback (most recent call last):
(pid=3223, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=3223, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=3223, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=3223, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3223, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3223, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3223, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3223, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3223, ip=172.31.27.86)
(pid=3223, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=3223, ip=172.31.27.86)
(pid=3223, ip=172.31.27.86) Traceback (most recent call last):
(pid=3223, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3223, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3223, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3223, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3223, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3223, ip=172.31.27.86) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=3223, ip=172.31.27.86) Traceback (most recent call last):
(pid=3223, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3223, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3223, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3223, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3223, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3734, ip=172.31.27.86) Traceback (most recent call last):
(pid=3734, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=3734, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=3734, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=3734, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3734, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=3734, ip=172.31.27.86)
(pid=3734, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=3734, ip=172.31.27.86)
(pid=3734, ip=172.31.27.86) Traceback (most recent call last):
(pid=3734, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=3734, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=3734, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=3734, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3734, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3734, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3734, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3734, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3734, ip=172.31.27.86)
(pid=3734, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=3734, ip=172.31.27.86)
(pid=3734, ip=172.31.27.86) Traceback (most recent call last):
(pid=3734, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3734, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3734, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3734, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3734, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3734, ip=172.31.27.86) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=3734, ip=172.31.27.86) Traceback (most recent call last):
(pid=3734, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3734, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3734, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3734, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3734, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3766, ip=172.31.27.86) Traceback (most recent call last):
(pid=3766, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=3766, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=3766, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=3766, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3766, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=3766, ip=172.31.27.86)
(pid=3766, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=3766, ip=172.31.27.86)
(pid=3766, ip=172.31.27.86) Traceback (most recent call last):
(pid=3766, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=3766, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=3766, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=3766, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3766, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3766, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3766, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3766, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3766, ip=172.31.27.86)
(pid=3766, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=3766, ip=172.31.27.86)
(pid=3766, ip=172.31.27.86) Traceback (most recent call last):
(pid=3766, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3766, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3766, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3766, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3766, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3766, ip=172.31.27.86) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=3766, ip=172.31.27.86) Traceback (most recent call last):
(pid=3766, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3766, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3766, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3766, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3766, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3754, ip=172.31.27.86) Traceback (most recent call last):
(pid=3754, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=3754, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=3754, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=3754, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3754, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=3754, ip=172.31.27.86)
(pid=3754, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=3754, ip=172.31.27.86)
(pid=3754, ip=172.31.27.86) Traceback (most recent call last):
(pid=3754, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=3754, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=3754, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=3754, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3754, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3754, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3754, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3754, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3754, ip=172.31.27.86)
(pid=3754, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=3754, ip=172.31.27.86)
(pid=3754, ip=172.31.27.86) Traceback (most recent call last):
(pid=3754, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3754, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3754, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3754, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3754, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3754, ip=172.31.27.86) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=3754, ip=172.31.27.86) Traceback (most recent call last):
(pid=3754, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3754, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3754, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3754, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3754, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=5031, ip=172.31.27.86) Traceback (most recent call last):
(pid=5031, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=5031, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=5031, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=5031, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=5031, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=5031, ip=172.31.27.86)
(pid=5031, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=5031, ip=172.31.27.86)
(pid=5031, ip=172.31.27.86) Traceback (most recent call last):
(pid=5031, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=5031, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=5031, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=5031, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=5031, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=5031, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=5031, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=5031, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=5031, ip=172.31.27.86)
(pid=5031, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=5031, ip=172.31.27.86)
(pid=5031, ip=172.31.27.86) Traceback (most recent call last):
(pid=5031, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=5031, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=5031, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=5031, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=5031, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=5031, ip=172.31.27.86) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=5031, ip=172.31.27.86) Traceback (most recent call last):
(pid=5031, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=5031, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=5031, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=5031, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=5031, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=2715, ip=172.31.27.86) Traceback (most recent call last):
(pid=2715, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=2715, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=2715, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=2715, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=2715, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=2715, ip=172.31.27.86)
(pid=2715, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=2715, ip=172.31.27.86)
(pid=2715, ip=172.31.27.86) Traceback (most recent call last):
(pid=2715, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=2715, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=2715, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=2715, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=2715, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=2715, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=2715, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=2715, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=2715, ip=172.31.27.86)
(pid=2715, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=2715, ip=172.31.27.86)
(pid=2715, ip=172.31.27.86) Traceback (most recent call last):
(pid=2715, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=2715, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=2715, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=2715, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=2715, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=2715, ip=172.31.27.86) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=2715, ip=172.31.27.86) Traceback (most recent call last):
(pid=2715, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=2715, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=2715, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=2715, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=2715, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=4780, ip=172.31.27.86) Traceback (most recent call last):
(pid=4780, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=4780, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=4780, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=4780, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=4780, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=4780, ip=172.31.27.86)
(pid=4780, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=4780, ip=172.31.27.86)
(pid=4780, ip=172.31.27.86) Traceback (most recent call last):
(pid=4780, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=4780, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=4780, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=4780, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=4780, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=4780, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=4780, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=4780, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=4780, ip=172.31.27.86)
(pid=4780, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=4780, ip=172.31.27.86)
(pid=4780, ip=172.31.27.86) Traceback (most recent call last):
(pid=4780, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=4780, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=4780, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=4780, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=4780, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=4780, ip=172.31.27.86) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=4780, ip=172.31.27.86) Traceback (most recent call last):
(pid=4780, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=4780, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=4780, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=4780, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=4780, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=5382, ip=172.31.27.86) Traceback (most recent call last):
(pid=5382, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=5382, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=5382, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=5382, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=5382, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=5382, ip=172.31.27.86)
(pid=5382, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=5382, ip=172.31.27.86)
(pid=5382, ip=172.31.27.86) Traceback (most recent call last):
(pid=5382, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=5382, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=5382, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=5382, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=5382, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=5382, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=5382, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=5382, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=5382, ip=172.31.27.86)
(pid=5382, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=5382, ip=172.31.27.86)
(pid=5382, ip=172.31.27.86) Traceback (most recent call last):
(pid=5382, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=5382, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=5382, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=5382, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=5382, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=5382, ip=172.31.27.86) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=5382, ip=172.31.27.86) Traceback (most recent call last):
(pid=5382, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=5382, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=5382, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=5382, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=5382, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3736, ip=172.31.27.86) Traceback (most recent call last):
(pid=3736, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=3736, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=3736, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=3736, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3736, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=3736, ip=172.31.27.86)
(pid=3736, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=3736, ip=172.31.27.86)
(pid=3736, ip=172.31.27.86) Traceback (most recent call last):
(pid=3736, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=3736, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=3736, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=3736, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3736, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3736, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3736, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3736, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3736, ip=172.31.27.86)
(pid=3736, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=3736, ip=172.31.27.86)
(pid=3736, ip=172.31.27.86) Traceback (most recent call last):
(pid=3736, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3736, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3736, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3736, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3736, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3736, ip=172.31.27.86) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=3736, ip=172.31.27.86) Traceback (most recent call last):
(pid=3736, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3736, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3736, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3736, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3736, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3251, ip=172.31.27.86) Traceback (most recent call last):
(pid=3251, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=3251, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=3251, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=3251, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3251, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=3251, ip=172.31.27.86)
(pid=3251, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=3251, ip=172.31.27.86)
(pid=3251, ip=172.31.27.86) Traceback (most recent call last):
(pid=3251, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=3251, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=3251, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=3251, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3251, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3251, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3251, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3251, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3251, ip=172.31.27.86)
(pid=3251, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=3251, ip=172.31.27.86)
(pid=3251, ip=172.31.27.86) Traceback (most recent call last):
(pid=3251, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3251, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3251, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3251, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3251, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3251, ip=172.31.27.86) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=3251, ip=172.31.27.86) Traceback (most recent call last):
(pid=3251, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3251, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3251, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3251, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3251, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3225, ip=172.31.27.86) Traceback (most recent call last):
(pid=3225, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=3225, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=3225, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=3225, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3225, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=3225, ip=172.31.27.86)
(pid=3225, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=3225, ip=172.31.27.86)
(pid=3225, ip=172.31.27.86) Traceback (most recent call last):
(pid=3225, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=3225, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=3225, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=3225, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3225, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3225, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3225, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3225, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3225, ip=172.31.27.86)
(pid=3225, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=3225, ip=172.31.27.86)
(pid=3225, ip=172.31.27.86) Traceback (most recent call last):
(pid=3225, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3225, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3225, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3225, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3225, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3225, ip=172.31.27.86) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=3225, ip=172.31.27.86) Traceback (most recent call last):
(pid=3225, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3225, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3225, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3225, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3225, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3248, ip=172.31.27.86) Traceback (most recent call last):
(pid=3248, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=3248, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=3248, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=3248, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3248, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=3248, ip=172.31.27.86)
(pid=3248, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=3248, ip=172.31.27.86)
(pid=3248, ip=172.31.27.86) Traceback (most recent call last):
(pid=3248, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=3248, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=3248, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=3248, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3248, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3248, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3248, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3248, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3248, ip=172.31.27.86)
(pid=3248, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=3248, ip=172.31.27.86)
(pid=3248, ip=172.31.27.86) Traceback (most recent call last):
(pid=3248, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3248, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3248, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3248, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3248, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3248, ip=172.31.27.86) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=3248, ip=172.31.27.86) Traceback (most recent call last):
(pid=3248, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3248, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3248, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3248, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3248, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=5036, ip=172.31.27.86) Traceback (most recent call last):
(pid=5036, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=5036, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=5036, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=5036, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=5036, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=5036, ip=172.31.27.86)
(pid=5036, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=5036, ip=172.31.27.86)
(pid=5036, ip=172.31.27.86) Traceback (most recent call last):
(pid=5036, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=5036, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=5036, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=5036, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=5036, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=5036, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=5036, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=5036, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=5036, ip=172.31.27.86)
(pid=5036, ip=172.31.27.86) During handling of the above exception, another exception occurred:
(pid=5036, ip=172.31.27.86)
(pid=5036, ip=172.31.27.86) Traceback (most recent call last):
(pid=5036, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=5036, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=5036, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=5036, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=5036, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=5036, ip=172.31.27.86) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=5036, ip=172.31.27.86) Traceback (most recent call last):
(pid=5036, ip=172.31.27.86) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=5036, ip=172.31.27.86) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=5036, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=5036, ip=172.31.27.86) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=5036, ip=172.31.27.86) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=5071, ip=172.31.24.215) Traceback (most recent call last):
(pid=5071, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=5071, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=5071, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=5071, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=5071, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=5071, ip=172.31.24.215)
(pid=5071, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=5071, ip=172.31.24.215)
(pid=5071, ip=172.31.24.215) Traceback (most recent call last):
(pid=5071, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=5071, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=5071, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=5071, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=5071, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=5071, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=5071, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=5071, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=5071, ip=172.31.24.215)
(pid=5071, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=5071, ip=172.31.24.215)
(pid=5071, ip=172.31.24.215) Traceback (most recent call last):
(pid=5071, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=5071, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=5071, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=5071, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=5071, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=5071, ip=172.31.24.215) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=5071, ip=172.31.24.215) Traceback (most recent call last):
(pid=5071, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=5071, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=5071, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=5071, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=5071, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=4265, ip=172.31.24.215) Traceback (most recent call last):
(pid=4265, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=4265, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=4265, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=4265, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=4265, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=4265, ip=172.31.24.215)
(pid=4265, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=4265, ip=172.31.24.215)
(pid=4265, ip=172.31.24.215) Traceback (most recent call last):
(pid=4265, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=4265, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=4265, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=4265, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=4265, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=4265, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=4265, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=4265, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=4265, ip=172.31.24.215)
(pid=4265, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=4265, ip=172.31.24.215)
(pid=4265, ip=172.31.24.215) Traceback (most recent call last):
(pid=4265, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=4265, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=4265, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=4265, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=4265, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=4265, ip=172.31.24.215) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=4265, ip=172.31.24.215) Traceback (most recent call last):
(pid=4265, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=4265, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=4265, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=4265, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=4265, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=4266, ip=172.31.24.215) Traceback (most recent call last):
(pid=4266, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=4266, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=4266, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=4266, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=4266, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=4266, ip=172.31.24.215)
(pid=4266, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=4266, ip=172.31.24.215)
(pid=4266, ip=172.31.24.215) Traceback (most recent call last):
(pid=4266, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=4266, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=4266, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=4266, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=4266, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=4266, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=4266, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=4266, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=4266, ip=172.31.24.215)
(pid=4266, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=4266, ip=172.31.24.215)
(pid=4266, ip=172.31.24.215) Traceback (most recent call last):
(pid=4266, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=4266, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=4266, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=4266, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=4266, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=4266, ip=172.31.24.215) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=4266, ip=172.31.24.215) Traceback (most recent call last):
(pid=4266, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=4266, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=4266, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=4266, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=4266, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=4271, ip=172.31.24.215) Traceback (most recent call last):
(pid=4271, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=4271, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=4271, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=4271, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=4271, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=4271, ip=172.31.24.215)
(pid=4271, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=4271, ip=172.31.24.215)
(pid=4271, ip=172.31.24.215) Traceback (most recent call last):
(pid=4271, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=4271, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=4271, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=4271, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=4271, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=4271, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=4271, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=4271, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=4271, ip=172.31.24.215)
(pid=4271, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=4271, ip=172.31.24.215)
(pid=4271, ip=172.31.24.215) Traceback (most recent call last):
(pid=4271, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=4271, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=4271, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=4271, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=4271, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=4271, ip=172.31.24.215) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=4271, ip=172.31.24.215) Traceback (most recent call last):
(pid=4271, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=4271, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=4271, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=4271, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=4271, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3688, ip=172.31.24.215) Traceback (most recent call last):
(pid=3688, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=3688, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=3688, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=3688, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3688, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=3688, ip=172.31.24.215)
(pid=3688, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=3688, ip=172.31.24.215)
(pid=3688, ip=172.31.24.215) Traceback (most recent call last):
(pid=3688, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=3688, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=3688, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=3688, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3688, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3688, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3688, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3688, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3688, ip=172.31.24.215)
(pid=3688, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=3688, ip=172.31.24.215)
(pid=3688, ip=172.31.24.215) Traceback (most recent call last):
(pid=3688, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3688, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3688, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3688, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3688, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3688, ip=172.31.24.215) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=3688, ip=172.31.24.215) Traceback (most recent call last):
(pid=3688, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3688, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3688, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3688, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3688, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3693, ip=172.31.24.215) Traceback (most recent call last):
(pid=3693, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=3693, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=3693, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=3693, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3693, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=3693, ip=172.31.24.215)
(pid=3693, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=3693, ip=172.31.24.215)
(pid=3693, ip=172.31.24.215) Traceback (most recent call last):
(pid=3693, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=3693, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=3693, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=3693, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3693, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3693, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3693, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3693, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3693, ip=172.31.24.215)
(pid=3693, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=3693, ip=172.31.24.215)
(pid=3693, ip=172.31.24.215) Traceback (most recent call last):
(pid=3693, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3693, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3693, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3693, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3693, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3693, ip=172.31.24.215) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=3693, ip=172.31.24.215) Traceback (most recent call last):
(pid=3693, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3693, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3693, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3693, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3693, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3708, ip=172.31.24.215) Traceback (most recent call last):
(pid=3708, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=3708, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=3708, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=3708, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3708, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=3708, ip=172.31.24.215)
(pid=3708, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=3708, ip=172.31.24.215)
(pid=3708, ip=172.31.24.215) Traceback (most recent call last):
(pid=3708, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=3708, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=3708, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=3708, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3708, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3708, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3708, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3708, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3708, ip=172.31.24.215)
(pid=3708, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=3708, ip=172.31.24.215)
(pid=3708, ip=172.31.24.215) Traceback (most recent call last):
(pid=3708, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3708, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3708, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3708, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3708, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3708, ip=172.31.24.215) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=3708, ip=172.31.24.215) Traceback (most recent call last):
(pid=3708, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3708, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3708, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3708, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3708, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3712, ip=172.31.24.215) Traceback (most recent call last):
(pid=3712, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=3712, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=3712, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=3712, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3712, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=3712, ip=172.31.24.215)
(pid=3712, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=3712, ip=172.31.24.215)
(pid=3712, ip=172.31.24.215) Traceback (most recent call last):
(pid=3712, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=3712, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=3712, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=3712, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3712, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3712, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3712, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3712, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3712, ip=172.31.24.215)
(pid=3712, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=3712, ip=172.31.24.215)
(pid=3712, ip=172.31.24.215) Traceback (most recent call last):
(pid=3712, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3712, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3712, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3712, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3712, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3712, ip=172.31.24.215) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=3712, ip=172.31.24.215) Traceback (most recent call last):
(pid=3712, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3712, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3712, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3712, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3712, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3711, ip=172.31.24.215) Traceback (most recent call last):
(pid=3711, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=3711, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=3711, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=3711, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3711, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=3711, ip=172.31.24.215)
(pid=3711, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=3711, ip=172.31.24.215)
(pid=3711, ip=172.31.24.215) Traceback (most recent call last):
(pid=3711, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=3711, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=3711, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=3711, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3711, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3711, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3711, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3711, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3711, ip=172.31.24.215)
(pid=3711, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=3711, ip=172.31.24.215)
(pid=3711, ip=172.31.24.215) Traceback (most recent call last):
(pid=3711, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3711, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3711, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3711, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3711, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3711, ip=172.31.24.215) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=3711, ip=172.31.24.215) Traceback (most recent call last):
(pid=3711, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3711, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3711, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3711, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3711, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3725, ip=172.31.24.215) Traceback (most recent call last):
(pid=3725, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=3725, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=3725, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=3725, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3725, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=3725, ip=172.31.24.215)
(pid=3725, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=3725, ip=172.31.24.215)
(pid=3725, ip=172.31.24.215) Traceback (most recent call last):
(pid=3725, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=3725, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=3725, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=3725, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3725, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3725, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3725, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3725, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3725, ip=172.31.24.215)
(pid=3725, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=3725, ip=172.31.24.215)
(pid=3725, ip=172.31.24.215) Traceback (most recent call last):
(pid=3725, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3725, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3725, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3725, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3725, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3725, ip=172.31.24.215) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=3725, ip=172.31.24.215) Traceback (most recent call last):
(pid=3725, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3725, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3725, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3725, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3725, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3723, ip=172.31.24.215) Traceback (most recent call last):
(pid=3723, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=3723, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=3723, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=3723, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3723, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=3723, ip=172.31.24.215)
(pid=3723, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=3723, ip=172.31.24.215)
(pid=3723, ip=172.31.24.215) Traceback (most recent call last):
(pid=3723, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=3723, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=3723, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=3723, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3723, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3723, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3723, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3723, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3723, ip=172.31.24.215)
(pid=3723, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=3723, ip=172.31.24.215)
(pid=3723, ip=172.31.24.215) Traceback (most recent call last):
(pid=3723, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3723, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3723, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3723, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3723, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3723, ip=172.31.24.215) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=3723, ip=172.31.24.215) Traceback (most recent call last):
(pid=3723, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3723, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3723, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3723, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3723, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3732, ip=172.31.24.215) Traceback (most recent call last):
(pid=3732, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=3732, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=3732, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=3732, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3732, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=3732, ip=172.31.24.215)
(pid=3732, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=3732, ip=172.31.24.215)
(pid=3732, ip=172.31.24.215) Traceback (most recent call last):
(pid=3732, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=3732, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=3732, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=3732, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3732, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3732, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3732, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3732, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3732, ip=172.31.24.215)
(pid=3732, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=3732, ip=172.31.24.215)
(pid=3732, ip=172.31.24.215) Traceback (most recent call last):
(pid=3732, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3732, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3732, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3732, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3732, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3732, ip=172.31.24.215) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=3732, ip=172.31.24.215) Traceback (most recent call last):
(pid=3732, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3732, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3732, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3732, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3732, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3716, ip=172.31.24.215) Traceback (most recent call last):
(pid=3716, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=3716, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=3716, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=3716, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3716, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=3716, ip=172.31.24.215)
(pid=3716, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=3716, ip=172.31.24.215)
(pid=3716, ip=172.31.24.215) Traceback (most recent call last):
(pid=3716, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=3716, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=3716, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=3716, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3716, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3716, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3716, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3716, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3716, ip=172.31.24.215)
(pid=3716, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=3716, ip=172.31.24.215)
(pid=3716, ip=172.31.24.215) Traceback (most recent call last):
(pid=3716, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3716, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3716, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3716, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3716, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3716, ip=172.31.24.215) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=3716, ip=172.31.24.215) Traceback (most recent call last):
(pid=3716, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3716, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3716, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3716, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3716, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3222, ip=172.31.24.215) Traceback (most recent call last):
(pid=3222, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=3222, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=3222, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=3222, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3222, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=3222, ip=172.31.24.215)
(pid=3222, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=3222, ip=172.31.24.215)
(pid=3222, ip=172.31.24.215) Traceback (most recent call last):
(pid=3222, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=3222, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=3222, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=3222, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3222, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3222, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3222, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3222, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3222, ip=172.31.24.215)
(pid=3222, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=3222, ip=172.31.24.215)
(pid=3222, ip=172.31.24.215) Traceback (most recent call last):
(pid=3222, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3222, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3222, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3222, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3222, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3222, ip=172.31.24.215) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=3222, ip=172.31.24.215) Traceback (most recent call last):
(pid=3222, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3222, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3222, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3222, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3222, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3221, ip=172.31.24.215) Traceback (most recent call last):
(pid=3221, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=3221, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=3221, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=3221, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3221, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=3221, ip=172.31.24.215)
(pid=3221, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=3221, ip=172.31.24.215)
(pid=3221, ip=172.31.24.215) Traceback (most recent call last):
(pid=3221, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=3221, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=3221, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=3221, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3221, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3221, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3221, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3221, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3221, ip=172.31.24.215)
(pid=3221, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=3221, ip=172.31.24.215)
(pid=3221, ip=172.31.24.215) Traceback (most recent call last):
(pid=3221, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3221, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3221, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3221, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3221, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3221, ip=172.31.24.215) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=3221, ip=172.31.24.215) Traceback (most recent call last):
(pid=3221, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3221, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3221, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3221, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3221, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3224, ip=172.31.24.215) Traceback (most recent call last):
(pid=3224, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=3224, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=3224, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=3224, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3224, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=3224, ip=172.31.24.215)
(pid=3224, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=3224, ip=172.31.24.215)
(pid=3224, ip=172.31.24.215) Traceback (most recent call last):
(pid=3224, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=3224, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=3224, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=3224, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3224, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3224, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3224, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3224, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3224, ip=172.31.24.215)
(pid=3224, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=3224, ip=172.31.24.215)
(pid=3224, ip=172.31.24.215) Traceback (most recent call last):
(pid=3224, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3224, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3224, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3224, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3224, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3224, ip=172.31.24.215) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=3224, ip=172.31.24.215) Traceback (most recent call last):
(pid=3224, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3224, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3224, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3224, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3224, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3227, ip=172.31.24.215) Traceback (most recent call last):
(pid=3227, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=3227, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=3227, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=3227, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3227, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=3227, ip=172.31.24.215)
(pid=3227, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=3227, ip=172.31.24.215)
(pid=3227, ip=172.31.24.215) Traceback (most recent call last):
(pid=3227, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=3227, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=3227, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=3227, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3227, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3227, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3227, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3227, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3227, ip=172.31.24.215)
(pid=3227, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=3227, ip=172.31.24.215)
(pid=3227, ip=172.31.24.215) Traceback (most recent call last):
(pid=3227, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3227, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3227, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3227, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3227, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3227, ip=172.31.24.215) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=3227, ip=172.31.24.215) Traceback (most recent call last):
(pid=3227, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3227, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3227, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3227, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3227, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3228, ip=172.31.24.215) Traceback (most recent call last):
(pid=3228, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=3228, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=3228, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=3228, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3228, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=3228, ip=172.31.24.215)
(pid=3228, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=3228, ip=172.31.24.215)
(pid=3228, ip=172.31.24.215) Traceback (most recent call last):
(pid=3228, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=3228, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=3228, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=3228, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3228, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3228, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3228, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3228, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3228, ip=172.31.24.215)
(pid=3228, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=3228, ip=172.31.24.215)
(pid=3228, ip=172.31.24.215) Traceback (most recent call last):
(pid=3228, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3228, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3228, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3228, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3228, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3228, ip=172.31.24.215) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=3228, ip=172.31.24.215) Traceback (most recent call last):
(pid=3228, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3228, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3228, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3228, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3228, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3234, ip=172.31.24.215) Traceback (most recent call last):
(pid=3234, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=3234, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=3234, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=3234, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3234, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=3234, ip=172.31.24.215)
(pid=3234, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=3234, ip=172.31.24.215)
(pid=3234, ip=172.31.24.215) Traceback (most recent call last):
(pid=3234, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=3234, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=3234, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=3234, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3234, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3234, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3234, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3234, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3234, ip=172.31.24.215)
(pid=3234, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=3234, ip=172.31.24.215)
(pid=3234, ip=172.31.24.215) Traceback (most recent call last):
(pid=3234, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3234, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3234, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3234, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3234, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3234, ip=172.31.24.215) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=3234, ip=172.31.24.215) Traceback (most recent call last):
(pid=3234, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3234, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3234, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3234, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3234, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3220, ip=172.31.24.215) Traceback (most recent call last):
(pid=3220, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=3220, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=3220, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=3220, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3220, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=3220, ip=172.31.24.215)
(pid=3220, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=3220, ip=172.31.24.215)
(pid=3220, ip=172.31.24.215) Traceback (most recent call last):
(pid=3220, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=3220, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=3220, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=3220, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3220, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3220, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3220, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3220, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3220, ip=172.31.24.215)
(pid=3220, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=3220, ip=172.31.24.215)
(pid=3220, ip=172.31.24.215) Traceback (most recent call last):
(pid=3220, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3220, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3220, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3220, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3220, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=3220, ip=172.31.24.215) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=3220, ip=172.31.24.215) Traceback (most recent call last):
(pid=3220, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=3220, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=3220, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=3220, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=3220, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=2701, ip=172.31.24.215) Traceback (most recent call last):
(pid=2701, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
(pid=2701, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
(pid=2701, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
(pid=2701, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=2701, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Connection reset by peer
(pid=2701, ip=172.31.24.215)
(pid=2701, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=2701, ip=172.31.24.215)
(pid=2701, ip=172.31.24.215) Traceback (most recent call last):
(pid=2701, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 597, in ray._raylet.task_execution_handler
(pid=2701, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 449, in ray._raylet.execute_task
(pid=2701, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 554, in ray._raylet.execute_task
(pid=2701, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=2701, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=2701, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=2701, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=2701, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=2701, ip=172.31.24.215)
(pid=2701, ip=172.31.24.215) During handling of the above exception, another exception occurred:
(pid=2701, ip=172.31.24.215)
(pid=2701, ip=172.31.24.215) Traceback (most recent call last):
(pid=2701, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=2701, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=2701, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=2701, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=2701, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(pid=2701, ip=172.31.24.215) Exception ignored in: 'ray._raylet.task_execution_handler'
(pid=2701, ip=172.31.24.215) Traceback (most recent call last):
(pid=2701, ip=172.31.24.215) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/utils.py", line 102, in push_error_to_driver
(pid=2701, ip=172.31.24.215) worker.core_worker.push_error(job_id, error_type, message, time.time())
(pid=2701, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 1690, in ray._raylet.CoreWorker.push_error
(pid=2701, ip=172.31.24.215) File "python/ray/_raylet.pyx", line 157, in ray._raylet.check_status
(pid=2701, ip=172.31.24.215) ray.exceptions.RaySystemError: System error: Broken pipe
(autoscaler +20m42s) Resized to 23904 CPUs.
(autoscaler +20m42s) Adding 1 nodes of type worker_node_t.
(raylet, ip=172.31.53.84) [2021-03-30 13:12:19,931 E 2266 2266] local_object_manager.cc:402: Failed to send restore spilled object request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:19,934 E 2266 2266] local_object_manager.cc:402: Failed to send restore spilled object request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:19,934 E 2266 2266] local_object_manager.cc:402: Failed to send restore spilled object request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:19,940 E 2266 2266] local_object_manager.cc:402: Failed to send restore spilled object request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:19,941 E 2266 2266] local_object_manager.cc:402: Failed to send restore spilled object request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:19,941 E 2266 2266] local_object_manager.cc:402: Failed to send restore spilled object request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:19,941 E 2266 2266] local_object_manager.cc:402: Failed to send restore spilled object request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:19,969 E 2266 2266] local_object_manager.cc:402: Failed to send restore spilled object request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,048 E 2266 2266] local_object_manager.cc:402: Failed to send restore spilled object request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,048 E 2266 2266] local_object_manager.cc:402: Failed to send restore spilled object request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,068 E 2266 2266] local_object_manager.cc:402: Failed to send restore spilled object request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,153 E 2266 2266] local_object_manager.cc:402: Failed to send restore spilled object request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,153 E 2266 2266] local_object_manager.cc:402: Failed to send restore spilled object request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,171 E 2266 2266] local_object_manager.cc:402: Failed to send restore spilled object request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,282 E 2266 2266] local_object_manager.cc:492: Failed to send delete spilled object request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,282 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,282 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,282 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,282 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,283 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,283 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,283 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,290 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,301 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,311 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,321 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,331 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,341 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,351 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,361 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,371 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,381 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,391 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,401 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,411 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,421 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,431 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,441 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,451 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,461 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,471 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,481 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,491 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,502 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,511 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,522 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,532 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,542 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,552 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,562 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,572 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,586 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,592 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,602 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,612 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,622 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,632 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,642 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,652 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,662 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,672 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,682 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,692 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,702 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
2021-03-30 13:12:25,162 WARNING worker.py:912 -- The driver may not be able to keep up with the stdout/stderr of the workers. To avoid forwarding logs to the driver, use 'ray.init(log_to_driver=False)'.
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,713 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,722 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,732 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,742 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,752 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,762 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,772 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,782 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,792 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,803 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,812 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,823 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,833 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,843 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,853 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,863 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,873 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,883 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,893 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,903 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,913 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,923 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,933 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,943 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,953 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,963 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,973 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,983 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:20,993 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:21,003 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:21,013 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:21,023 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:21,033 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:21,043 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:21,053 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:21,063 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:21,073 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:21,083 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:21,093 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:21,105 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:21,113 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:21,123 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:21,133 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:21,144 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:21,154 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:21,164 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:21,174 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(raylet, ip=172.31.53.84) [2021-03-30 13:12:21,184 E 2266 2266] local_object_manager.cc:243: Failed to send object spilling request: IOError: 14: failed to connect to all addresses
(autoscaler +20m42s) Resized to 23808 CPUs.
(autoscaler +20m42s) Restarting 1 nodes of type worker_node_t (lost contact with raylet).
Traceback (most recent call last):
File "workloads/dask_on_ray_large_scale_test.py", line 323, in <module>
main()
File "workloads/dask_on_ray_large_scale_test.py", line 318, in main
ray_scheduler=ray_dask_get,
File "workloads/dask_on_ray_large_scale_test.py", line 259, in save_all_xarrays
res = dask.compute(delayed_files, scheduler=ray_scheduler)
File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/dask/base.py", line 565, in compute
results = schedule(dsk, keys, **kwargs)
File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/util/dask/scheduler.py", line 123, in ray_dask_get
result = ray_get_unpack(object_refs)
File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/util/dask/scheduler.py", line 379, in ray_get_unpack
computed_result = ray.get(object_refs)
File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/_private/client_mode_hook.py", line 47, in wrapper
return func(*args, **kwargs)
File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/ray/worker.py", line 1435, in get
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError: ray::dask:save_xarray-1949dac3-3e21-4947-99f7-bcafd65eb737 (pid=4233, ip=172.31.56.177)
File "python/ray/_raylet.pyx", line 456, in ray._raylet.execute_task
File "python/ray/_raylet.pyx", line 477, in ray._raylet.execute_task
File "python/ray/_raylet.pyx", line 347, in ray._raylet.raise_if_dependency_failed
ray.exceptions.RayTaskError: ray::dask:finalize-3225d24e-06c3-48fc-985a-5ce2294cce94 (pid=3783, ip=172.31.21.61)
File "python/ray/_raylet.pyx", line 532, in ray._raylet.execute_task
File "python/ray/_raylet.pyx", line 533, in ray._raylet.execute_task
File "python/ray/_raylet.pyx", line 1591, in ray._raylet.CoreWorker.store_task_outputs
File "python/ray/_raylet.pyx", line 149, in ray._raylet.check_status
ray.exceptions.ObjectStoreFullError: Failed to put object 181c043258306766ffffffffffffffffffffffff0100000001000000 in object store because it is full. Object size is 8640021673 bytes.
The local object store is full of objects that are still in scope and cannot be evicted. Tip: Use the `ray memory` command to list active objects in the cluster.
cc @rkooo567
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 17 (17 by maintainers)
@krfricke I verified parameters in this PR succeed in the master. Can you try again with the same param? https://github.com/ray-project/ray/pull/15048
It is probably parameters for the script is too optimistic. Let me actually try running it by today and get back to you