locust: Unhandled exception: stream ended at an unexpected time

Describe the bug

I running a test pulling large files (some > 250mb) using the fast client, when connection gets dropped (“stream ended at an unexpected time”) on some files it throws an unhandled exception and does not log a failure.

Exception: Traceback (most recent call last): File “/home/mkarlovich/.pyenv/versions/3.8.2/lib/python3.8/site-packages/geventhttpclient/useragent.py”, line 184, in content return self._cached_content AttributeError: _cached_content

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File “/home/mkarlovich/.pyenv/versions/3.8.2/lib/python3.8/site-packages/locust/user/task.py”, line 284, in run self.execute_next_task() File “/home/mkarlovich/.pyenv/versions/3.8.2/lib/python3.8/site-packages/locust/user/task.py”, line 309, in execute_next_task self.execute_task(self._task_queue.pop(0)) File “/home/mkarlovich/.pyenv/versions/3.8.2/lib/python3.8/site-packages/locust/user/task.py”, line 416, in execute_task task(self.user) File “/home/mkarlovich/source/locust-plugins/examples/fast_csvreader_ex.py”, line 17, in index self.client.get(url, headers=headers) File “/home/mkarlovich/.pyenv/versions/3.8.2/lib/python3.8/site-packages/locust/contrib/fasthttp.py”, line 272, in get return self.request(“GET”, path, **kwargs) File “/home/mkarlovich/.pyenv/versions/3.8.2/lib/python3.8/site-packages/locust/contrib/fasthttp.py”, line 237, in request request_meta[“content_size”] = len(response.content or “”) File “/home/mkarlovich/.pyenv/versions/3.8.2/lib/python3.8/site-packages/geventhttpclient/useragent.py”, line 186, in content self._cached_content = self._content() File “/home/mkarlovich/.pyenv/versions/3.8.2/lib/python3.8/site-packages/locust/contrib/fasthttp.py”, line 340, in _content return super(FastResponse, self)._content() File “/home/mkarlovich/.pyenv/versions/3.8.2/lib/python3.8/site-packages/geventhttpclient/useragent.py”, line 201, in _content ret = self._response.read() File “/home/mkarlovich/.pyenv/versions/3.8.2/lib/python3.8/site-packages/geventhttpclient/response.py”, line 260, in read self.feed(data) _parser.HTTPParseError: (‘stream ended at an unexpected time’, 11)

Expected behavior

It should handle this exception and report it as a failure.

Actual behavior

Workers throw exception but no failure is recorded

Steps to reproduce

I can’t reproduce it except under load, code is using script below which constructs file download urls from csv file.

Environment

• OS: Ubuntu 16.04 master and workers
• Python version: 3.8.2
• Locust version: 1.0.3
• Locust command line that you ran:

locust -f fast_csvreader_ex.py --master --expect-workers 10 --headless -u 100 -r 1 -t 60m --csv=test100u60m10w locust -f fast_csvreader_ex.py --worker --master-host=10.66.11.21 & # run 5 per worker machine

• Locust file contents (anonymized if necessary):

from locust_plugins.csvreader import CSVReader
from locust import task
from locust.contrib.fasthttp import FastHttpUser
import traceback, logging, sys
import locust.stats
locust.stats.CSV_STATS_INTERVAL_SEC = 5

id_reader = CSVReader("ids.csv")

class MyUser(FastHttpUser):
    @task
    def index(self):
        headers = { "Authorization": "apikey", "Accept": "application/json" }
        feed_id, file_id = next(id_reader)
        url = f'/feeds/{feed_id}/files/{file_id}'
        self.client.get(url, headers=headers)
        
    host = 'https://my-test-sever.com'
    connection_timeout = 6000.0
    network_timeout = 6000.0