b2_fuse: 503 errors not handled

Hi,

According to Backblaze it is common to receive a 503 response to a request if a storage pod is busy. From this page on uploading:

// Try several times to upload the file.  It's normal
// for uploads to fail if the target storage pod is
// too busy.  It's also normal (but infrequent) to get
// a 429 Too Many Requests if you are uploading a LOT
// of files.

Here is the output from b2_fuse on such a response:

Headers: {'Content-Type': 'b2/x-auto', 'X-Bz-File-Name': 'data/0/839', 'X-Bz-Content-Sha1': '02894f9352d498d39517c01c2557cd8555a0bb4d', 'Content-Length': '13
99322', 'Authorization': '***********************************************************'}

{
  "code": "service_unavailable",
  "message": "c001_v0001001_t0037 is too busy",
  "status": 503
}
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 495, in _wrapper
    return func(*args, **kwargs) or 0
  File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 621, in flush
    return self.operations('flush', self._decode_optional_path(path), fh)
  File "/usr/local/lib/python2.7/dist-packages/fuse.py", line 800, in __call__
    return getattr(self, op)(*args)
  File "b2fuse.py", line 456, in flush
    self.bucket.put_file(path, self.open_files[path])
  File "/home/mystx/b2_fuse/b2bucket_cached.py", line 149, in put_file
  File "/home/mystx/b2_fuse/b2bucket.py", line 148, in _put_file
  File "/home/mystx/b2_fuse/b2_python_pusher.py", line 71, in __enter__
SystemExit: 1

This causes my backup program to stop. Is there a way to get b2_fuse to retry?

Thanks

About this issue

Original URL
State: closed
Created 8 years ago
Comments: 30 (29 by maintainers)

Commits related to this issue

Release of b2_fuse version 1.3 potential fix for issues #16 #14 #8 — committed to falense/b2_fuse by falense 8 years ago

Most upvoted comments

Hi @sondree, b2 CLI developer here. When you started this project, b2 CLI was glued together with a console tool, which used print and sys.exit(1) if case of any error. Therefore it was unusable for the purpose of b2_fuse and you implemented the communication with B2 servers by yourself.

Since then, b2 CLI has changed a lot. Now the console tool is clearly separated from the API layer which is specifically designed for being used in other python programs such as filesystems.

Using b2 CLI API interface would give you several advantages over a custom built implementation of the communication layer:

transient errors such as 503, 429, timeouts etc are retried automatically with the exponential backoff required by the recent interface change
if the remote api changes, b2 CLI team will adapt the python API interface, so you’d just need to change the version of the CLI that b2_fuse depends on
we use large file upload api when needed, we use threads etc

ppolewicz on Sep 14, 2016

Just a status update. Work is under way to convert B2Fuse to use B2 Command Line Tool as a backend. This will resolve this issue as B2 Command Line Tool gracefully handles retries and errors. See https://github.com/Backblaze/B2_Command_Line_Tool/issues/250 for details.

falense on Oct 17, 2016

@ppolewicz This is a really good suggestion. I will look into it.

falense on Sep 15, 2016