django-storages: S3Boto3Storage raises ValueError: I/O operation on closed file.

Using a custom StorageClass fix the issue :

from storages.backends.s3boto3 import S3Boto3Storage, SpooledTemporaryFile
import os


class CustomS3Boto3Storage(S3Boto3Storage):

    def _save_content(self, obj, content, parameters):
        """
        We create a clone of the content file as when this is passed to boto3 it wrongly closes
        the file upon upload where as the storage backend expects it to still be open
        """
        # Seek our content back to the start
        content.seek(0, os.SEEK_SET)

        # Create a temporary file that will write to disk after a specified size
        content_autoclose = SpooledTemporaryFile()

        # Write our original content into our copy that will be closed by boto3
        content_autoclose.write(content.read())

        # Upload the object which will auto close the content_autoclose instance
        super(CustomS3Boto3Storage, self)._save_content(obj, content_autoclose, parameters)

        # Cleanup if this is fixed upstream our duplicate should always close
        if not content_autoclose.closed:
            content_autoclose.close()

@pasevin Thank you! It doesn’t work because the method doesn’t return a result from the parent class. @voiddragon Here is the fixed version:

class CustomS3Boto3Storage(S3Boto3Storage, ABC):
    """
    This is our custom version of S3Boto3Storage that fixes a bug in
    boto3 where the passed in file is closed upon upload.
    From:
    https://github.com/matthewwithanm/django-imagekit/issues/391#issuecomment-275367006
    https://github.com/boto/boto3/issues/929
    https://github.com/matthewwithanm/django-imagekit/issues/391
    """

    def _save(self, name, content):
        """
        We create a clone of the content file as when this is passed to
        boto3 it wrongly closes the file upon upload where as the storage
        backend expects it to still be open
        """
        # Seek our content back to the start
        content.seek(0, os.SEEK_SET)

        # Create a temporary file that will write to disk after a specified
        # size. This file will be automatically deleted when closed by
        # boto3 or after exiting the `with` statement if the boto3 is fixed
        with SpooledTemporaryFile() as content_autoclose:

            # Write our original content into our copy that will be closed by boto3
            content_autoclose.write(content.read())

            # Upload the object which will auto close the
            # content_autoclose instance
            return super(CustomS3Boto3Storage, self)._save(name, content_autoclose)

@jschneier Can you please specify what tag/release this is fixed in?

I still experience this issue in django-storages 1.7.1 .

Would it be possible to get a new release out?

Thank you to the kind souls who update this page. I visit after every major Django update to grab the latest workaround. You don’t have unit tests, CI, or release notes, but like migrating birds we know where to come. In the spirit of all the duct tape that holds the internet together, Bravo.

@charlesthk can you make a pull request with this to the master?

This looks like a better solution, can we possibly have a patch for this and new version? I know it’s because of s3transfer now, but this could be nice to have in django-storages.

For me, a patch like this (which does not involve copying) fixes the issues.

--- site-packages/storages/backends/s3boto3.py.orig     2020-07-31 11:46:19.309504453 +0300
+++ site-packages/storages/backends/s3boto3.py  2020-07-31 11:58:25.400758620 +0300
@@ -38,10 +38,16 @@


 boto3_version_info = tuple([int(i) for i in boto3_version.split('.')])


+# https://github.com/boto/s3transfer/issues/80#issuecomment-562356142
+class NonCloseableBufferedReader(io.BufferedReader):
+    def close(self):
+        self.flush()
+
+
 @deconstructible
 class S3Boto3StorageFile(File):

     """
     The default file object used by the S3Boto3Storage backend.
@@ -542,11 +548,12 @@
         obj = self.bucket.Object(encoded_name)
         if self.preload_metadata:
             self._entries[encoded_name] = obj

         content.seek(0, os.SEEK_SET)
-        obj.upload_fileobj(content, ExtraArgs=params)
+        with NonCloseableBufferedReader(content) as reader:
+            obj.upload_fileobj(reader, ExtraArgs=params)
         return cleaned_name

     def delete(self, name):
         name = self._normalize_name(self._clean_name(name))
         self.bucket.Object(self._encode_name(name)).delete()

@pasevin Thank you! It doesn’t work because the method doesn’t return a result from the parent class. @voiddragon Here is the fixed version:

class CustomS3Boto3Storage(S3Boto3Storage, ABC):
    """
    This is our custom version of S3Boto3Storage that fixes a bug in
    boto3 where the passed in file is closed upon upload.
    From:
    https://github.com/matthewwithanm/django-imagekit/issues/391#issuecomment-275367006
    https://github.com/boto/boto3/issues/929
    https://github.com/matthewwithanm/django-imagekit/issues/391
    """

    def _save(self, name, content):
        """
        We create a clone of the content file as when this is passed to
        boto3 it wrongly closes the file upon upload where as the storage
        backend expects it to still be open
        """
        # Seek our content back to the start
        content.seek(0, os.SEEK_SET)

        # Create a temporary file that will write to disk after a specified
        # size. This file will be automatically deleted when closed by
        # boto3 or after exiting the `with` statement if the boto3 is fixed
        with SpooledTemporaryFile() as content_autoclose:

            # Write our original content into our copy that will be closed by boto3
            content_autoclose.write(content.read())

            # Upload the object which will auto close the
            # content_autoclose instance
            return super(CustomS3Boto3Storage, self)._save(name, content_autoclose)

Thanks! This fix works for django-storages version 1.9.1.

this issue should be reopened. I still get ValueError: I/O operation on closed file. And the custom class https://github.com/jschneier/django-storages/issues/382#issuecomment-592876060 solves it.

Django==2.2.12
django-storages==1.9.1
boto3==1.13.19
botocore==1.16.19

Pull request with @charlesthk’s solution is here: https://github.com/jschneier/django-storages/pull/717

@abhinavnair thx, downgrade to 1.5.0 helps

I just tried going back versions even further and at 1.5.0, this issue doesn’t occur. It occurs on every version later than 1.5.0. Moreover this issue only arises when I am trying to upload an image file after cropping it. For all other cases, there is no problem at all.

Here is a snippet for uploading a cropped image file:-

def decodeImage(dataurl):
    from django.core.files.temp import NamedTemporaryFile
    from django.core.files import File
    imgstr = re.search(r'base64,(.*)', dataurl).group(1)
    img_temp = NamedTemporaryFile(delete=True)
    img_temp.write(imgstr.decode('base64'))
    img_temp.flush()
    return File(img_temp)

def post(self, request, *args, **kwargs):
    dataurl = request.POST['cropped']
    new_pic = decodeImage(dataurl)
    from django.utils.timezone import now
    self.myobject.profile_picture.save(str(now()), new_pic)

Pull request opened for this fix here: https://github.com/jschneier/django-storages/pull/905

@voiddragon

Note: as others have found out there’s a bug in this rewrite, refer to https://github.com/jschneier/django-storages/issues/382#issuecomment-592876060 for a fixed version.

Here’s rewritten custom class:

class CustomS3Boto3Storage(S3Boto3Storage, ABC):
    """
    This is our custom version of S3Boto3Storage that fixes a bug in
    boto3 where the passed in file is closed upon upload.
    From:
    https://github.com/matthewwithanm/django-imagekit/issues/391#issuecomment-275367006
    https://github.com/boto/boto3/issues/929
    https://github.com/matthewwithanm/django-imagekit/issues/391
    """

    def _save(self, name, content):
        """
        We create a clone of the content file as when this is passed to
        boto3 it wrongly closes the file upon upload where as the storage
        backend expects it to still be open
        """
        # Seek our content back to the start
        content.seek(0, os.SEEK_SET)

        # Create a temporary file that will write to disk after a specified
        # size
        content_autoclose = SpooledTemporaryFile()

        # Write our original content into our copy that will be closed by boto3
        content_autoclose.write(content.read())

        # Upload the object which will auto close the content_autoclose
        # instance
        super(CustomS3Boto3Storage, self)._save(name, content_autoclose)

        # Cleanup if this is fixed upstream our duplicate should always close
        if not content_autoclose.closed:
            content_autoclose.close()

+1 on this, I also used charlesthk solution temporarily and it worked with:

django-storages==1.7.1
boto3==1.9.168
botocore==1.12.168
Django==1.11.21