aspnetcore: Return 500 instead of 400 when temp directory isn't writeable

Is there an existing issue for this?

I have searched the existing issues

Describe the bug

I have just upgraded my Web App from 6.0.9 to 6.0.10 and am now getting a 400 Bad Request response when making large requests to my API. My web app is hosted on a Kubernetes container using Linux with a read-only root file system with the ASPNETCORE_TEMP environment variable to be a separate file system which is writable. If I make the root file system on the container writable then the problem does not occur.

The problem occurs when making requests which exceed 64KB and results in response below being returned to the client.

{type: "https://tools.ietf.org/html/rfc7231#section-6.5.1",…}
errors
: 
{"": ["Failed to read the request form. Read-only file system"]}
""
: 
["Failed to read the request form. Read-only file system"]
status
: 
400
title
: 
"One or more validation errors occurred."
traceId
: 
"00-c7f84b7e6c1e57951d1568eb3f94cab6-aeec3a125ea5243e-00"
type
: 
"https://tools.ietf.org/html/rfc7231#section-6.5.1"

On investigating the issue, I suspected that the problem was to do with large requests being written out to the file system and having looked into the code changes for 6.0.10 I suspect that the changes made in Use Path.GetTempFileName() for 600 Unix file mode’ could be the cause.

Some of the changes have introduced calls to Path.GetTemplFileName() which creates a temporary file in the temp folder of the root file system, and if the root file system is read-only then this will be a problem. For example, the method below retrieves the temp folder location, taking into account ASPNETCORE_TEMP, but then in the case of a non-Windows platform then calls Path.GetTemplFileName() which will use the root file system.

aspnetcore/src/Http/WebUtilities/src/FileBufferingWriteStream.cs

    [MemberNotNull(nameof(FileStream))]
    private void EnsureFileStream()
    {
        if (FileStream == null)
        {
            var tempFileDirectory = _tempFileDirectoryAccessor();
            var tempFileName = Path.Combine(tempFileDirectory, "ASPNETCORE_" + Guid.NewGuid() + ".tmp");

            // Create a temp file with the correct Unix file mode before moving it to the assigned tempFileName in the _tempFileDirectory.
            if (!RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
            {
                var tempTempFileName = Path.GetTempFileName();
                File.Move(tempTempFileName, tempFileName);
            }

            FileStream = new FileStream(
                tempFileName,
                FileMode.Create,
                FileAccess.Write,
                FileShare.Delete | FileShare.ReadWrite,
                bufferSize: 1,
                FileOptions.SequentialScan | FileOptions.DeleteOnClose);
        }
    }

Expected Behavior

I expect that by setting the temporary folder location via the ASPNETCORE_TEMP environment variable that all temporary files will be written to this folder, and so avoiding any errors caused by attempting to write to a read-only root file system.

Steps To Reproduce

Hopefully the code example I have shared is enough to go on to fix this.

Exceptions (if any)

I have not been able to locate any exceptions being raised for this problem - the only clue as to what the problem is is the message returned in the response ‘Failed to read the request form. Read-only file system’. Maybe because the problem occurs when trying to access the form data the error is treated as being a problem with the form data supplied by the client and so the exception is caught and sent back to the client as a ‘400 Bad Request’.

.NET Version

6.0.402

Anything else?

ASP.NET Core 6.0.10 Azure Kubernetes 1.23.12

About this issue

Original URL
State: open
Created 2 years ago
Reactions: 5
Comments: 23 (12 by maintainers)

Most upvoted comments

Are we going to write this topic up as guidance somewhere (more than a bug report)?

I think it’s the bare minimum we should do here. We can also describe workarounds (like doing it all in memory and paying that cost).

davidfowl on Jan 23, 2023

@tarunkumarrajak: As mentioned above, the workaround is to set the TMPDIR to a writable location. So you will need to add the following config to your Kubernetes Deployment YAML:

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
        env:
        - name: TMPDIR
          value: "/mnt/tmp"
        volumeMounts:
        - name: tmpfs
          mountPath: /mnt/tmp  
      volumes:
      - name: tmpfs
        emptyDir: {}

M1keF on Jan 5, 2023

@tarunkumarrajak: As mentioned above, the workaround is to set the TMPDIR to a writable location. So you will need to add the following config to your Kubernetes Deployment YAML:
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
        env:
        - name: TMPDIR
          value: "/mnt/tmp"
        volumeMounts:
        - name: tmpfs
          mountPath: /mnt/tmp  
      volumes:
      - name: tmpfs
        emptyDir: {}

Thanks for your help. @M1keF It worked

tarunkumarrajak on Jan 6, 2023

Note the 400 vs 500 is an aesthetic issue, the real problem is that the disk isn’t writable and that requires changes to your deployment environment. E.g. we can’t make this problem go away for you, but we can more clearly indicate what the problem is.

I don’t think that this is an aesthetic issue for two big reasons:

We’re having an information disclosure, it’s no good to inform the client about the state of the server (e.g., that there is no disk space left anymore).
Our monitoring systems don’t trigger alerts because 4xx are assumed to be client errors which can be ignored from an operations view.

So it’s a real issue for us, and we hope you can at least provide a workaround if you’re not backporting a fix.

Best regards, D.R.

drauch on Jan 5, 2023

I don’t know if I should create a separate ticket for it, but as an addition:

If there is not enough disk space to read the form data (e.g., huge file upload), also a 400 is returned (incl. the not-enough-diskspace information!) instead of a plain 500.

Best regards, D.R.

drauch on Dec 12, 2022

Thanks - I can confirm that the workaround works. For my use case, it leads me to question whether it would be better to just set TMPDIR to cover all eventualities and not use ASPNETCORE_TEMP.

Something I discovered whilst testing this out is that the response returned to the client is leaking out the location of the temp folder if it doesn’t exist - is there a setting to prevent this detail being returned? I would also expect a 500 Internal Server error in this case rather than a 400 Bad Request. I can raise this as a separate issue if appropriate.

{type: "https://tools.ietf.org/html/rfc7231#section-6.5.1",…}
errors
: 
{"": ["Failed to read the request form. /mnt/tmpz"]}
status
: 
400
title
: 
"One or more validation errors occurred."
traceId
: 
"00-7b8b047142c3b9c70d1106ad5f9498fe-a7a8da7c389c9047-00"
type
: 
"https://tools.ietf.org/html/rfc7231#section-6.5.1"

Thanks for your help.

M1keF on Oct 28, 2022