goofys: Unexpected "Operation not supported" returned when using sequential write

Hi,

I have a simple Java application for coping a 1GB file to S3 storage via goofys.

  FileInputStream fis = new FileInputStream(source);
  int bufferSize = 8 * 1024;
  try (FileOutputStream outputStream = new FileOutputStream(destination)){
            byte[] buffer = new byte[bufferSize];
            int n;
            while((n = fis.read(buffer)) > -1){
                outputStream.write(buffer, 0, n);
            }
   }

There is Exception when running outputStream.write

java.io.IOException: Operation not supported at java.io.FileOutputStream.writeBytes(Native Method) at java.io.FileOutputStream.write(FileOutputStream.java:326) at demo.rest.RestController.copyFileSimpleGet(RestController.java:80)

It is reported that it is caused by non sequential write operation “ERROR WriteFile: only sequential writes supported”

I cross checked with the system call used by the application using monitoring tool sysdig and I do not find any non sequential write operations java (32537:48) > write fd=1103(<f>/opt/goofys/assets/upload/1G.bin) size=8192 java (32537:48) < write res=8192 data=................................................................................ java (32537:48) > write fd=1103(<f>/opt/goofys/assets/upload/1G.bin) size=8192 java (32537:48) < write res=8192 data=................................................................................ java (32537:48) > write fd=1103(<f>/opt/goofys/assets/upload/1G.bin) size=8192 java (32537:48) < write res=8192 data=................................................................................ java (32537:48) > write fd=1103(<f>/opt/goofys/assets/upload/1G.bin) size=8192 java (32537:48) < write res=8192 data=................................................................................ java (32537:48) > write fd=1103(<f>/opt/goofys/assets/upload/1G.bin) size=8192 java (32537:48) < write res=8192 data=................................................................................ java (32537:48) > write fd=1103(<f>/opt/goofys/assets/upload/1G.bin) size=8192 java (32537:48) < write res=8192 data=................................................................................ java (32537:48) > write fd=1103(<f>/opt/goofys/assets/upload/1G.bin) size=8192 java (32537:48) < write res=8192 data=................................................................................ java (32537:48) > write fd=1103(<f>/opt/goofys/assets/upload/1G.bin) size=8192 java (32537:48) < write res=8192 data=................................................................................ java (32537:48) > write fd=1103(<f>/opt/goofys/assets/upload/1G.bin) size=8192 java (32537:48) < write res=8192 data=................................................................................ java (32537:48) > write fd=1103(<f>/opt/goofys/assets/upload/1G.bin) size=8192 java (32537:48) < write res=-95(EOPNOTSUPP) data=................................................................................ java (32537:48) > close fd=1103(<f>/opt/goofys/assets/upload/1G.bin) java (32537:48) < close res=-95(EOPNOTSUPP) java (32361:21) > close fd=1103(<f>/opt/goofys/assets/upload/1G.bin) java (32361:21) < close res=0

Do you know why “Operation not supported” is returned even the sequential write operations are ised? Is it a bug or anything can do to avoid this.

The latest version of goofys is used.

Thanks

Regards, Francis

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 1
Comments: 15 (8 by maintainers)

Commits related to this issue

Expose PID as metadata in CreateFile, OpenFile and FlushFile operations This will help us with kahing/goofys#273 — committed to kahing/fusego by dotslash 5 years ago
Update fuse submodule to kahing/fusego Fix for github.com/kahing/goofys/issues/273: If the PID that opened the file for write is not same as the one that is flushing a file, the FlushFile operation s... — committed to kahing/goofys by dotslash 5 years ago
Update fuse submodule to kahing/fusego Fix for github.com/kahing/goofys/issues/273: If the PID that opened the file for write is not same as the one that is flushing a file, the FlushFile operation s... — committed to yoskini/goofys by dotslash 5 years ago
FlushFile flushes only if the file operation is issued by the same process that opened the file Fixes kahing/goofys#273: If the TGID of the process that opened the file for write is not same as the o... — committed to yoskini/goofys by dotslash 5 years ago
Expose PID as metadata in CreateFile, OpenFile and FlushFile operations This will help us with kahing/goofys#273 rebase from master — committed to xinmeigui-db/fuse by dotslash 5 years ago
Add pid in all fuse ops (#92) Expose PID as metadata in CreateFile, OpenFile and FlushFile operations This will help us with kahing/goofys#273 Co-authored-by: Sai Teja Suram <pratap130492@gmail... — committed to jacobsa/fuse by xinmeigui-db 4 years ago

Most upvoted comments

For the sake of completion, let me describe the problem and fix

If a process forks it self while holding an open file, the forked process will inherit the file descriptor. So when the forked process exits, it issues a close on the open file. So single open file can have multiple FlushFile fuse operations.
This causes the write failure.

The fix was to ignore the FlushFile op if the opening process and the flushing process is different.

Kahing and I managed to reproduce the issue by making a small change to the program in https://github.com/kahing/goofys/issues/273#issuecomment-366152967. See the java program below. (You can also check the unit test added in the fix TestClientForkExec)

import java.io.*;
public class Cp {
    public static void main(String[] argv) throws Exception {
        String source = argv[0];
        String destination = argv[1];
        if (new File(destination).exists()) {
            throw new RuntimeException("File already exists");
        }

        FileInputStream fis = new FileInputStream(source);
        int bufferSize = 8 * 1024;
        try (FileOutputStream outputStream = new FileOutputStream(destination)){
            // !!!!!========= Run some command which causes fork + exec =====!!!!
            Runtime.getRuntime().exec("ls");
            byte[] buffer = new byte[bufferSize];
            int n;
            while((n = fis.read(buffer)) > -1){
                outputStream.write(buffer, 0, n);
            }
        }
    }
}

dotslash on Sep 5, 2019