noobaa-core: s3 bucket to bucket copy operation failed, 1 GB files - Linking did not work

Environment info

• NooBaa Version: master-20210607
• Platform: OCP 4.6.16

Actual behavior

1. Copy operation from one bucket to another failed

Expected behavior

1. Copy operation should not fail

Steps to reproduce

1. Created 2 buckets - bucket-1, bucket-21.
2. Uploaded multiple 1GB files to s3://bucket-1
3. Attempted copy operation from s3://bucket-1 to s3://bucket-21

Few files got copied but got failure for others:

[root@hpo-node1 akshat]# s3u1 ls s3://bucket-1/
urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 's3-noobaa.apps.ocp-akshat-1.cp.fyre.ibm.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
2021-06-08 21:15:06 1024000000 small_chunk_1
2021-06-08 21:15:28 1024000000 small_chunk_1._read
2021-06-08 21:15:12 1024000000 small_chunk_10
2021-06-08 21:15:26 1024000000 small_chunk_10._read
2021-06-08 21:15:27 1024000000 small_chunk_1_read
2021-06-08 21:16:15 1024000000 small_chunk_2
2021-06-08 21:16:18 1024000000 small_chunk_2._read
2021-06-08 21:16:53 1024000000 small_chunk_3
2021-06-08 21:16:25 1024000000 small_chunk_3._read
2021-06-08 21:16:08 1024000000 small_chunk_4
2021-06-08 21:17:12 1024000000 small_chunk_4._read
2021-06-08 21:17:25 1024000000 small_chunk_5
2021-06-08 21:17:36 1024000000 small_chunk_5._read
2021-06-08 21:17:44 1024000000 small_chunk_6
2021-06-08 21:18:36 1024000000 small_chunk_6._read
2021-06-08 21:19:13 1024000000 small_chunk_7
2021-06-08 21:19:05 1024000000 small_chunk_7._read
2021-06-08 21:19:22 1024000000 small_chunk_8
2021-06-08 21:19:29 1024000000 small_chunk_8._read
2021-06-08 21:19:39 1024000000 small_chunk_9
2021-06-08 21:19:37 1024000000 small_chunk_9._read
[root@hpo-node1 akshat]# 
[root@hpo-node1 akshat]# 
[root@hpo-node1 akshat]# s3u1 ls s3://bucket-21/
urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 's3-noobaa.apps.ocp-akshat-1.cp.fyre.ibm.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
2021-06-08 21:30:55 1024000000 small_chunk_8
2021-06-08 21:30:55 1024000000 small_chunk_8._read
2021-06-08 21:30:55 1024000000 small_chunk_9
2021-06-08 21:30:55 1024000000 small_chunk_9._read

Command used to copy files: s3u1 cp --recursive s3://bucket-1 s3://bucket-21/

Errors:

copy failed: s3://bucket-1/small_chunk_9 to s3://bucket-21/small_chunk_9 An error occurred (NoSuchKey) when calling the CompleteMultipartUpload operation (reached max retries: 2): The specified key does not exist.
copy failed: s3://bucket-1/small_chunk_8 to s3://bucket-21/small_chunk_8 An error occurred (NoSuchKey) when calling the CompleteMultipartUpload operation (reached max retries: 2): The specified key does not exist.
copy failed: s3://bucket-1/small_chunk_9._read to s3://bucket-21/small_chunk_9._read An error occurred (NoSuchKey) when calling the CompleteMultipartUpload operation (reached max retries: 2): The specified key does not exist.

More information - Screenshots / Logs / Other output

Jun-9 4:29:35.711 [Endpoint/9]   [LOG] CONSOLE:: update_endpoint_stats. namespace_stats = []
Jun-9 4:30:05.205 [Endpoint/9]   [LOG] CONSOLE:: NamespaceFS: abort_object_upload /nsfs/nsfs-nsr-1/bucket-21/.noobaa-nsfs_60c0406ba0157e00233de2dc/multipart-uploads/f989c13a-82c4-40e7-b1ba-71162a3
def85
Jun-9 4:30:05.897 [Endpoint/9] [ERROR] CONSOLE::  [Error: No such file or directory] { code: 'ENOENT' }
Jun-9 4:30:05.897 [Endpoint/9] [ERROR] CONSOLE::  [Error: No such file or directory] { code: 'ENOENT' }
Jun-9 4:30:05.995 [Endpoint/9] [ERROR] core.endpoint.s3.s3_rest:: S3 ERROR <?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><R
esource>/bucket-21/small_chunk_4._read?uploadId=f989c13a-82c4-40e7-b1ba-71162a3def85</Resource><RequestId>kpoz3eu6-6rwwuk-1aal</RequestId></Error> POST /bucket-21/small_chunk_4._read?uploadId=f989
c13a-82c4-40e7-b1ba-71162a3def85 {"accept-encoding":"identity","user-agent":"aws-cli/2.1.29 Python/3.8.8 Linux/4.18.0-305.3.1.el8_4.x86_64 exe/x86_64.rhel.8 prompt/off command/s3.cp","x-amz-date":
"20210609T042933Z","x-amz-content-sha256":"cc82905fa5a8bc75908709678e6972231fe14ce147ac9ef3611ec94ac4575f9f","authorization":"AWS4-HMAC-SHA256 Credential=YrpJ6lYpmcU6u8ODljQG/20210609/us-east-1/s3
/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=fad4887a005b0cd9e9fe190aadbbc283719e3d4af452b41385d8c04b97cf428e","content-length":"11641","host":"s3-noobaa.apps.ocp-a
kshat-1.cp.fyre.ibm.com","x-forwarded-host":"s3-noobaa.apps.ocp-akshat-1.cp.fyre.ibm.com","x-forwarded-port":"443","x-forwarded-proto":"https","forwarded":"for=10.17.66.193;host=s3-noobaa.apps.ocp
-akshat-1.cp.fyre.ibm.com;proto=https","x-forwarded-for":"10.17.66.193"} Error: No such file or directory
Jun-9 4:30:05.996 [Endpoint/9] [ERROR] core.endpoint.s3.s3_rest:: S3 ERROR <?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><R
esource>/bucket-21/small_chunk_4._read?uploadId=f989c13a-82c4-40e7-b1ba-71162a3def85</Resource><RequestId>kpoz2p77-313rnn-1076</RequestId></Error> POST /bucket-21/small_chunk_4._read?uploadId=f989
c13a-82c4-40e7-b1ba-71162a3def85 {"accept-encoding":"identity","user-agent":"aws-cli/2.1.29 Python/3.8.8 Linux/4.18.0-305.3.1.el8_4.x86_64 exe/x86_64.rhel.8 prompt/off command/s3.cp","x-amz-date":
"20210609T042901Z","x-amz-content-sha256":"cc82905fa5a8bc75908709678e6972231fe14ce147ac9ef3611ec94ac4575f9f","authorization":"AWS4-HMAC-SHA256 Credential=YrpJ6lYpmcU6u8ODljQG/20210609/us-east-1/s3
/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=886bd10e1ca6928f0e01cb53ed17f8675de6e6d42d8ab351484e2dbf3f25c192","content-length":"11641","host":"s3-noobaa.apps.ocp-a
kshat-1.cp.fyre.ibm.com","x-forwarded-host":"s3-noobaa.apps.ocp-akshat-1.cp.fyre.ibm.com","x-forwarded-port":"443","x-forwarded-proto":"https","forwarded":"for=10.17.66.193;host=s3-noobaa.apps.ocp
-akshat-1.cp.fyre.ibm.com;proto=https","x-forwarded-for":"10.17.66.193"} Error: No such file or directory
Jun-9 4:30:12.901 [Endpoint/9]   [LOG] CONSOLE:: NamespaceFS: abort_object_upload /nsfs/nsfs-nsr-1/bucket-21/.noobaa-nsfs_60c0406ba0157e00233de2dc/multipart-uploads/c33a6269-c625-4d62-9d9c-1b34258
23bd3
Jun-9 4:30:13.298 [Endpoint/9] [ERROR] CONSOLE::  [Error: No such file or directory] { code: 'ENOENT' }