aws-ebs-csi-driver: 0.8.1: Could not format gp3 drives with 1TB (works with smaller disk sizes)

/kind bug

What happened? When configuring a gp3 drive for 1TB we frequently hit the following (4 of 5 times)

  Warning  FailedMount  2m34s (x389 over 12h)  kubelet, ip-10-103-110-243.ec2.internal  MountVolume.MountDevice 
failed for volume "pvc-a6381ba8-b51c-4466-9503-ba2331e6826c" : rpc error: code =
 Internal desc = could not format "/dev/xvdbd" and mount it 
at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a6381ba8-b51c-4466-9503-ba2331e6826c/globalmount"

When we reduced the disk size to 512GB we had no issue

What you expected to happen? No formatting issue

How to reproduce it (as minimally and precisely as possible)? Provision a drive with 1TB and Max IOPS/Throughput

Anything else we need to know?:

Here’s our definition

  - name: storageclass
    namespace: kube-system
    release: aws-general-purpose-3-max-csi
    repo: cogito-stable
    state: present
    version: 1.0.0
    values:
      storage:
        allowVolumeExpansion: true
        annotations:
          storageclass.kubernetes.io/is-default-class: 'true'
        name: general-purpose-3-max-csi
        provisioner: ebs.csi.aws.com
        parameters:
          csi.storage.k8s.io/fstype: xfs
          encrypted: "true"
          iops: "16000"
          kmsKeyId: arn:aws:kms:us-east-1:xxxxxxx:alias/yyyyyyy
          throughput: "1000"
          type: gp3
        volumeBindingMode: WaitForFirstConsumer

Thanks for supporting gp3 so quickly

Environment

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.10", GitCommit:"575467a0eaf3ca1f20eb86215b3bde40a5ae617a", GitTreeState:"clean", BuildDate:"2019-12-11T12:41:00Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.12-eks-7684af", GitCommit:"7684af4ac41370dd109ac13817023cb8063e3d45", GitTreeState:"clean", BuildDate:"2020-10-20T22:57:40Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

• Driver version: 0.8.1

app.kubernetes.io/version=0.8.1                                                                                                                                                                                                                                                                                                                                                                                                                                                     
helm.sh/chart=aws-ebs-csi-driver-0.8.2