cri-o: cri-o 1.12/master with devicemapper driver fails to remove pods

Description

Using cri-o v1.12.0 and master branch and devicemapper driver. When trying to remove a pod I got:

FATA[0000] removing the pod sandbox "d12ec82743668" failed: rpc error: code = Unknown desc = failed to remove pod sandbox d12ec82743668c07869212f771132e9f77cc1a5754b0258c51425f6c02b3a76b: invalid argument

This happens using runc or kata-runtime.

My storage configuration on /etc/crio/crio.conf:

# Storage driver used to manage the storage of images and containers. Please
# refer to containers-storage.conf(5) to see all available storage drivers.
storage_driver = "devicemapper"

# List to pass options to the storage driver. Please refer to
# containers-storage.conf(5) to see all available storage options.
storage_option = [
        "dm.directlvm_device=/dev/sdb",
        "dm.directlvm_device_force=true",
        "dm.thinp_percent=95",
        "dm.thinp_metapercent=1",
        "dm.thinp_autoextend_threshold=80",
        "dm.thinp_autoextend_percent=20"
]

Steps to reproduce the issue:

1. Build cri-o v1.12.0 or master
2. Configure to use devicemapper
3. Create a pod and then try to remove it.

Describe the results you received: The pod could not be removed:

fuentess@kata-k8s:~/go/src/github.com/kubernetes-incubator/cri-o/test$ sudo crictl pods
POD ID              CREATED             STATE               NAME                NAMESPACE           ATTEMPT
d12ec82743668       2 minutes ago       Ready               podsandbox1         redhat.test.crio    1
fuentess@kata-k8s:~/go/src/github.com/kubernetes-incubator/cri-o/test$ sudo crictl rmp d12ec82743668
FATA[0000] removing the pod sandbox "d12ec82743668" failed: rpc error: code = Unknown desc = failed to remove pod sandbox d12ec82743668c07869212f771132e9f77cc1a5754b0258c51425f6c02b3a76b: invalid argument
fuentess@kata-k8s:~/go/src/github.com/kubernetes-incubator/cri-o/test$ sudo crictl pods
POD ID              CREATED             STATE               NAME                NAMESPACE           ATTEMPT
d12ec82743668       2 minutes ago       NotReady            podsandbox1         redhat.test.crio    1

Logs from the crio journal:

Nov 01 14:48:48 kata-k8s crio[9931]: time="2018-11-01 14:48:48.508673679Z" level=debug msg="devmapper: deactivateDevice START(8a2682fc4b22b135a6a78528184d8fe0cbdcaa72d72a57d1d9b52320e45b36ae)"
Nov 01 14:48:48 kata-k8s crio[9931]: time="2018-11-01 14:48:48.508810679Z" level=debug msg="devmapper: removeDevice START(container-8:1-775614-8a2682fc4b22b135a6a78528184d8fe0cbdcaa72d72a57d1d9b52320e45b36ae)"
Nov 01 14:48:48 kata-k8s crio[9931]: time="2018-11-01 14:48:48.531688222Z" level=debug msg="devmapper: removeDevice END(container-8:1-775614-8a2682fc4b22b135a6a78528184d8fe0cbdcaa72d72a57d1d9b52320e45b36ae)"
Nov 01 14:48:48 kata-k8s crio[9931]: time="2018-11-01 14:48:48.531724123Z" level=debug msg="devmapper: deactivateDevice END(8a2682fc4b22b135a6a78528184d8fe0cbdcaa72d72a57d1d9b52320e45b36ae)"
Nov 01 14:48:48 kata-k8s crio[9931]: time="2018-11-01 14:48:48.531733123Z" level=debug msg="devmapper: UnmountDevice END(hash=8a2682fc4b22b135a6a78528184d8fe0cbdcaa72d72a57d1d9b52320e45b36ae)"
Nov 01 14:48:48 kata-k8s crio[9931]: time="2018-11-01 14:48:48.550445458Z" level=debug msg="unmounted container "d12ec82743668c07869212f771132e9f77cc1a5754b0258c51425f6c02b3a76b""
Nov 01 14:48:48 kata-k8s crio[9931]: time="2018-11-01 14:48:48.550506258Z" level=debug msg="devmapper: UnmountDevice START(hash=8a2682fc4b22b135a6a78528184d8fe0cbdcaa72d72a57d1d9b52320e45b36ae)"
Nov 01 14:48:48 kata-k8s crio[9931]: time="2018-11-01 14:48:48.550515158Z" level=debug msg="devmapper: Unmount(/var/lib/containers/storage/devicemapper/mnt/8a2682fc4b22b135a6a78528184d8fe0cbdcaa72d72a57d1d9b5232
0e45b36ae)"
Nov 01 14:48:48 kata-k8s crio[9931]: time="2018-11-01 14:48:48.550541258Z" level=debug msg="devmapper: UnmountDevice END(hash=8a2682fc4b22b135a6a78528184d8fe0cbdcaa72d72a57d1d9b52320e45b36ae)"
Nov 01 14:48:48 kata-k8s crio[9931]: time="2018-11-01 14:48:48.550550358Z" level=error msg="devmapper: Error unmounting device 8a2682fc4b22b135a6a78528184d8fe0cbdcaa72d72a57d1d9b52320e45b36ae: invalid argument"
Nov 01 14:48:48 kata-k8s crio[9931]: time="2018-11-01 14:48:48.550560458Z" level=debug msg="failed to delete pod sandbox "d12ec82743668c07869212f771132e9f77cc1a5754b0258c51425f6c02b3a76b": invalid argument"
Nov 01 14:49:03 kata-k8s crio[9931]: time="2018-11-01 14:49:03.225207330Z" level=debug msg="ListPodSandboxRequest &ListPodSandboxRequest{Filter:&PodSandboxFilter{Id:,State:nil,LabelSelector:map[string]string{},}
,}"

Describe the results you expected: Pod should be removed successfully

Additional information you deem important (e.g. issue happens only occasionally):

Output of crio --version:

fuentess@kata-k8s:~/go/src/github.com/kubernetes-incubator/cri-o/test$ crio --version
crio version 1.12.0-dev
commit: "f58419d6cf462070a0c3727ad2dc554ef151e832"

Additional environment details (AWS, VirtualBox, physical, etc.): Ubuntu 16.04 VM on Azure.