longhorn: [BUG] [ARM] Longhorn refuses to create a PVC bigger than 1.9G
Describe the bug I cannot provision a PVC using longhorn storage class larger than 1.9G To Reproduce Apply the below manifests (from examples):
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: longhorn-volv-pvc
spec:
accessModes:
- ReadWriteOnce
storageClassName: longhorn
resources:
requests:
storage: 10Gi
---
apiVersion: v1
kind: Pod
metadata:
name: volume-test
namespace: default
spec:
restartPolicy: Always
containers:
- name: volume-test
image: nginx:stable-alpine
imagePullPolicy: IfNotPresent
livenessProbe:
exec:
command:
- ls
- /data/lost+found
initialDelaySeconds: 5
periodSeconds: 5
volumeMounts:
- name: volv
mountPath: /data
ports:
- containerPort: 80
volumes:
- name: volv
persistentVolumeClaim:
claimName: longhorn-volv-pvc
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: storage
operator: In
values:
- enabled
Expected behavior
Exec into the pod and run df -h I should see the /data mountpoint with ~10G filesystem. Instead, as shown below, I see 1.9G
/ # df -h
Filesystem Size Used Available Use% Mounted on
overlay 58.3G 8.4G 47.5G 15% /
tmpfs 64.0M 0 64.0M 0% /dev
tmpfs 3.8G 0 3.8G 0% /sys/fs/cgroup
/dev/longhorn/pvc-7487fc6a-1357-4741-9855-c79f9707e826
1.9G 6.0M 1.9G 0% /data
/dev/mmcblk0p2 58.3G 8.4G 47.5G 15% /etc/hosts
/dev/mmcblk0p2 58.3G 8.4G 47.5G 15% /dev/termination-log
/dev/mmcblk0p2 58.3G 8.4G 47.5G 15% /etc/hostname
/dev/mmcblk0p2 58.3G 8.4G 47.5G 15% /etc/resolv.conf
shm 64.0M 0 64.0M 0% /dev/shm
tmpfs 3.8G 12.0K 3.8G 0% /run/secrets/kubernetes.io/serviceaccount
tmpfs 64.0M 0 64.0M 0% /proc/kcore
tmpfs 64.0M 0 64.0M 0% /proc/keys
tmpfs 64.0M 0 64.0M 0% /proc/latency_stats
tmpfs 64.0M 0 64.0M 0% /proc/timer_list
tmpfs 64.0M 0 64.0M 0% /proc/sched_debug
tmpfs 3.8G 0 3.8G 0% /proc/scsi
tmpfs 3.8G 0 3.8G 0% /sys/firmware
/ #
Log N/A
Environment:
- Longhorn version: v.1.0.1
- Kubernetes version: Client Version: v1.19.0-beta.0 Server Version: v1.18.6+k3s1
- Node OS type and version:
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
worker4 Ready <none> 21d v1.18.6+k3s1 10.40.0.14 <none> Ubuntu 20.04.1 LTS 5.4.0-1015-raspi containerd://1.3.3-k3s2
worker7 Ready <none> 19d v1.18.6+k3s1 10.40.0.17 <none> Ubuntu 20.04.1 LTS 5.4.0-1018-raspi containerd://1.3.3-k3s2
worker3 Ready <none> 21d v1.18.6+k3s1 10.40.0.13 <none> Ubuntu 20.04.1 LTS 5.4.0-1015-raspi containerd://1.3.3-k3s2
worker5 Ready <none> 21d v1.18.6+k3s1 10.40.0.15 <none> Ubuntu 20.04.1 LTS 5.4.0-1016-raspi containerd://1.3.3-k3s2
worker6 Ready <none> 19d v1.18.6+k3s1 10.40.0.16 <none> Ubuntu 20.04.1 LTS 5.4.0-1015-raspi containerd://1.3.3-k3s2
ctrl1 Ready master 21d v1.18.6+k3s1 10.40.0.101 <none> Ubuntu 20.04.1 LTS 5.4.0-1018-raspi containerd://1.3.3-k3s2
worker1 Ready <none> 21d v1.18.6+k3s1 10.40.0.11 <none> Ubuntu 20.04.1 LTS 5.4.0-1018-raspi containerd://1.3.3-k3s2
worker2 Ready <none> 21d v1.18.6+k3s1 10.40.0.12 <none> Ubuntu 20.04.1 LTS 5.4.0-1015-raspi containerd://1.3.3-k3s2
Additional context Longhorn UI shows no errors, all replicas healthy. Running on as rasberry pi 4 cluster.
PVC yaml dump
apiVersion: v1
items:
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"longhorn-volv-pvc","namespace":"default"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"10Gi"}},"storageClassName":"longhorn"}}
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: driver.longhorn.io
creationTimestamp: "2020-10-02T21:10:10Z"
finalizers:
- kubernetes.io/pvc-protection
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:kubectl.kubernetes.io/last-applied-configuration: {}
f:spec:
f:accessModes: {}
f:resources:
f:requests:
.: {}
f:storage: {}
f:storageClassName: {}
f:volumeMode: {}
manager: kubectl-client-side-apply
operation: Update
time: "2020-10-02T21:10:10Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
f:pv.kubernetes.io/bind-completed: {}
f:pv.kubernetes.io/bound-by-controller: {}
f:volume.beta.kubernetes.io/storage-provisioner: {}
f:spec:
f:volumeName: {}
f:status:
f:accessModes: {}
f:capacity:
.: {}
f:storage: {}
f:phase: {}
manager: k3s
operation: Update
time: "2020-10-02T21:10:12Z"
name: longhorn-volv-pvc
namespace: default
resourceVersion: "5009265"
selfLink: /api/v1/namespaces/default/persistentvolumeclaims/longhorn-volv-pvc
uid: 7487fc6a-1357-4741-9855-c79f9707e826
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: longhorn
volumeMode: Filesystem
volumeName: pvc-7487fc6a-1357-4741-9855-c79f9707e826
status:
accessModes:
- ReadWriteOnce
capacity:
storage: 10Gi
phase: Bound
kind: List
metadata:
resourceVersion: ""
selfLink: ""
PV yaml dump
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
pv.kubernetes.io/provisioned-by: driver.longhorn.io
creationTimestamp: "2020-10-02T21:10:12Z"
finalizers:
- kubernetes.io/pv-protection
- external-attacher/driver-longhorn-io
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:pv.kubernetes.io/provisioned-by: {}
f:spec:
f:accessModes: {}
f:capacity:
.: {}
f:storage: {}
f:claimRef:
.: {}
f:apiVersion: {}
f:kind: {}
f:name: {}
f:namespace: {}
f:resourceVersion: {}
f:uid: {}
f:csi:
.: {}
f:driver: {}
f:fsType: {}
f:volumeAttributes:
.: {}
f:fromBackup: {}
f:numberOfReplicas: {}
f:staleReplicaTimeout: {}
f:storage.kubernetes.io/csiProvisionerIdentity: {}
f:volumeHandle: {}
f:persistentVolumeReclaimPolicy: {}
f:storageClassName: {}
f:volumeMode: {}
manager: csi-provisioner
operation: Update
time: "2020-10-02T21:10:12Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:status:
f:phase: {}
manager: k3s
operation: Update
time: "2020-10-02T21:10:12Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:finalizers:
v:"external-attacher/driver-longhorn-io": {}
manager: csi-attacher
operation: Update
time: "2020-10-02T21:10:13Z"
name: pvc-7487fc6a-1357-4741-9855-c79f9707e826
resourceVersion: "5009273"
selfLink: /api/v1/persistentvolumes/pvc-7487fc6a-1357-4741-9855-c79f9707e826
uid: 6e99957a-b217-4eda-b14a-d9d4d18d16d0
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 10Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: longhorn-volv-pvc
namespace: default
resourceVersion: "5009223"
uid: 7487fc6a-1357-4741-9855-c79f9707e826
csi:
driver: driver.longhorn.io
fsType: ext4
volumeAttributes:
fromBackup: ""
numberOfReplicas: "3"
staleReplicaTimeout: "2880"
storage.kubernetes.io/csiProvisionerIdentity: 1599863377275-8081-driver.longhorn.io
volumeHandle: pvc-7487fc6a-1357-4741-9855-c79f9707e826
persistentVolumeReclaimPolicy: Delete
storageClassName: longhorn
volumeMode: Filesystem
status:
phase: Bound
Longhorn deployment manifests
apiVersion: v1
kind: Namespace
metadata:
name: longhorn-system
---
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: longhorn
parameters:
fromBackup: ""
numberOfReplicas: "3"
staleReplicaTimeout: "2880"
provisioner: driver.longhorn.io
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
labels:
longhorn-manager: EngineImage
name: engineimages.longhorn.io
spec:
group: longhorn.io
names:
kind: EngineImage
listKind: EngineImageList
plural: engineimages
shortNames:
- lhei
singular: engineimage
scope: Namespaced
subresources:
status: {}
version: v1beta1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
labels:
longhorn-manager: Engine
name: engines.longhorn.io
spec:
group: longhorn.io
names:
kind: Engine
listKind: EngineList
plural: engines
shortNames:
- lhe
singular: engine
scope: Namespaced
subresources:
status: {}
version: v1beta1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
labels:
longhorn-manager: InstanceManager
name: instancemanagers.longhorn.io
spec:
group: longhorn.io
names:
kind: InstanceManager
listKind: InstanceManagerList
plural: instancemanagers
shortNames:
- lhim
singular: instancemanager
scope: Namespaced
subresources:
status: {}
version: v1beta1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
labels:
longhorn-manager: Node
name: nodes.longhorn.io
spec:
group: longhorn.io
names:
kind: Node
listKind: NodeList
plural: nodes
shortNames:
- lhn
singular: node
scope: Namespaced
subresources:
status: {}
version: v1beta1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
labels:
longhorn-manager: Replica
name: replicas.longhorn.io
spec:
group: longhorn.io
names:
kind: Replica
listKind: ReplicaList
plural: replicas
shortNames:
- lhr
singular: replica
scope: Namespaced
subresources:
status: {}
version: v1beta1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
labels:
longhorn-manager: Setting
name: settings.longhorn.io
spec:
group: longhorn.io
names:
kind: Setting
listKind: SettingList
plural: settings
shortNames:
- lhs
singular: setting
scope: Namespaced
version: v1beta1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
labels:
longhorn-manager: Volume
name: volumes.longhorn.io
spec:
group: longhorn.io
names:
kind: Volume
listKind: VolumeList
plural: volumes
shortNames:
- lhv
singular: volume
scope: Namespaced
subresources:
status: {}
version: v1beta1
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: longhorn-service-account
namespace: longhorn-system
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: longhorn-psp
spec:
allowPrivilegeEscalation: true
allowedCapabilities:
- SYS_ADMIN
fsGroup:
rule: RunAsAny
hostIPC: false
hostNetwork: false
hostPID: true
privileged: true
requiredDropCapabilities:
- NET_RAW
runAsUser:
rule: RunAsAny
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
volumes:
- configMap
- downwardAPI
- emptyDir
- secret
- projected
- hostPath
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: longhorn-psp-role
namespace: longhorn-system
rules:
- apiGroups:
- policy
resourceNames:
- longhorn-psp
resources:
- podsecuritypolicies
verbs:
- use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: longhorn-role
rules:
- apiGroups:
- apiextensions.k8s.io
resources:
- customresourcedefinitions
verbs:
- '*'
- apiGroups:
- ""
resources:
- pods
- events
- persistentvolumes
- persistentvolumeclaims
- persistentvolumeclaims/status
- nodes
- proxy/nodes
- pods/log
- secrets
- services
- endpoints
- configmaps
verbs:
- '*'
- apiGroups:
- ""
resources:
- namespaces
verbs:
- get
- list
- apiGroups:
- apps
resources:
- daemonsets
- statefulsets
- deployments
verbs:
- '*'
- apiGroups:
- batch
resources:
- jobs
- cronjobs
verbs:
- '*'
- apiGroups:
- scheduling.k8s.io
resources:
- priorityclasses
verbs:
- watch
- list
- apiGroups:
- storage.k8s.io
resources:
- storageclasses
- volumeattachments
- csinodes
- csidrivers
verbs:
- '*'
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- get
- watch
- list
- delete
- update
- create
- apiGroups:
- longhorn.io
resources:
- volumes
- volumes/status
- engines
- engines/status
- replicas
- replicas/status
- settings
- engineimages
- engineimages/status
- nodes
- nodes/status
- instancemanagers
- instancemanagers/status
verbs:
- '*'
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- '*'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: longhorn-psp-binding
namespace: longhorn-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: longhorn-psp-role
subjects:
- kind: ServiceAccount
name: longhorn-service-account
namespace: longhorn-system
- kind: ServiceAccount
name: default
namespace: longhorn-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: longhorn-bind
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: longhorn-role
subjects:
- kind: ServiceAccount
name: longhorn-service-account
namespace: longhorn-system
---
apiVersion: v1
data:
default-setting.yaml: |-
backup-target:
backup-target-credential-secret:
create-default-disk-labeled-nodes: true
default-data-path: /data/disk0
replica-soft-anti-affinity:
storage-over-provisioning-percentage:
storage-minimal-available-percentage:
upgrade-checker:
default-replica-count: 3
guaranteed-engine-cpu: 0.125
default-longhorn-static-storage-class:
backupstore-poll-interval:
taint-toleration:
priority-class:
registry-secret:
auto-salvage:
disable-scheduling-on-cordoned-node:
replica-zone-soft-anti-affinity:
volume-attachment-recovery-policy:
mkfs-ext4-parameters:
kind: ConfigMap
metadata:
name: longhorn-default-setting
namespace: longhorn-system
---
apiVersion: v1
kind: Service
metadata:
labels:
app: longhorn-manager
name: longhorn-backend
namespace: longhorn-system
spec:
ports:
- name: manager
port: 9500
targetPort: manager
selector:
app: longhorn-manager
sessionAffinity: ClientIP
type: ClusterIP
---
apiVersion: v1
kind: Service
metadata:
labels:
app: longhorn-ui
name: longhorn-frontend
namespace: longhorn-system
spec:
ports:
- name: http
nodePort: null
port: 80
targetPort: http
selector:
app: longhorn-ui
type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: longhorn-driver-deployer
namespace: longhorn-system
spec:
replicas: 1
selector:
matchLabels:
app: longhorn-driver-deployer
template:
metadata:
labels:
app: longhorn-driver-deployer
spec:
containers:
- command:
- longhorn-manager
- -d
- deploy-driver
- --manager-image
- vsellier/longhorn-manager:v1.0.1
- --manager-url
- http://longhorn-backend:9500/v1
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: SERVICE_ACCOUNT
valueFrom:
fieldRef:
fieldPath: spec.serviceAccountName
- name: CSI_ATTACHER_IMAGE
value: vsellier/csi-attacher:v2.0.0
- name: CSI_PROVISIONER_IMAGE
value: vsellier/csi-provisioner:v1.4.0
- name: CSI_NODE_DRIVER_REGISTRAR_IMAGE
value: vsellier/csi-node-driver-registrar:v1.2.0
- name: CSI_RESIZER_IMAGE
value: vsellier/csi-resizer:v0.3.0
image: vsellier/longhorn-manager:v1.0.1
imagePullPolicy: IfNotPresent
name: longhorn-driver-deployer
initContainers:
- command:
- sh
- -c
- while [ $(curl -m 1 -s -o /dev/null -w "%{http_code}" http://longhorn-backend:9500/v1) != "200" ]; do echo waiting; sleep 2; done
image: vsellier/longhorn-manager:v1.0.1
name: wait-longhorn-manager
securityContext:
runAsUser: 0
serviceAccountName: longhorn-service-account
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: longhorn-ui
name: longhorn-ui
namespace: longhorn-system
spec:
replicas: 1
selector:
matchLabels:
app: longhorn-ui
template:
metadata:
labels:
app: longhorn-ui
spec:
containers:
- env:
- name: LONGHORN_MANAGER_IP
value: http://longhorn-backend:9500
image: vsellier/longhorn-ui:v1.0.1
imagePullPolicy: IfNotPresent
name: longhorn-ui
ports:
- containerPort: 8000
name: http
securityContext:
runAsUser: 0
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
app: longhorn-manager
name: longhorn-manager
namespace: longhorn-system
spec:
selector:
matchLabels:
app: longhorn-manager
template:
metadata:
labels:
app: longhorn-manager
spec:
containers:
- command:
- longhorn-manager
- -d
- daemon
- --engine-image
- vsellier/longhorn-engine:v1.0.1
- --instance-manager-image
- vsellier/longhorn-instance-manager:v1_20200514_1
- --manager-image
- vsellier/longhorn-manager:v1.0.1
- --service-account
- longhorn-service-account
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: DEFAULT_SETTING_PATH
value: /var/lib/longhorn-setting/default-setting.yaml
image: vsellier/longhorn-manager:v1.0.1
imagePullPolicy: IfNotPresent
name: longhorn-manager
ports:
- containerPort: 9500
name: manager
readinessProbe:
tcpSocket:
port: 9500
securityContext:
privileged: true
volumeMounts:
- mountPath: /host/dev/
name: dev
- mountPath: /host/proc/
name: proc
- mountPath: /var/run/
mountPropagation: Bidirectional
name: varrun
- mountPath: /var/lib/longhorn/
mountPropagation: Bidirectional
name: longhorn
- mountPath: /var/lib/longhorn-setting/
name: longhorn-default-setting
serviceAccountName: longhorn-service-account
volumes:
- hostPath:
path: /dev/
name: dev
- hostPath:
path: /proc/
name: proc
- hostPath:
path: /var/run/
name: varrun
- hostPath:
path: /var/lib/longhorn/
name: longhorn
- configMap:
name: longhorn-default-setting
name: longhorn-default-setting
updateStrategy:
rollingUpdate:
maxUnavailable: 100%
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod-cluster-issuer
kubernetes.io/ingress.class: nginx
name: longhorn-ingress
namespace: longhorn-system
spec:
rules:
- host: longhorn.k8s.b5y.gs
http:
paths:
- backend:
serviceName: longhorn-frontend
servicePort: 80
path: /
tls:
- hosts:
- longhorn.k8s.b5y.gs
secretName: tls-secret-le-longhorn
About this issue
- Original URL
- State: open
- Created 4 years ago
- Comments: 27 (10 by maintainers)
Hi, I’m really surprised to see these images used by someone else. I have created these images during this summer to try to deploy longhorn on k3s with a cluster of rpi3/rpi4 (32bits). It was only in a learning purpose.
I finally gave up because guess what… I was not able to create a volume greater than 2gb 😃.
I suppose there are some limitations somewhere with the tools compiled in 32bits. For example, I made a dirty patch on bufbuild/buf to be able to compile it in 32bit.
As it’s used in the longhorn-engine image, it can be relative to this change but I haven’t been able to prove it.
I could gather all my notes on the work I have done to build the images if you think it can help, but honestly, it was only a poc with a lot of dirty work and it was not intended to be reused.
Hum it should not be amd64, probably a mistake during the last tests I have done. I have rebuilt and pushed the image for arm this time. I hope this could help you.
Regards.