infrastructure: [Support] LIS: Admins cannot write to shared-readwrite folder

The Freshdesk ticket link

https://2i2c.freshdesk.com/a/tickets/1349

Ticket request type

Feature Request

Ticket impact

🟧 High

Short ticket description

Matthew needs to write a dataset to the shared-readwrite folder but does not have the correct permissions, despite being an admin.

I then used deployer exec homes to move the dataset (which had been added to Matthew’s home dir) into shared-readwrite, but it is not visible via the shared folder when I log in to my server.

The dataset needs to be available by 13:30 GMT on Monday.

(Optional) Investigation results

I am also suffering the same permissions error 😕

jovyan@jupyter-sgibson91:~$ mkdir -p shared-readwrite/test
mkdir: cannot create directory ‘shared-readwrite/test’: Permission denied
jovyan@jupyter-sgibson91:~$ sudo mkdir -p shared-readwrite/test
sudo: The "no new privileges" flag is set, which prevents sudo from running as root.
sudo: If sudo is running in a container, you may need to adjust the container configuration to disable the flag.
jovyan@jupyter-sgibson91:~$ 

I used the following command

deployer exec homes 2i2c-uk lis

This gave me a shell with the home directories mounted. I found Matt’s home directory and the data/acsids directory he mentioned, and used the following command to copy it into shared-readwrite

# From within Matt's home dir
cp -r data/acsids shared-readwrite

An ls showed the folder was now there

root@2i2c-uk-lis-shell:/home/matthew-2dbrett# ls shared-readwrite/
acsids

I then restarted my server on the lis hub, but could not find the folder in the shared-readwrite or shared folders, I don’t know what’s going on here 😕

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 16 (16 by maintainers)

Most upvoted comments

I ran kubectl get pod -o yaml against my own server and could not find any mention of shared-readwrite in the manifest, which I should have because I am an admin. So I think something is wrong with the mounting of the shared-readwrite folder for admins.

Pod manifest
apiVersion: v1
kind: Pod
metadata:
  annotations:
    cni.projectcalico.org/containerID: 10c1ecc8a1ad44cc4f5e87a1e27297364bd3e100750a28d9270bad2881f8602e
    cni.projectcalico.org/podIP: 10.100.27.44/32
    cni.projectcalico.org/podIPs: 10.100.27.44/32
    hub.jupyter.org/username: sgibson91
  creationTimestamp: "2024-02-26T14:35:25Z"
  labels:
    app: jupyterhub
    chart: jupyterhub-3.2.1
    component: singleuser-server
    heritage: jupyterhub
    hub.jupyter.org/network-access-hub: "true"
    hub.jupyter.org/servername: ""
    hub.jupyter.org/username: sgibson91
    release: lis
  name: jupyter-sgibson91
  namespace: lis
  resourceVersion: "368299164"
  uid: 69a9f542-c1e3-47de-a526-c6ee84ec0595
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: hub.jupyter.org/node-purpose
            operator: In
            values:
            - user
  automountServiceAccountToken: false
  containers:
  - args:
    - jupyterhub-singleuser
    env:
    - name: CPU_GUARANTEE
      value: "0.05"
    - name: JPY_API_TOKEN
      value: 87485f592a2b4e9a9027a81ae2a3c86f
    - name: JUPYTERHUB_ACTIVITY_URL
      value: http://hub:8081/hub/api/users/sgibson91/activity
    - name: JUPYTERHUB_ADMIN_ACCESS
      value: "1"
    - name: JUPYTERHUB_API_TOKEN
      value: 87485f592a2b4e9a9027a81ae2a3c86f
    - name: JUPYTERHUB_API_URL
      value: http://hub:8081/hub/api
    - name: JUPYTERHUB_BASE_URL
      value: /
    - name: JUPYTERHUB_CLIENT_ID
      value: jupyterhub-user-sgibson91
    - name: JUPYTERHUB_DEFAULT_URL
      value: /lab
    - name: JUPYTERHUB_HOST
    - name: JUPYTERHUB_OAUTH_ACCESS_SCOPES
      value: '["access:servers!server=sgibson91/", "access:servers!user=sgibson91"]'
    - name: JUPYTERHUB_OAUTH_CALLBACK_URL
      value: /user/sgibson91/oauth_callback
    - name: JUPYTERHUB_OAUTH_CLIENT_ALLOWED_SCOPES
      value: '[]'
    - name: JUPYTERHUB_OAUTH_SCOPES
      value: '["access:servers!server=sgibson91/", "access:servers!user=sgibson91"]'
    - name: JUPYTERHUB_SERVER_NAME
    - name: JUPYTERHUB_SERVICE_PREFIX
      value: /user/sgibson91/
    - name: JUPYTERHUB_SERVICE_URL
      value: http://0.0.0.0:8888/user/sgibson91/
    - name: JUPYTERHUB_USER
      value: sgibson91
    - name: JUPYTER_IMAGE
      value: lisacuk/lishub-base:94c2a17
    - name: JUPYTER_IMAGE_SPEC
      value: lisacuk/lishub-base:94c2a17
    - name: JUPYTER_RUNTIME_DIR
      value: /tmp/.jupyter-runtime
    - name: MEM_GUARANTEE
      value: "536870912"
    - name: MEM_LIMIT
      value: "2147483648"
    - name: SHELL
      value: /bin/bash
    image: lisacuk/lishub-base:94c2a17
    imagePullPolicy: IfNotPresent
    lifecycle: {}
    name: notebook
    ports:
    - containerPort: 8888
      name: notebook-port
      protocol: TCP
    resources:
      limits:
        memory: "2147483648"
      requests:
        cpu: 50m
        memory: "536870912"
    securityContext:
      allowPrivilegeEscalation: false
      runAsUser: 1000
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /home/jovyan
      name: home
      subPath: sgibson91
    - mountPath: /usr/local/etc/ipython/ipython_kernel_config.json
      name: files
      subPath: ipython_kernel_config.json
    - mountPath: /usr/local/etc/jupyter/jupyter_notebook_config.json
      name: files
      subPath: jupyter_notebook_config.json
    - mountPath: /usr/local/etc/jupyter/jupyter_server_config.json
      name: files
      subPath: jupyter_server_config.json
    - mountPath: /home/jovyan/shared
      name: home
      readOnly: true
      subPath: _shared
    - mountPath: /home/rstudio
      name: home
      subPath: sgibson91
    - mountPath: /home/rstudio/shared
      name: home
      readOnly: true
      subPath: _shared
    - mountPath: /home/jovyan/allusers
      name: home
    workingDir: /home/jovyan
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  initContainers:
  - command:
    - sh
    - -c
    - id && chown 1000:1000 /home/jovyan /home/jovyan/shared && ls -lhd /home/jovyan
    image: busybox:1.36.1
    imagePullPolicy: IfNotPresent
    name: volume-mount-ownership-fix
    resources: {}
    securityContext:
      runAsUser: 0
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /home/jovyan
      name: home
      subPath: sgibson91
    - mountPath: /home/jovyan/shared
      name: home
      subPath: _shared
  - command:
    - iptables
    - --append
    - OUTPUT
    - --protocol
    - tcp
    - --destination
    - 169.254.169.254
    - --destination-port
    - "80"
    - --jump
    - DROP
    image: quay.io/jupyterhub/k8s-network-tools:3.2.1
    imagePullPolicy: IfNotPresent
    name: block-cloud-metadata
    resources: {}
    securityContext:
      capabilities:
        add:
        - NET_ADMIN
      privileged: true
      runAsUser: 0
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
  nodeName: gke-two-eye-two-see-u-nb-n2-highmem-4-a8475a2d-rbg2
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  priorityClassName: lis-default-priority
  restartPolicy: OnFailure
  schedulerName: gke.io/optimize-utilization-scheduler
  securityContext:
    fsGroup: 100
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoSchedule
    key: hub.jupyter.org/dedicated
    operator: Equal
    value: user
  - effect: NoSchedule
    key: hub.jupyter.org_dedicated
    operator: Equal
    value: user
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: home
    persistentVolumeClaim:
      claimName: home-nfs
  - name: files
    secret:
      defaultMode: 420
      items:
      - key: ipython_kernel_config.json
        path: ipython_kernel_config.json
      - key: jupyter_notebook_config.json
        path: jupyter_notebook_config.json
      - key: jupyter_server_config.json
        path: jupyter_server_config.json
      secretName: singleuser
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2024-02-26T14:35:31Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2024-02-26T14:35:32Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2024-02-26T14:35:32Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2024-02-26T14:35:25Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://0cfd1cf7173d7190c97c7ee418e7ad58b6720064356032355048a38e653ac867
    image: docker.io/lisacuk/lishub-base:94c2a17
    imageID: docker.io/lisacuk/lishub-base@sha256:83f5b52f795ac4fa6799030bb33accaf2a5deac69df200fc74ac8e924a987c1b
    lastState: {}
    name: notebook
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2024-02-26T14:35:31Z"
  hostIP: 10.154.0.15
  initContainerStatuses:
  - containerID: containerd://796751691c157c15d61ced0a44506fe3abeb5ba78490e3290c5566eb8e283643
    image: docker.io/library/busybox:1.36.1
    imageID: docker.io/library/busybox@sha256:6d9ac9237a84afe1516540f40a0fafdc86859b2141954b4d643af7066d598b74
    lastState: {}
    name: volume-mount-ownership-fix
    ready: true
    restartCount: 0
    state:
      terminated:
        containerID: containerd://796751691c157c15d61ced0a44506fe3abeb5ba78490e3290c5566eb8e283643
        exitCode: 0
        finishedAt: "2024-02-26T14:35:29Z"
        reason: Completed
        startedAt: "2024-02-26T14:35:29Z"
  - containerID: containerd://51a1447dbe9554e2af16a0adb32934842a91da5b91bad734f30f26b3af81956e
    image: quay.io/jupyterhub/k8s-network-tools:3.2.1
    imageID: quay.io/jupyterhub/k8s-network-tools@sha256:e532a2dc4761685cedc87d4c0df864b8b1fad45f4dd54535b1b5dff23897d870
    lastState: {}
    name: block-cloud-metadata
    ready: true
    restartCount: 0
    state:
      terminated:
        containerID: containerd://51a1447dbe9554e2af16a0adb32934842a91da5b91bad734f30f26b3af81956e
        exitCode: 0
        finishedAt: "2024-02-26T14:35:30Z"
        reason: Completed
        startedAt: "2024-02-26T14:35:30Z"
  phase: Running
  podIP: 10.100.27.44
  podIPs:
  - ip: 10.100.27.44
  qosClass: Burstable
  startTime: "2024-02-26T14:35:25Z"

(EDIT: added a new line before code block to ensure it rendered correctly)