operating-system: Failed to start Grow File System on /mnt/data

Installed HassOS OVA 3.6 in Proxmox hypervisor way back when, upgrades always worked fine (all the way up to the last 5.X release). After updating to 6.0 however, I’m greeted with this message upon boot in the terminal/VNC:

[FAILED] Failed to start Grow File System on /mnt/data
See 'systemctl status systemd-growfs@mnt-data.service' for details.

The service status is the same as the guy in this thread: https://community.home-assistant.io/t/hass-os-6-0-vdi-after-upgrade-fails-every-boot-failed-to-start-grow-file-system-on-mnt-data/315819

I’ve tried to disable the service with systemctl disable, but that doesn’t seem to work/persist through reboot.

Ctrl-D to continue works like a charm, no difference in performance is noticed from HA. Only after my full backup once a week (with reboot) do I need to login to the console and press ctrl-D.

My VM has a single disk (on zfs on my hypervisor) of 32GB and an EFI disk of 128K, OVMF (UEFI) bios, i440fx machine type (default), LSI 53C895A SCSI controller (default) and a virtio network device.

Home Assistant OS release:

System Health

version: core-2021.6.5 installation_type: Home Assistant OS dev: false hassio: true docker: true virtualenv: false python_version: 3.8.9 os_name: Linux os_version: 5.10.42 arch: x86_64 timezone: Europe/Amsterdam

GitHub API: ok Github API Calls Remaining: 4714 Installed Version: 1.12.4 Stage: running Available Repositories: 914 Installed Repositories: 13

logged_in: false can_reach_cert_server: ok can_reach_cloud_auth: ok can_reach_cloud: ok

host_os: Home Assistant OS 6.0 update_channel: stable supervisor_version: supervisor-2021.06.3 docker_version: 20.10.6 disk_total: 30.8 GB disk_used: 17.3 GB healthy: true supported: true board: ova supervisor_api: ok version_api: ok installed_addons: Check Home Assistant configuration (3.7.1), Terminal & SSH (9.1.3), Grafana (7.0.1), InfluxDB (4.1.1), File editor (5.3.1), Node-RED (9.1.4), Glances (0.12.2), Samba share (9.5.1), TasmoAdmin (0.15.0), Visual Studio Code (3.4.1), ESPHome (1.19.1), SQLite Web (3.1.0), Portainer (1.5.1), Zigbee2mqtt (1.19.1-1)

dashboards: 3 resources: 8 views: 7 mode: storage

Supervisor logs: 21-06-20 20:12:14 INFO (SyncWorker_1) [supervisor.docker.interface] Attaching to homeassistant/amd64-hassio-cli with version 2021.05.1 21-06-20 20:12:14 INFO (MainThread) [supervisor.plugins.cli] Starting CLI plugin 21-06-20 20:12:14 INFO (SyncWorker_1) [supervisor.docker.interface] Cleaning hassio_cli application 21-06-20 20:12:15 INFO (SyncWorker_1) [supervisor.docker.cli] Starting CLI homeassistant/amd64-hassio-cli with version 2021.05.1 - 172.30.32.5 21-06-20 20:12:15 INFO (SyncWorker_0) [supervisor.docker.interface] Attaching to ghcr.io/home-assistant/amd64-hassio-observer with version 2021.06.0 21-06-20 20:12:15 INFO (SyncWorker_0) [supervisor.docker.interface] Attaching to homeassistant/amd64-hassio-multicast with version 2021.04.0 21-06-20 20:12:15 INFO (MainThread) [supervisor.plugins.multicast] Starting Multicast plugin 21-06-20 20:12:15 INFO (SyncWorker_0) [supervisor.docker.interface] Cleaning hassio_multicast application 21-06-20 20:12:16 INFO (SyncWorker_0) [supervisor.docker.multicast] Starting Multicast homeassistant/amd64-hassio-multicast with version 2021.04.0 - Host 21-06-20 20:12:16 INFO (MainThread) [supervisor.updater] Fetching update data from https://version.home-assistant.io/stable.json 21-06-20 20:12:17 INFO (SyncWorker_0) [supervisor.docker.interface] Attaching to ghcr.io/home-assistant/qemux86-64-homeassistant with version 2021.6.5 21-06-20 20:12:17 INFO (MainThread) [supervisor.homeassistant.secrets] Loaded 3 Home Assistant secrets 21-06-20 20:12:17 INFO (MainThread) [supervisor.hassos] Detect Home Assistant Operating System 6.0 / BootSlot B 21-06-20 20:12:18 WARNING (MainThread) [supervisor.addons.validate] Add-on have full device access, and selective device access in the configuration. Please report this to the maintainer of Frigate NVR Beta (outdated) 21-06-20 20:12:18 WARNING (MainThread) [supervisor.addons.validate] Add-on have full device access, and selective device access in the configuration. Please report this to the maintainer of Frigate NVR 21-06-20 20:12:18 INFO (MainThread) [supervisor.store.git] Loading add-on /data/addons/core repository 21-06-20 20:12:18 INFO (MainThread) [supervisor.store.git] Loading add-on /data/addons/git/6e66619d repository 21-06-20 20:12:18 INFO (MainThread) [supervisor.store.git] Loading add-on /data/addons/git/45df7312 repository 21-06-20 20:12:18 INFO (MainThread) [supervisor.store.git] Loading add-on /data/addons/git/a0d7b954 repository 21-06-20 20:12:18 INFO (MainThread) [supervisor.store.git] Loading add-on /data/addons/git/ccab4aaf repository 21-06-20 20:12:18 INFO (MainThread) [supervisor.store.git] Loading add-on /data/addons/git/15ef4d2f repository 21-06-20 20:12:20 WARNING (MainThread) [supervisor.addons.validate] Add-on have full device access, and selective device access in the configuration. Please report this to the maintainer of Frigate NVR Beta (outdated) 21-06-20 20:12:20 WARNING (MainThread) [supervisor.addons.validate] Add-on have full device access, and selective device access in the configuration. Please report this to the maintainer of Frigate NVR 21-06-20 20:12:20 INFO (MainThread) [supervisor.store] Loading add-ons from store: 72 all - 72 new - 0 remove 21-06-20 20:12:20 INFO (MainThread) [supervisor.addons] Found 14 installed add-ons 21-06-20 20:12:20 INFO (SyncWorker_1) [supervisor.docker.interface] Attaching to ghcr.io/hassio-addons/node-red/amd64 with version 9.1.4 21-06-20 20:12:21 INFO (SyncWorker_2) [supervisor.docker.interface] Attaching to zigbee2mqtt/zigbee2mqtt-amd64 with version 1.19.1-1 21-06-20 20:12:21 INFO (SyncWorker_1) [supervisor.docker.interface] Attaching to homeassistant/amd64-addon-configurator with version 5.3.1 21-06-20 20:12:21 INFO (SyncWorker_7) [supervisor.docker.interface] Attaching to ghcr.io/hassio-addons/grafana/amd64 with version 7.0.1 21-06-20 20:12:21 INFO (SyncWorker_0) [supervisor.docker.interface] Attaching to ghcr.io/hassio-addons/sqlite-web/amd64 with version 3.1.0 21-06-20 20:12:21 INFO (SyncWorker_3) [supervisor.docker.interface] Attaching to ghcr.io/hassio-addons/influxdb/amd64 with version 4.1.1 21-06-20 20:12:21 INFO (SyncWorker_4) [supervisor.docker.interface] Attaching to homeassistant/amd64-addon-check_config with version 3.7.1 21-06-20 20:12:21 INFO (SyncWorker_5) [supervisor.docker.interface] Attaching to ghcr.io/hassio-addons/portainer/amd64 with version 1.5.1 21-06-20 20:12:21 INFO (SyncWorker_11) [supervisor.docker.interface] Attaching to esphome/esphome-hassio-amd64 with version 1.19.1 21-06-20 20:12:21 INFO (SyncWorker_8) [supervisor.docker.interface] Attaching to ghcr.io/hassio-addons/vscode/amd64 with version 3.4.1 21-06-20 20:12:21 INFO (SyncWorker_6) [supervisor.docker.interface] Attaching to homeassistant/amd64-addon-ssh with version 9.1.3 21-06-20 20:12:21 INFO (SyncWorker_9) [supervisor.docker.interface] Attaching to ghcr.io/hassio-addons/glances/amd64 with version 0.12.2 21-06-20 20:12:21 INFO (SyncWorker_10) [supervisor.docker.interface] Attaching to homeassistant/amd64-addon-samba with version 9.5.1 21-06-20 20:12:21 INFO (SyncWorker_2) [supervisor.docker.interface] Attaching to ghcr.io/hassio-addons/tasmoadmin/amd64 with version 0.15.0 21-06-20 20:12:21 INFO (MainThread) [supervisor.snapshots] Found 0 snapshot files 21-06-20 20:12:21 INFO (MainThread) [supervisor.discovery] Loaded 0 messages 21-06-20 20:12:21 INFO (MainThread) [supervisor.ingress] Loaded 8 ingress sessions 21-06-20 20:12:21 INFO (MainThread) [supervisor.resolution.check] Starting system checks with state CoreState.SETUP 21-06-20 20:12:21 INFO (MainThread) [supervisor.resolution.check] System checks complete 21-06-20 20:12:21 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state CoreState.SETUP 21-06-20 20:12:21 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete 21-06-20 20:12:21 INFO (MainThread) [supervisor.jobs] ‘ResolutionFixup.run_autofix’ blocked from execution, system is not running - CoreState.SETUP 21-06-20 20:12:21 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state CoreState.SETUP 21-06-20 20:12:21 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete 21-06-20 20:12:21 INFO (MainThread) [main] Running Supervisor 21-06-20 20:12:21 INFO (MainThread) [supervisor.hassos] Rauc: B - marked slot kernel.1 as good 21-06-20 20:12:21 INFO (MainThread) [supervisor.addons] Phase ‘AddonStartup.INITIALIZE’ starting 0 add-ons 21-06-20 20:12:21 INFO (MainThread) [supervisor.addons] Phase ‘AddonStartup.SYSTEM’ starting 1 add-ons 21-06-20 20:12:22 INFO (SyncWorker_5) [supervisor.docker.addon] Starting Docker add-on ghcr.io/hassio-addons/tasmoadmin/amd64 with version 0.15.0 21-06-20 20:12:27 INFO (MainThread) [supervisor.addons] Phase ‘AddonStartup.SERVICES’ starting 3 add-ons 21-06-20 20:12:29 INFO (SyncWorker_1) [supervisor.docker.addon] Starting Docker add-on ghcr.io/hassio-addons/grafana/amd64 with version 7.0.1 21-06-20 20:12:31 INFO (SyncWorker_10) [supervisor.docker.addon] Starting Docker add-on ghcr.io/hassio-addons/influxdb/amd64 with version 4.1.1 21-06-20 20:12:32 INFO (SyncWorker_0) [supervisor.docker.addon] Starting Docker add-on ghcr.io/hassio-addons/vscode/amd64 with version 3.4.1 21-06-20 20:12:32 INFO (MainThread) [supervisor.api.middleware.security] /dns/info access from a0d7b954_influxdb 21-06-20 20:12:37 INFO (MainThread) [supervisor.core] Start Home Assistant Core 21-06-20 20:12:37 INFO (SyncWorker_10) [supervisor.docker.interface] Starting homeassistant 21-06-20 20:12:37 INFO (MainThread) [supervisor.homeassistant.core] Wait until Home Assistant is ready 21-06-20 20:12:48 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state CoreState.STARTUP 21-06-20 20:12:48 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete 21-06-20 20:12:53 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token 21-06-20 20:13:03 INFO (MainThread) [supervisor.homeassistant.core] Detect a running Home Assistant instance 21-06-20 20:13:03 INFO (MainThread) [supervisor.addons] Phase ‘AddonStartup.APPLICATION’ starting 4 add-ons 21-06-20 20:13:04 INFO (SyncWorker_6) [supervisor.docker.interface] Cleaning addon_core_configurator application 21-06-20 20:13:05 INFO (SyncWorker_6) [supervisor.docker.addon] Starting Docker add-on homeassistant/amd64-addon-configurator with version 5.3.1 21-06-20 20:13:06 INFO (SyncWorker_2) [supervisor.docker.interface] Cleaning addon_a0d7b954_nodered application 21-06-20 20:13:07 INFO (SyncWorker_2) [supervisor.docker.addon] Starting Docker add-on ghcr.io/hassio-addons/node-red/amd64 with version 9.1.4 21-06-20 20:13:07 INFO (SyncWorker_8) [supervisor.docker.interface] Cleaning addon_15ef4d2f_esphome application 21-06-20 20:13:07 INFO (SyncWorker_8) [supervisor.docker.addon] Starting Docker add-on esphome/esphome-hassio-amd64 with version 1.19.1 21-06-20 20:13:07 INFO (SyncWorker_3) [supervisor.docker.interface] Cleaning addon_45df7312_zigbee2mqtt application 21-06-20 20:13:08 INFO (SyncWorker_3) [supervisor.docker.addon] Starting Docker add-on zigbee2mqtt/zigbee2mqtt-amd64 with version 1.19.1-1 21-06-20 20:13:13 INFO (MainThread) [supervisor.misc.tasks] All core tasks are scheduled 21-06-20 20:13:13 INFO (MainThread) [supervisor.core] Supervisor is up and running 21-06-20 20:13:13 INFO (MainThread) [supervisor.host.info] Updating local host information 21-06-20 20:13:13 INFO (MainThread) [supervisor.updater] Fetching update data from https://version.home-assistant.io/stable.json 21-06-20 20:13:13 INFO (MainThread) [supervisor.resolution.check] Starting system checks with state CoreState.RUNNING 21-06-20 20:13:13 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.FREE_SPACE/ContextType.SYSTEM 21-06-20 20:13:13 INFO (MainThread) [supervisor.resolution.module] Create new suggestion SuggestionType.CREATE_FULL_SNAPSHOT - ContextType.SYSTEM / None 21-06-20 20:13:13 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.PWNED/ContextType.ADDON 21-06-20 20:13:13 WARNING (MainThread) [supervisor.addons.options] Option ‘interface’ does not exist in the schema for Samba share (core_samba) 21-06-20 20:13:13 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.SECURITY/ContextType.CORE 21-06-20 20:13:13 INFO (MainThread) [supervisor.resolution.check] System checks complete 21-06-20 20:13:13 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state CoreState.RUNNING 21-06-20 20:13:14 INFO (MainThread) [supervisor.host.services] Updating service information 21-06-20 20:13:14 INFO (MainThread) [supervisor.host.network] Updating local network information 21-06-20 20:13:15 INFO (MainThread) [supervisor.host.sound] Updating PulseAudio information 21-06-20 20:13:15 INFO (MainThread) [supervisor.host] Host information reload completed 21-06-20 20:13:16 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete 21-06-20 20:13:16 INFO (MainThread) [supervisor.resolution.fixup] Starting system autofix at state CoreState.RUNNING 21-06-20 20:13:16 INFO (MainThread) [supervisor.resolution.fixup] System autofix complete 21-06-20 20:13:54 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request initialize 21-06-20 20:13:54 INFO (MainThread) [supervisor.api.proxy] WebSocket access from a0d7b954_nodered 21-06-20 20:13:54 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request running 21-06-20 20:13:58 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request initialize 21-06-20 20:13:58 INFO (MainThread) [supervisor.api.proxy] WebSocket access from a0d7b954_nodered 21-06-20 20:13:58 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request running

Journal logs:

Kernel logs:

I don’t know why, but I can’t for the life of me enable ssh on the host VM. Did the whole shebang with a usbstick (passthrough) with an authorized_keys file in ANSI and linux line endings, no dice ([supervisor.hardware.monitor] Ignore device /sys/devices/pci0000:00/0000:00:1e.0/0000:01:1b.0/usb2/2-2/2-2:1.0/host8/target8:0:0/8:0:0:0/block/sdb / failes to initialize by udev). Any other way to ssh into the root os? Or maybe access it from the terminal?

Description of problem:

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 36 (13 by maintainers)

Most upvoted comments

Alright, after stopping systemd-journald I could unmount the data volume. Check started and gave me the following things:

Inodes that were part of a corrupted orphan linked list found. (8 fixed)
Deleted inode XXXXXX has zero dtime.
Block bitmap differences: a lot of numbers here
Free blocks count wrong for group #30 (8458, counted 9002).
Same for group #209
Free block count wrong
Inode bitmap differences: more numbers
more lines with free inodes count wrong

rebooted, and everything works fine now!

So, to recap:

Press <enter> when the system stops during boot
systemctl stop systemd-journald
umount -A /dev/disk/by-label/hassos-data
fsck.ext4 /dev/disk/by-label/hassos-data
Keep pressing <enter> through all the prompts
reboot

Hm, there are still errors:

[    7.335143] EXT4-fs warning (device sda8): ext4_resize_begin:72: There are errors in the filesystem, so online resizing is not allowed

When it gets stuck during boot up, and you use “enter” instead of Ctrl+D, you should get a root prompt where a manual fsck with the following commands should be possible:

umount -A /dev/disk/by-label/hassos-data
fsck.ext4 /dev/disk/by-label/hassos-data

From a running system, you get to a similar state using systemctl isolate rescue.target and systemctl stop systemd-journald.