client: keybase client v6.2.3-20231016183016+06cb935ee3 causing `/` mount issues under Ubuntu 22.04, Fedora 34

With the new version of the keybase ( 5.9.0-20211217212642.29bfd9d39f ) that @heronhaye released last week, it’s causing a very worrisome issue (which I initially thought to be a hard drive problem).

I’ve observed it now on two different unrelated Ubuntu 18.04 systems. Initially I didn’t realize it was keybase until it happened on the second system and I thought it might be mount/fuse related.

Symptoms

Intermittently the / mount in the OS stops working. Running ls / hangs and doesn’t return. Launching nautilus hangs and doesn’t launch. Other applications (in my case a java app called SmartGit) that try to read from / also hang. Other mounts appear ok, for example ls /boot works fine. Apps which don’t do anything in the / mount (for example my browser which I guess works entirely in my home directory, and home is a distinct mount, works fine)

Solution

After trying some different solutions heronhaye seems to have fixed this with the v5.9.3 release which you can update to via the normal means : https://keybase.io/docs/the_app/install_linux

Big thanks to heronhaye!

Note : If you’d used the workaround below while this was being worked out and put the keybase package on hold in apt make sure to run

sudo apt-mark unhold keybase

Workaround

When this happened on the second Ubuntu machine and I realized it might be keybase, I ran keybase ctl stop and immediately / became available, nautilus would launch, everything was fixed.

I was able to discover the previous version number at https://prerelease.keybase.io/linux_binaries/deb/index.html

Then either

sudo apt install keybase=5.8.1-20210930160723.fefa22edc1

or fetch the deb package listed on https://prerelease.keybase.io/linux_binaries/deb/index.html and install it

wget https://s3.amazonaws.com/prerelease.keybase.io/linux_binaries/deb/keybase_5.8.1-20210930160723.fefa22edc1_amd64.deb
sudo dpkg -i keybase_5.8.1-20210930160723.fefa22edc1_amd64.deb

Then run this to tell apt to hold and not upgrade keybase to 5.9.0

sudo apt-mark hold keybase

Once this issue is fixed you can unhold keybase with

sudo apt-mark unhold keybase

MAKE SURE NOT TO FORGET THAT YOU PUT KEYBASE ON HOLD AS OTHERWISE IT WILL NEVER GET UPDATES AGAIN

About this issue

Original URL
State: open
Created 3 years ago
Reactions: 9
Comments: 49 (13 by maintainers)

Links to this issue

FYI : Keybase update from last week (v5.9.0) is causing mount and launch issues

Most upvoted comments

Sure thing.

I confirmed the code signature by running gpg -v keybase_5.9.3-20220208204733.d72d1e68b3_amd64.deb.sig and comparing the fingerprint of the signing key to the one on the doc page you linked.

I’ve upgrade from 5.9.0-20220120174718.95a3939b3a to 5.9.3-20220208204733.d72d1e68b3 successfully (no errors). I’ll continue running this new version and will report back here if the problem manifests.

gene1wood on Feb 8, 2022

I’ve done the following to try prevent this issue on my end, I hope this helps someone else.

Disable keybase on systemctl. This prevents keybase running on startup

systemctl --user disable keybase.service

Create a reset_keybase script

/root/scripts/reset_keybase.sh

#!/usr/bin/env bash

/usr/bin/fusermount -uz /keybase

(and remember to make script executable)

Add the reset_keybase script to the sudoers.d directory (allows non-root user to run this script as sudo)

Don’t put periods in file name

/etc/sudoers.d/reset_keybase

Cmnd_Alias RESET_KEYBASE_CMDS = /root/scripts/reset_keybase.sh

alexiswl ALL=(ALL) NOPASSWD: RESET_KEYBASE_CMDS

Create script for user

scripts/reset_keybase_on_reboot.sh

#!/usr/bin/env bash

set -euo pipefail

# Wait a minute for everything to be set up
/usr/bin/sleep 60

# Run fusermount
# This wont trigger a password since we have permissions to run this script
sudo /root/scripts/reset_keybase.sh 

# Now reset keybase
/usr/bin/run_keybase

and remember to make executable.

Create user crontab

crontab -e

And add the following line

@reboot nohup /home/alexiswl/scripts/reset_keybase_on_reboot.sh 1>/dev/null 2>&1 &

alexiswl on Nov 21, 2023

I’m also having sporadic issues with this after restarting my desktop (ubuntu 22.04). ls / will hang for me, which causes all sorts of issues, backup problems, unable to access files from GUI etc.

Keybase version

keybase --version
keybase version 6.2.3-20231016183016+06cb935ee3

System Info

uname -a
Linux ubuntu-desktop 6.2.0-37-generic #38~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Nov  2 18:01:13 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

For me, the suggestion on fusermount resolves the issue for me

sudo fusermount -uz /keybase
run_keybase

alexiswl on Nov 21, 2023

@heronhaye Working so far with 5.9.3.

jacksongoode on Feb 9, 2022

I think my plan at this point is to either try adding the KEYBASE_NO_KBFS environment variable to my system to be able to use Keybase without KBFS or just uninstall it from the machines that I have it on.

Following this suggestion, I edited /usr/lib/systemd/user/keybase.service to read:

# Added KEYBASE_NO_KBFS=1 to suppress broken /keybase mount.  See <https://github.com/keybase/client/issues/24764#issuecomment-1821788495>.  RB 2023-12-20.
Environment=KEYBASE_SERVICE_TYPE=systemd KEYBASE_NO_KBFS=1

and for good measure I did:

systemctl --user disable kbfs

and now operations on /keybase fail immediately, which is better than hanging

rb@kiwi:~$ df /keybase
df: /keybase: Transport endpoint is not connected

Also, Keybase no longer crashes on launch of the GUI app.

rptb1 on Dec 20, 2023

I get fairly frequent hangs in Ubuntu 22 with Keybase, where various apps will seize up while trying to open files, dialogs, etc. The command df /keybase will also hang. Stopping Keybase with keybase ctl stop unblocks them and allows them to continue.

This may be related to the fact that the Keybase menu is unresponsive on login, and always crashes when I launch the app.

Once I’ve launched the app things seem to go smoothly.

Looking at @alexiswl 's workaround above, combined with my symptoms, it seems likely that Keybase isn’t starting correctly.

Keybase itself isn’t completely stuck though. I can get a red notification dot on the Keybase menu after login, even though the menu doesn’t respond to clicks and crashes on launch.

rb@kiwi:~$ uname -a
Linux kiwi 6.2.0-39-generic #40~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Nov 16 10:53:04 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
rb@kiwi:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.3 LTS
Release:	22.04
Codename:	jammy
rb@kiwi:~$ dpkg-query -l '*keybase*'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version                         Architecture Description
+++-==============-===============================-============-==========================================
ii  keybase        6.2.3-20231016183016.06cb935ee3 amd64        The Keybase Go client, filesystem, and GUI

rptb1 on Dec 20, 2023

@heronhaye Ya, I can confirm this new version doesn’t seem to fix the issue. On version 5.9.0-20220120174718.95a3939b3a on Ubuntu 18.04, immediately on boot doing an ls / hangs. Running sudo fusermount -uz /keybase unhangs it.

@heronhaye Should we open a new issue as this and #24749 are currently closed?

gene1wood on Jan 22, 2022

Yes. We did not realize that setreuid modified the suid bit; however, in previous Go versions, the calling goroutine took place in a different OS thread, so the bug did not manifest. In the latest Go version, the goroutine was scheduled on the same OS thread so the modified suid bit persisted.

heronhaye on Jan 20, 2022

I just ran into the same problem on an Ubuntu Server 20.04 LTS server after a routine upgrade. /keybase is maybe-sorta-kinda mounted though it should not be (I use a private mount under ~/.config/keybase/kbfs/ instead). It’s hanging up df, lsof, and pretty much anything else that queries FS state. Also ls /keybase (separated from the previous sentence to clarify).

Can confirm that running keybase ctl stop as non-root worked.

Running the given sudo apt command to install v5.8.1-20210930160723.fefa22edc1 did not work. “E: Version ‘5.8.1-20210930160723.fefa22edc1’ for ‘keybase’ was not found”

It would be nice if we could browse the S3 bucket, but whatever. Downloading the .deb worked, but installing it did not, or at least it seems like it didn’t work:

oot@exocortex:~# dpkg -i ~drwho/keybase_5.8.1-20210930160723.fefa22edc1_amd64.d
eb 
dpkg: warning: downgrading keybase from 5.9.0-20211217212642.29bfd9d39f to 5.8.1
-20210930160723.fefa22edc1
(Reading database ... 179816 files and directories currently installed.)
Preparing to unpack .../keybase_5.8.1-20210930160723.fefa22edc1_amd64.deb ...
Unpacking keybase (5.8.1-20210930160723.fefa22edc1) over (5.9.0-20211217212642.29bfd9d39f) ...
Setting up keybase (5.8.1-20210930160723.fefa22edc1) ...
mkdir: cannot create directory ‘/keybase’: File exists
chown: cannot access '/keybase': Transport endpoint is not connected
chmod: cannot access '/keybase': Transport endpoint is not connected
dpkg: error processing package keybase (--install):
 installed keybase package post-installation script subprocess returned error ex it status 1
Processing triggers for mime-support (3.64ubuntu1) ...
Processing triggers for hicolor-icon-theme (0.17-2) ...
Processing triggers for shared-mime-info (1.15-1) ...
Errors were encountered while processing:
 keybase

Mountpoint /keybase did not exit. Attempting the installation again resulted in its creation and “Device or resource busy” when trying to remove it. keybase.mount is showing up as loaded, active, and mounted in the output of systemtl, however. After some fiddling, I got systemctl stop keybase.mount to work and then I could manually install the .deb package. Marked as “held.”

Starting my isolated Keybase processes with systemctl --user worked.

virtadpt on Jan 9, 2022

@orgcontrib #24364 is a report from 2020 on Keybase 5.5.2 (not a new issue that began in 5.9.0 and which can be worked around by downgrading to 5.8.1).

Also #24364 refers to an inability to use keybase mounts, not symptoms that involve hanging any process that tries to take actions on the / mount (not the keybase mount)

gene1wood on Jan 3, 2022