notary: yubikey HSM driver not loading on OSX 10.11.6 (15G1004)

Hello, I’m trying to test out notary with the Yubikey 4 support, and having some troubles getting it to work.

I’m using notary version:

notary
 Version:    0.4.2
 Git commit: c8aa8cf

When running notary init, it logs an error loading /usr/local/lib/libykcs11.dylib:

notary init  example.com/collection -D -s http://notary-server:8080
DEBU[0000] Using the following trust directory: /Users/exampleuser/.notary
DEBU[0000] Trusting 1 certs
ERRO[0000] could not reach http://notary-server:8080: Get http://notary-server:8080/v2/: dial tcp 192.168.64.2:8080: getsockopt: connection refused
INFO[0000] continuing in offline mode
DEBU[0000] No yubikey found, using alternative key storage: found library /usr/local/lib/libykcs11.dylib, but initialize error pkcs11: 0x6: CKR_FUNCTION_FAILED
DEBU[0000] No yubikey found, using alternative key storage: found library /usr/local/lib/libykcs11.dylib, but initialize error pkcs11: 0x6: CKR_FUNCTION_FAILED
No root keys found. Generating a new root key...
DEBU[0000] generated ECDSA key with keyID: ad74b12de90f93ed5acad8daded968e5f4f95509906728cfb24bbc3b8b01c156
DEBU[0000] generated new ecdsa key for role: root and keyID: ad74b12de90f93ed5acad8daded968e5f4f95509906728cfb24bbc3b8b01c156
DEBU[0000] No yubikey found, using alternative key storage: found library /usr/local/lib/libykcs11.dylib, but initialize error pkcs11: 0x6: CKR_FUNCTION_FAILED
You are about to create a new root signing key passphrase. This passphrase
will be used to protect the most sensitive key in your signing system. Please
choose a long, complex passphrase and be careful to keep the password and the
key file itself secure and backed up. It is highly recommended that you use a
password manager to generate the passphrase and keep it safe. There will be no
way to recover this key. You can find the key in your config directory.
Enter passphrase for new root key with ID ad74b12:

About this issue

Original URL
State: closed
Created 8 years ago
Comments: 21 (11 by maintainers)

Most upvoted comments

Update:

So, on a hunch after reading some bug reports about problems with scdaemon Yubikey access & PCSC library access randomly stopping I decided to test something out:

Verify that gpg2 --card-status is able to contact Yubikey & list keys
Kill all running scdaemon processes (which appear to get started by gpg-agent on-demand automatically)
Retry the notary init -D command

Results:

notary init -D  docker.io/trinitronx/build-tools
DEBU[0000] Using the following trust directory: /Users/exampleuser/.notary
DEBU[0000] Trusting 1 certs
DEBU[0000] Initialized PKCS11 library /usr/local/lib/libykcs11.dylib and started HSM session
DEBU[0000] Failed to list key from the yubikey: No keys found in yubikey.
DEBU[0000] Initialized PKCS11 library /usr/local/lib/libykcs11.dylib and started HSM session
DEBU[0000] Failed to list key from the yubikey: No keys found in yubikey.
No root keys found. Generating a new root key...
DEBU[0000] generated ECDSA key with keyID: 2d39526716b1c3ef236892701bd6bc0c342a98246748d8a4e8c2de39e8de11ff
DEBU[0000] generated new ecdsa key for role: root and keyID: 2d39526716b1c3ef236892701bd6bc0c342a98246748d8a4e8c2de39e8de11ff
DEBU[0000] Initialized PKCS11 library /usr/local/lib/libykcs11.dylib and started HSM session
DEBU[0000] Attempting to store key using yubikey slot [2]
DEBU[0000] Attempting to add key to yubikey with ID: 2d39526716b1c3ef236892701bd6bc0c342a98246748d8a4e8c2de39e8de11ff
You are about to create a new root signing key passphrase. This passphrase
will be used to protect the most sensitive key in your signing system. Please
choose a long, complex passphrase and be careful to keep the password and the
key file itself secure and backed up. It is highly recommended that you use a
password manager to generate the passphrase and keep it safe. There will be no
way to recover this key. You can find the key in your config directory.
Enter passphrase for new root key with ID 2d39526:

Observations:

It looks like the scdaemon process started by gpg-agent locks up the Yubikey for exclusive access, which prevents notary from accessing the key. Killing scdaemon works for a brief period of time until the next gpg operation that needs to access the key & automatically starts up scdaemon again, blocking access to the key from anything else.

Additionally, there are bugs in the way scdaemon handles the Yubikey 4 on OSX, causing the key to become inaccessible even to the same scdaemon and gpg-agent processes that were able to access it before. This issue begins to happen after some period of time in the order of 30-mins to 1-2 (unknown?) hours of time.

scdaemon.log file shows errors: pcsc_connect failed: sharing violation (0x8010000b), new_reader_slot: out of slots.

Killing scdaemon temporarily fixed the access issue by GPG, then gpg2 --card-status worked. Killing scdaemon again finally allowed notary init -D to work!

scdaemon.log:

2016-10-17 09:49:03 scdaemon[2939] detected reader 'Yubico Yubikey 4 OTP+U2F+CCID'
2016-10-17 09:49:03 scdaemon[2939] pcsc_control failed: not transacted (0x80100016)
2016-10-17 09:49:03 scdaemon[2939] pcsc_vendor_specific_init: GET_FEATURE_REQUEST failed: 65547
2016-10-17 09:49:03 scdaemon[2939] updating reader 0 (0) status: 0x0000->0x0007 (0->1)
2016-10-17 09:49:03 scdaemon[2939] sending signal 31 to client 527
2016-10-17 09:54:03 scdaemon[2939] signatures created so far: 703
2016-10-17 09:54:03 scdaemon[2939] DBG: asking for PIN '||Please enter the PIN%0A[sigs done: 703]'
2016-10-17 10:32:18 scdaemon[2939] updating reader 0 (0) status: 0x0007->0x0000 (1->2)
2016-10-17 10:32:18 scdaemon[2939] DBG: Removal of a card: 0
2016-10-17 10:32:18 scdaemon[2939] DBG: application has been released
2016-10-17 10:32:18 scdaemon[2939] sending signal 31 to client 527
2016-10-17 10:53:50 scdaemon[2939] detected reader 'Yubico Yubikey 4 OTP+U2F+CCID'
2016-10-17 10:53:50 scdaemon[2939] pcsc_connect failed: sharing violation (0x8010000b)
2016-10-17 10:53:50 scdaemon[2939] DBG: Removal of a card: 0
2016-10-17 10:53:50 scdaemon[2939] detected reader 'Yubico Yubikey 4 OTP+U2F+CCID'
2016-10-17 10:53:50 scdaemon[2939] pcsc_connect failed: sharing violation (0x8010000b)
2016-10-17 10:53:50 scdaemon[2939] DBG: Removal of a card: 0
2016-10-17 10:53:50 scdaemon[2939] detected reader 'Yubico Yubikey 4 OTP+U2F+CCID'
2016-10-17 10:53:50 scdaemon[2939] pcsc_connect failed: sharing violation (0x8010000b)
2016-10-17 10:53:50 scdaemon[2939] DBG: Removal of a card: 0
2016-10-17 10:53:50 scdaemon[2939] new_reader_slot: out of slots
2016-10-17 10:53:50 scdaemon[2939] new_reader_slot: out of slots
2016-10-17 10:53:50 scdaemon[2939] new_reader_slot: out of slots
[...SNIP...]
[...SAME MESSAGE REPEATS...]
[...SNIP...]
[...SCDAEMON KILLED HERE...]
2016-10-17 11:23:23 scdaemon[2939] SIGTERM received - shutting down ...
2016-10-17 11:23:23 scdaemon[2939] scdaemon (GnuPG) 2.1.15 stopped
2016-10-17 11:23:23 scdaemon[2939] scdaemon (GnuPG) 2.1.15 stopped
[...RE-RUN `gpg2 --card-status` HERE...]
2016-10-17 11:23:28 scdaemon[19552] detected reader 'Yubico Yubikey 4 OTP+U2F+CCID'
2016-10-17 11:23:28 scdaemon[19552] pcsc_control failed: not transacted (0x80100016)
2016-10-17 11:23:28 scdaemon[19552] pcsc_vendor_specific_init: GET_FEATURE_REQUEST failed: 65547
2016-10-17 11:23:28 scdaemon[19552] updating reader 0 (0) status: 0x0000->0x0007 (0->1)
2016-10-17 11:23:28 scdaemon[19552] sending signal 31 to client 527
[...SCDAEMON KILLED HERE...]
2016-10-17 11:23:52 scdaemon[19552] SIGTERM received - shutting down ...
2016-10-17 11:23:52 scdaemon[19552] scdaemon (GnuPG) 2.1.15 stopped

[...RE-RAN `notary init -D` HERE...]
[...IT WORKED!...]

There is an unaccepted patch which some say alleviates this issue. However, it was not accepted by official upstream GnuPG, and the bug was closed as wontfix. The reasoning was as follows:

The patch is a work for problem somewhere in the PC/SC implementaion. I am also not sure whether a pthread_cancel for a buggy PC/SC library is a good idea. Terminating the process seems to be a better solution.

If gpgtools wants to apply this patch, they might of course do so but I don’t want to apply it upstream in particular not to an older version (2.1 is current).

There are reported to be other workarounds which require disabling OSX built-in ifdhandler (sudo launchctl unload /System/Library/LaunchDaemons/com.apple.ifdreader.plist). GPGTools recommended to this user to try the latest nightly build as of Jan 26th 2015, and it was accepted that it was fixed.

However, because I’m using gpg21 from Homebrew for gpg-agent socket forwarding support (brew tap homebrew/versions && brew install gnupg21), I cannot benefit from the fixes in GPGTools. GPGTools has stated that they are considering switch to gnupg 2.1, but not in the near future. There is a bug ticket for tracking GnuPG 2.1 support in GPGTools / MacGPG, as mentioned in this official comment.

TLDR;

There are bugs with Yubikey OpenPGP access in upstream GPG 2.0.x AND 2.1.x series releases on at least OSX versions:

10.10 (Yosemite)
10.11 (El Capitan)
10.12 (Sierra) GPGTools not compatible with this release yet!

scdaemon is launched to access the Yubikey smart card, and then begins to fail to access it after a period of time. scdaemon also locks up access to the card while it is running by other applications including at least:

Yubico Authenticator
notary
Possibly other tools needing access to Yubikey PIV slots?

GNUPG accepted workaround: kill $(pidof scdaemon)

trinitronx on Oct 17, 2016

@riyazdf CCID mode confirmed. The Yubikey is showing 0x0407 which is the “everything enabled” mode.

endophage on Oct 14, 2016