features: Terraform - Downloading GPG Key step gets stuck

Hello,

Since a few days I notice that the GPG Key download step in the Terraform feature hangs:

=> [internal] load metadata for mcr.microsoft.com/devcontainers/base:j  608.5s
 => => extracting sha256:9a586f3d84de83b25cb9ca6d0e733d37d5283da35a837917  0.8s
 => => extracting sha256:5ac951de24f647413f348bfa4183b92c9962b167e0a3106d  0.3s
 => => extracting sha256:1aa90a4f5c993fd6d1e5f0885a1794ee59253ee8d35463ea  0.0s
 => => extracting sha256:074fbd7f2200028d96665f87580813cabf3361b85a4217ce  0.0s
 => => extracting sha256:ee36e8b50d1a0a2f6169666f26f3938fe1ea3efabfb6df01  0.0s
 => => extracting sha256:9e8c9f9ef24f45765c7592d55a952cbe2d0d4fefcd797237  5.5s
 => [dev_containers_feature_content_normalize 2/3] COPY --from=dev_contai  4.8s
 => [dev_containers_feature_content_normalize 3/3] RUN chmod -R 0700 /tmp  1.7s
 => [dev_containers_target_stage  2/12] COPY --from=dev_containers_featur  0.3s
 => [dev_containers_target_stage  3/12] RUN echo "_CONTAINER_USER_HOME=$(  0.7s
 => [dev_containers_target_stage  4/12] RUN cd /tmp/build-features/azure  61.0s
 => [dev_containers_target_stage  5/12] RUN cd /tmp/build-features/github  6.1s
 => [dev_containers_target_stage  6/12] RUN cd /tmp/build-features/kubec  51.5s
 => [dev_containers_target_stage  7/12] RUN cd /tmp/build-features/power  34.3s
 => [dev_containers_target_stage  8/12] RUN cd /tmp/build-features/terr  431.1s
 => => # Downloading terraform...
 => => # TERRAFORM_GPG_KEY=72D7468F
 => => # GPG_KEY_SERVERS=keyserver hkp://keyserver.ubuntu.com:80
 => => # keyserver hkps://keys.openpgp.org
 => => # keyserver hkp://keyserver.pgp.com
 => => # (*) Downloading GPG key...

Screenshot:

I have tried this on various systems/connections and it seems to consistently hang. As you can see, other features install fine.

Any idea how to fix this? Thanks!

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 3
Comments: 39 (13 by maintainers)

Most upvoted comments

Is there anyway we can update the https://aka.ms/vscode-dev-containers/script-library/settings.env and remove the :80 reference?

I could help remove the port 80 suffix from GPG_KEY_SERVERS for the Features, but I’d like to confirm if it solves the issue for everyone and doesn’t cause a regression.

Can I get a 👍 if the following dev config works for you?


{
    "image": "mcr.microsoft.com/devcontainers/base:jammy",
    "features": {
        "ghcr.io/samruddhikhandale/test-terraform/terraform:1": {}
    }
}

I have created a temporary terraform Feature on my personal repo which removes :80

https://github.com/samruddhikhandale/test-terraform/blob/main/src/terraform/install.sh#L29

samruddhikhandale on Dec 20, 2022

Hi again @samruddhikhandale ,

Can confirm, the issue is the VPN. Tried without and built it without any issues.

alexandreczg on May 12, 2023

Do we need new issues opened for that?

You may, but I have already started working on updating all other Features. I would report here once completed. Shouldn’t take more than an hour.

samruddhikhandale on Dec 21, 2022

Your change seems to be working, thank you

nabil-becker on Dec 21, 2022

I was able to work around the issue.

I created a folder in the .devcontainers folder, called it 0terraform.

I then copied the devcontainer-feature.json and install.sh files for the terraform feature into that folder and updated the devcontainer.json to reference it.

It looks like at line 66 of the install.sh, it calls the get_common_setting function which looks like it downloads an env file (https://aka.ms/vscode-dev-containers/script-library/settings.env) and applies it.

I added lines to override the GPG_KEY_SERVERS variable, removing the :80 for the ubuntu keyserver, which resolved the issue.

Is there anyway we can update the https://aka.ms/vscode-dev-containers/script-library/settings.env and remove the :80 reference?

nabil-becker on Dec 20, 2022

I’m still getting the same issue, unfortunately.

But I’ve played around with it and discovered that the issue for me was port 80 on the ubuntu keyserver.

/tmp/tmp-gnupg/dirmngr.conf:

disable-ipv6
log-file /tmp/tmp-gnupg/dirmngr.log
keyserver hkp://keyserver.ubuntu.com:80
keyserver hkps://keys.openpgp.org
keyserver hkp://keyserver.pgp.com

Running the command gpg --verbose --recv-keys 72D7468F gave me the following:

gpg: keybox '/tmp/tmp-gnupg/pubring.kbx' created
gpg: no running Dirmngr - starting '/usr/bin/dirmngr'
gpg: waiting for the dirmngr to come up ... (5s)
gpg: connection to dirmngr established

And it just hangs there.

/tmp/tmp-gnupg/dirmngr.log:

2022-12-20 06:52:03 dirmngr[8928.0] permanently loaded certificates: 125
2022-12-20 06:52:03 dirmngr[8928.0]     runtime cached certificates: 0
2022-12-20 06:52:03 dirmngr[8928.0]            trusted certificates: 125 (124,0,0,1)
2022-12-20 06:52:03 dirmngr[8928.0] failed to open cache dir file '/tmp/tmp-gnupg/crls.d/DIR.txt': No such file or directory
2022-12-20 06:52:03 dirmngr[8928.0] creating directory '/tmp/tmp-gnupg/crls.d'
2022-12-20 06:52:03 dirmngr[8928.0] new cache dir file '/tmp/tmp-gnupg/crls.d/DIR.txt' created
2022-12-20 06:52:03 dirmngr[8928.6] resolve_dns_addr for 'keyserver.ubuntu.com': '162.213.33.9'
2022-12-20 06:52:03 dirmngr[8928.6] resolve_dns_addr for 'keyserver.ubuntu.com': '162.213.33.8'
2022-12-20 06:52:03 dirmngr[8928.6] number of system provided CAs: 124

If I change my dirmngr.conf to the following and run it again, it imports it successfully (i.e. remove the :80 on the ubuntu line):

disable-ipv6
log-file /tmp/tmp-gnupg/dirmngr.log
keyserver hkp://keyserver.ubuntu.com
keyserver hkps://keys.openpgp.org
keyserver hkp://keyserver.pgp.com

$ gpg --verbose --recv-keys 72D7468F
gpg: keybox '/tmp/tmp-gnupg/pubring.kbx' created
gpg: no running Dirmngr - starting '/usr/bin/dirmngr'
gpg: waiting for the dirmngr to come up ... (5s)
gpg: connection to dirmngr established
gpg: data source: http://162.213.33.9:11371
gpg: armor header: Comment: Hostname:
gpg: armor header: Version: Hockeypuck 2.1.0-189-g15ebf24
gpg: pub  rsa4096/34365D9472D7468F 2021-04-19  HashiCorp Security (hashicorp.com/security) <security@hashicorp.com>
gpg: /tmp/tmp-gnupg/trustdb.gpg: trustdb created
gpg: using pgp trust model
gpg: Note: signature key B0B441097685B676 expired Wed 20 Apr 2022 12:08:32 AM UTC
gpg: key 34365D9472D7468F: public key "HashiCorp Security (hashicorp.com/security) <security@hashicorp.com>" imported
gpg: no running gpg-agent - starting '/usr/bin/gpg-agent'
gpg: waiting for the agent to come up ... (5s)
gpg: connection to agent established
gpg: Total number processed: 1
gpg:               imported: 1

But I’ve tried each of the keyservers independently and found issues with all of them.

hkp://keyserver.pgp.com: seems to time out.

gpg: keyserver receive failed: Connection timed out

2022-12-20 07:05:09 dirmngr[12791.6] resolve_dns_addr for 'keyserver.pgp.com': '35.169.187.251'
2022-12-20 07:05:09 dirmngr[12791.6] resolve_dns_addr for 'keyserver.pgp.com': '18.206.36.226'
2022-12-20 07:05:09 dirmngr[12791.6] number of system provided CAs: 124
2022-12-20 07:05:24 dirmngr[12791.6] can't connect to '18.206.36.226': Connection timed out
2022-12-20 07:05:24 dirmngr[12791.6] error connecting to 'http://18.206.36.226:11371': Connection timed out
2022-12-20 07:05:24 dirmngr[12791.6] selecting a different host due to a timeout
2022-12-20 07:05:39 dirmngr[12791.6] can't connect to '18.206.36.226': Connection timed out
2022-12-20 07:05:39 dirmngr[12791.6] error connecting to 'http://18.206.36.226:11371': Connection timed out
2022-12-20 07:05:39 dirmngr[12791.6] selecting a different host due to a timeout
2022-12-20 07:05:54 dirmngr[12791.6] can't connect to '18.206.36.226': Connection timed out
2022-12-20 07:05:54 dirmngr[12791.6] error connecting to 'http://18.206.36.226:11371': Connection timed out
2022-12-20 07:05:54 dirmngr[12791.6] selecting a different host due to a timeout
2022-12-20 07:06:09 dirmngr[12791.6] can't connect to '35.169.187.251': Connection timed out
2022-12-20 07:06:09 dirmngr[12791.6] error connecting to 'http://35.169.187.251:11371': Connection timed out
2022-12-20 07:06:09 dirmngr[12791.6] command 'KS_GET' failed: Connection timed out

hkps://keys.openpgp.org: fails with the short format (works just fine with C874011F0AB405110D02105534365D9472D7468F).

gpg: data source: https://keys.openpgp.org:443
gpg: keyserver receive failed: No data

2022-12-20 07:12:03 dirmngr[14298.6] error accessing 'https://keys.openpgp.org:443/pks/lookup?op=get&options=mr&search=0x72D7468F': http status 400
2022-12-20 07:12:03 dirmngr[14298.6] command 'KS_GET' failed: No data

I’ve also tried getting the key via curl from the ubuntu keyserver on port 80, and it worked just fine: $ curl 'http://162.213.33.9:80/pks/lookup?op=get&options=mr&search=0x72D7468F'

I don’t know what can be done to fix this, though since it seems to be behaving differently for everyone. I just wanted to share my findings.

eitamal on Dec 20, 2022