ignition: Ignition appears to swallow S3 errors

Issue by @colhom

Issue Report

Bug

Container Linux Version

stable @ 1520.8.0 ignition config v2.1.0

Environment

AWS (us-west-2, m3.medium, t2.medium)

Expected Behavior

Ignition fetches s3 bucket object content and uses as a config.

S3 object exists and is accessible to IAM profile of ec2 instance hosting ignition
Tried with and without a verification hash

Actual Behavior

ignition-disks fails to fetch the config contents from s3, with the following output:

[    7.007728] systemd-networkd[234]: eth0: Gained IPv6LL
(1 of 2) A start job is running for…-mapper-usr.device (8s / no limit)
(1 of 2) A start job is running for…-mapper-usr.device (9s / no limit)
(1 of 2) A start job is running for…-mapper-usr.device (9s / no limit)
(2 of 2) A start job is running for Ignition (disks) (10s / no limit)
(2 of 2) A start job is running for Ignition (disks) (10s / no limit)
... 
(2 of 2) A start job is running for…ition (disks) (1min 1s / no limit)
(2 of 2) A start job is running for…ition (disks) (1min 1s / no limit)
(2 of 2) A start job is running for…ition (disks) (1min 2s / no limit)
(1 of 2) A start job is running for…er-usr.device (1min 2s / no limit)
(1 of 2) A start job is running for…er-usr.device (1min 3s / no limit)
[   66.496720] ignition[387]: failed to fetch config: RequestCanceled: request context canceled
caused by: context deadline exceeded
[   66.501156] ignition[387]: failed to acquire config: RequestCanceled: request context canceled
caused by: context deadline exceeded
Failed to start Ignition (disks).
See 'systemctl status ignition-disks.service' for details.
Dependency failed for Ignition (files).
Dependency failed for Ignition (record completion).
Dependency failed for Initrd Default Target.

Naive analysis indicates it’s hitting the context deadline either when fetching the IAM credentials or the s3 object contents.

Reproduction Steps

Host ignition config as s3 bucket object

{
    "ignition": {
      "config": {
        "append": [
          {
            "source": "s3://<config-s3-bucket>/ign/registry-cache-masters.json",
            "verification": {}
          },
          {
            "source": "s3://<config-s3-bucket>/ign/custom-cacerts-masters.json",
            "verification": {}
          }
        ]
      },
      "timeouts": {},
      "version": "2.1.0"
    }

\cc @dgonyeo

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 32 (31 by maintainers)

Most upvoted comments

Closing due to inactivity, feel free to reopen if you’re still hitting this.

ajeddeloh on Jan 3, 2020

Going to post here since it might help someone else out. My Ignition config was keeping my CoreOS instance from coming up on EC2 (as described above). If I removed the s3 remote files, it worked. I was getting these types of errors when viewing the EC2 System Log (Actions > Instance Settings > Get System Log):

RequestError: send request failed
caused by: dial tcp 54.231.168.248:443: i/o timeout

The IP in the error message above is for s3 but the problem was actually that my box couldn’t access the internet at all during first boot. My plan was to boot the instance and then associate an Elastic IP. However, due to being in a VPC (without a NAT instance), a public IP is required in order for the box to access the internet (and s3). For more info, see https://forums.aws.amazon.com/thread.jspa?threadID=96369.

The solution was to set “Auto-assign Public IP” to “Enable” when launching the instance. This allows the box (and Ignition) to have access to the internet (and s3) on first boot. Then once the machine was up, I associated the Elastic IP (which dropped the temporary public IP that it was assigned when it first booted).

So keep in mind that adding remote s3 files will make your machine dependent on an internet connection in order to launch. This could maybe be added as an error message hint if the s3 remote request fails.

collinkrawll on Sep 27, 2018