terrakube: Output and State not being saved to S3 or Minio buckets,
Bug description 🐞
Running into 404 errors when attempting to run a job:
S3 Not found: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID...
My Terrakube behind a proxy; currently it is configured for a real S3 bucket with Access Keys, I also saw this error when it was configured with the default minio container.
I don’t know if the proxy is causing this, I’ve experimented with having terrakube-minio and the S3 endpoint on the non-proxy list for the api/executor/registry and it hasn’t made a difference.
Steps to reproduce
Attempt to run a job in a workspace. Job will hang under queue status infinitely, says ‘Waiting Logs’ when expanding the terminal output window. Sometimes it will be marked as a fail in the UI.
Expected behavior
Job completes.
Example repository
No response
Anything else?
No response
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 31 (15 by maintainers)
This conversation continued in my Gist but to wrap up: @alfespa17 and I figured out the issue. There were several problems:
The semicolon separating the fields in the registry query calls were not being encoded by Traefik. This is disabled by default for some reason. I set two env vars to true:
TRAEFIK_ENTRYPOINTS_WEB_HTTP_ENCODEQUERYSEMICOLONSandTRAEFIK_ENTRYPOINTS_WEBSECURE_HTTP_ENCODEQUERYSEMICOLONS. Web and websecure are arbitrary names for the 80/443 entry points respectively. This change enabled the modules to partially load in the registry.A custom header for Terrakube was missing,
x-terraform-get. Adding that to theaccessControlExposeHeadersfield enabled full functionality. Reminder: when authorization header is enabled, headers must be explicitly set,*does not work in this scenario.An ancillary problem @alfespa17 solved was getting Terrakube to accept a self-signed cert. I need to study his method and will update this post after I have replicated his work.
Glad we got this problem solved 🥳🚀
There were a few more issues after my last post, occurring only behind my institutions’ web proxy, everything worked on my local:
I believe the TK team made some proxy changes in 2.21.2 version because I am happy to say everything is working behind a web proxy 🚀!!
One issue I discovered with the TK Registry: Your tags/versions/releases must have 3 digits, e.g.
1.1.1, if your tag is1.1the registry container will append a.0to the search path in S3, then fail because it doesn’t exist. I am unsure if this is a bug or part of the Terraform registry spec.I am glad that you got it working 👍
Big progress – Successful Terraform Apply! Let me sum up what I’ve done: Remember I am using the docker compose method, modified to have everything behind the Reverse Proxy Traefik on ingress. I am in an AWS account behind my company’s Webproxy on egress.
Set DNS
Had to set the DNS as the same one the EC2 used, I think requests to terraform.io were going to the wrong IP until this was changed.
Use HTTP_PROXY, http_proxy ENV Vars and JAVA_TOOL_OPTIONS
This was a big one, if you’re behind a Webproxy you need both the
AND you need the traditional
The Java app uses the JAVA_TOOL_OPTIONS config while the container uses the HTTP_PROXY config. When pulling from the terraform registry the container is doing the pulling not the Java app.
Fix CORS
For my setup everything talks to each other through Traefik, no web container has a port (LDAP/Redis/Postgres do have ports attached). I didn’t have CORS setup correctly my final setup looks like this:
I had wildcards (“*”) for
accesscontrolallowheadersandaccesscontrolalloworiginlistYou cannot use wildcards with CORS whenaccesscontrolallowcredentials: trueMust specify an origin and have Authorization specified as a headerThanks for the example, since you have 8080 as the port, I am going to assume this is pointing to your API server and not the UI address which by default has port 3000 on the host.
ETA: I got it working! It was the silliest thing: I had a typo for the
AzBuilderApiUrlvariable in the executor env. Executor could not find API I guess 😃