wasmCloud: [BUG] Trace is not using OTEL_EXPORTER_OTLP_TRACES_ENDPOINT env variable even though it set
Affected project(s)
- documentation
- examples
- wasmCloud host
- wasmCloud CLI (wash)
- wasmCloud dashboard UI (washboard)
- capability providers
- provider bindgen
- control interface client
- other / not sure
Describe the bug
I am setting the environment variable OTEL_EXPORTER_OTLP_TRACES_ENDPOINT to send traces to my custom endpoint but it seems wasmcloud is still trying to send to default http://localhost:4318/v1/traces) .
Followed this doc : https://wasmcloud.com/docs/deployment/observability/tracing
Steps to reproduce
I have used the following environment variables to setup traces using otel and export it to my end point (instead of default http://localhost:4318/v1/traces) .
export WASMCLOUD_TRACES_ENABLED="true"
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://myendpoint.com/ot/v1/traces
wash up
I have a wasm component with wadm file as below
spec:
components:
- name: http-component
type: component
properties:
image: file://./build/http_client_tinygo_s.wasm
traits:
# Govern the spread/scheduling of the component
- type: spreadscaler
properties:
replicas: 1
- type: link
properties:
target: httpclient
namespace: wasi
package: http
interfaces: [outgoing-handler]
# Add a capability provider that enables HTTP access
- name: httpserver
type: capability
properties:
image: myhost.com/ghcr-io-proxy/ghcr.io/wasmcloud/http-server:0.21.0
traits:
# Link the httpserver to the component, and configure the HTTP server
# to listen on port 8080 for incoming requests
- type: link
properties:
target: http-component
namespace: wasi
package: http
interfaces: [incoming-handler]
source_config:
- name: default-http
properties:
address: 127.0.0.1:8080
- name: httpclient
type: capability
properties:
image: myhost.com/ghcr-io-proxy/ghcr.io/wasmcloud/http-client:0.11.0
I did wash build and wash app deploy wadm.yaml
The app is started and I am able hit the curl http://localhost:8080 and see the result.
I expect traces to be sent to my endpoint, but in the terminal I see below error message where it is still trying to send to localhost
OpenTelemetry trace error occurred. error sending request for url (http://localhost:4318/v1/traces): error trying to connect: tcp connect error: Connection refused (os error 61)
Other Options tried:
I saw this code and thought I can send wash up --override-traces-endpoint=<myendpoint>, but seems wash up cli command doesn’t support that.
https://github.com/wasmCloud/wasmCloud/blob/d92af18f183ba36e3794d4983905a7437ff2efce/src/main.rs#L259-L261
Expected behavior
I expect traces to be sent to my endpoint, but in the terminal I see below error message where it is still trying to send to localhost
OpenTelemetry trace error occurred. error sending request for url (http://localhost:4318/v1/traces): error trying to connect: tcp connect error: Connection refused (os error 61)
Environment
- OS: MacOS
- Shell: zsh
- wash version: 0.29.2
Screenshots / Logs / Additional context
No response
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 15 (9 by maintainers)
Hey @saikiran1043 we’ve just released
wasmcloud-provider-sdkv0.9.1 which has a fix to this feature, but what you need to do is use provider configuration to set the OTEL variables.Long story short, the choice to do it this way makes sense for two reasons:
So, if you’re building a new provider, you can use the new
wasmcloud-provider-sdk– and if you’re not, we’ll have new versions of all the providers released very soon!This PR has examples of what the config would look like, for your reference 😃
Thanks for checking! It seems we might have figured out what was wrong!
So unfortunately, right now you can’t – we don’t yet have a great solution for this, unfortunately.
Since the provider is a subprocess its a bit hard to see the ENV values provided to the provider from the outside, if you’re running on Linux could do:
That said, since we have the code I can tell you that the only env var passed through is
RUST_LOG.Yes!
Providers can be configured (here’s an example) – the config that you pass in there (either just named, or defined inline) can be used during provider setup to perform the configuration you’d like.
What we need to do is make that possible/easy to do from the
initialize_observabilitymacro that providers should call.I’ll get working on this and get a change in to
provider-sdk[EDIT] One thing that is odd is that providers are supposed to inherit their settings from the host – it looks like that’s not happening, so I’ll see if I can find the reason for that first.
sure @brooksmtownsend , I will update my wash version and try this.