wasmCloud: [BUG] Trace is not using OTEL_EXPORTER_OTLP_TRACES_ENDPOINT env variable even though it set

Affected project(s)

  • documentation
  • examples
  • wasmCloud host
  • wasmCloud CLI (wash)
  • wasmCloud dashboard UI (washboard)
  • capability providers
  • provider bindgen
  • control interface client
  • other / not sure

Describe the bug

I am setting the environment variable OTEL_EXPORTER_OTLP_TRACES_ENDPOINT to send traces to my custom endpoint but it seems wasmcloud is still trying to send to default http://localhost:4318/v1/traces) .

Followed this doc : https://wasmcloud.com/docs/deployment/observability/tracing

Steps to reproduce

I have used the following environment variables to setup traces using otel and export it to my end point (instead of default http://localhost:4318/v1/traces) .

export WASMCLOUD_TRACES_ENABLED="true"
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://myendpoint.com/ot/v1/traces
wash up

I have a wasm component with wadm file as below

spec:
  components:
    - name: http-component
      type: component
      properties:
        image: file://./build/http_client_tinygo_s.wasm
      traits:
        # Govern the spread/scheduling of the component
        - type: spreadscaler
          properties:
            replicas: 1
        - type: link
          properties:
            target: httpclient
            namespace: wasi
            package: http
            interfaces: [outgoing-handler]

    # Add a capability provider that enables HTTP access
    - name: httpserver
      type: capability
      properties:
        image: myhost.com/ghcr-io-proxy/ghcr.io/wasmcloud/http-server:0.21.0
      traits:
        # Link the httpserver to the component, and configure the HTTP server
        # to listen on port 8080 for incoming requests
        - type: link
          properties:
            target: http-component
            namespace: wasi
            package: http
            interfaces: [incoming-handler]
            source_config:
              - name: default-http
                properties:
                  address: 127.0.0.1:8080

    - name: httpclient
      type: capability
      properties:
        image: myhost.com/ghcr-io-proxy/ghcr.io/wasmcloud/http-client:0.11.0 

I did wash build and wash app deploy wadm.yaml

The app is started and I am able hit the curl http://localhost:8080 and see the result.

I expect traces to be sent to my endpoint, but in the terminal I see below error message where it is still trying to send to localhost

OpenTelemetry trace error occurred. error sending request for url (http://localhost:4318/v1/traces): error trying to connect: tcp connect error: Connection refused (os error 61)

Other Options tried: I saw this code and thought I can send wash up --override-traces-endpoint=<myendpoint>, but seems wash up cli command doesn’t support that. https://github.com/wasmCloud/wasmCloud/blob/d92af18f183ba36e3794d4983905a7437ff2efce/src/main.rs#L259-L261

Expected behavior

I expect traces to be sent to my endpoint, but in the terminal I see below error message where it is still trying to send to localhost

OpenTelemetry trace error occurred. error sending request for url (http://localhost:4318/v1/traces): error trying to connect: tcp connect error: Connection refused (os error 61)

Environment

  • OS: MacOS
  • Shell: zsh
  • wash version: 0.29.2

Screenshots / Logs / Additional context

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 15 (9 by maintainers)

Most upvoted comments

Hey @saikiran1043 we’ve just released wasmcloud-provider-sdk v0.9.1 which has a fix to this feature, but what you need to do is use provider configuration to set the OTEL variables.

Long story short, the choice to do it this way makes sense for two reasons:

  • Provider config is language/platform agnostic
  • Some OTEL clients ignore whatever value you give it, and always use the values available in ENV. This means that if we pass values from the host, it will be later impossible to use distinct OTEL backends for host and launched providers.

So, if you’re building a new provider, you can use the new wasmcloud-provider-sdk – and if you’re not, we’ll have new versions of all the providers released very soon!

This PR has examples of what the config would look like, for your reference 😃

Thanks for checking! It seems we might have figured out what was wrong!

How can I pass the env (or) OTEL env values to the provider?

So unfortunately, right now you can’t – we don’t yet have a great solution for this, unfortunately.

How do I see env values in provider individually?

Since the provider is a subprocess its a bit hard to see the ENV values provided to the provider from the outside, if you’re running on Linux could do:

cat /sys/<pid>/environ

That said, since we have the code I can tell you that the only env var passed through is RUST_LOG.

Do we have any option in the wadm.yaml to pass the env variables when specifying the capabilities?

Yes!

Providers can be configured (here’s an example) – the config that you pass in there (either just named, or defined inline) can be used during provider setup to perform the configuration you’d like.

What we need to do is make that possible/easy to do from the initialize_observability macro that providers should call.

I’ll get working on this and get a change in to provider-sdk

[EDIT] One thing that is odd is that providers are supposed to inherit their settings from the host – it looks like that’s not happening, so I’ll see if I can find the reason for that first.

sure @brooksmtownsend , I will update my wash version and try this.