envoy: Incorrect behavior with Routing based on JWT Token Dynamic Metadata

Title: Incorrect behavior with Routing based on JWT Token Dynamic Metadata

Description: When using the payload_in_metadata flag of the JWT Authentication filter, dynamic metadata is attached to the stream and visible in logging. However, this dynamic metadata cannot normally be used for HTTP routing.

If I set an unrelated header in lua, the dynamic metadata can be used for routing, see the repro steps below.

Repro steps: Run envoy (version: a9d72603c68da3a10a1c0d021d01c7877e6f2a30/1.21.0/Clean/RELEASE/BoringSSL) with the config provided pointing to the provided keys.json. Send an example request with a valid JWT and shard_id in the payload of either 1 or 2. Example requests that are valid with the provided keys.json are below.

(if you use the config as shown below)

$ curl <ip> -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiYWRtaW4iOnRydWUsImlhdCI6MTY0NDM0Mjg1Miwic2hhcmRfaWQiOiIyIn0.Y0iDgULUicqYV9v3-q_JulhonBtEjEFS4aawPj5sVU4cNVqE2sCQ9YtzO7LxqiL5FN4lcD7D__JsJ7swyB6Wpqf-DEqpE_vKsRCtPllBhArsDpNXiozWlBaJrhIM0L3OI9pMDQlQxU6BI3ZMux8K6OsaTTlqKnpcNnZ7BHTiYRIcbWzvNtQBaul-whdSpZwDH_cKzeoEAHeJyzMZ9iMSP_srlwXjI1iFHQt9BWXLnKDxUvosmg2ffrkDIiPmywdUTpoUU-FKnwahAoDb7h7Jvw3XcX0bobQ9JKKvp9bY8F7VDqy0KbQuw9DkB_aG754_KqMHuwSJX63sbJ_xFDKTww"
did not get to a specific cluster

(if you uncomment the single lua line request_handle:headers():add("a-header-that-doesnt-matter", "nop-value"))

$ curl  <ip> -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiYWRtaW4iOnRydWUsImlhdCI6MTY0NDM0Mjg1Miwic2hhcmRfaWQiOiIyIn0.Y0iDgULUicqYV9v3-q_JulhonBtEjEFS4aawPj5sVU4cNVqE2sCQ9YtzO7LxqiL5FN4lcD7D__JsJ7swyB6Wpqf-DEqpE_vKsRCtPllBhArsDpNXiozWlBaJrhIM0L3OI9pMDQlQxU6BI3ZMux8K6OsaTTlqKnpcNnZ7BHTiYRIcbWzvNtQBaul-whdSpZwDH_cKzeoEAHeJyzMZ9iMSP_srlwXjI1iFHQt9BWXLnKDxUvosmg2ffrkDIiPmywdUTpoUU-FKnwahAoDb7h7Jvw3XcX0bobQ9JKKvp9bY8F7VDqy0KbQuw9DkB_aG754_KqMHuwSJX63sbJ_xFDKTww"
this is cluster 2

I expect to get this is cluster 2 each time, as shard_id is set to 2 in the payload, but the routing based on dynamic metadata only works if the lua line is uncommented.

Config:

admin:
  address:
    socket_address:
      address: 127.0.0.1
      port_value: 9090
static_resources:
  listeners:
  - address:
      socket_address:
        address: 0.0.0.0
        port_value: 80
    per_connection_buffer_limit_bytes: 32768 # 32 KiB
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: http
          use_remote_address: true
          normalize_path: true
          merge_slashes: true
          path_with_escaped_slashes_action: UNESCAPE_AND_REDIRECT
          http2_protocol_options:
            max_concurrent_streams: 100
            initial_stream_window_size: 65536 # 64 KiB
            initial_connection_window_size: 1048576 # 1 MiB
          server_header_transformation: PASS_THROUGH
          common_http_protocol_options:
            idle_timeout: 900s
          stream_idle_timeout: 300s # 5 mins, must be disabled for long-lived and streaming requests
          request_timeout: 120s # 2 mins, must be disabled for long-lived and streaming requests
          request_headers_timeout: 15s
          stream_error_on_invalid_http_message: true
          route_config:
            virtual_hosts:
            - name: default
              domains:
              - "*"
              response_headers_to_remove:
                - x-envoy-upstream-service-time
              routes:
              - match:
                  prefix: /
                  dynamic_metadata:
                    - filter: envoy.filters.http.jwt_authn
                      path:
                      - key: payload
                      - key: shard_id
                      value:
                        string_match:
                          exact: "1"
                direct_response:
                  status: 200
                  body:
                    inline_string: |
                      this is cluster 1
              - match:
                  prefix: /
                  dynamic_metadata:
                    - filter: envoy.filters.http.jwt_authn
                      path:
                      - key: payload
                      - key: shard_id
                      value:
                        string_match:
                          exact: "2"
                direct_response:
                  status: 200
                  body:
                    inline_string: |
                      this is cluster 2
              - match:
                  prefix: /
                direct_response:
                  status: 200
                  body:
                    inline_string: |
                      did not get to a specific cluster
          access_log:
            - name: envoy.access_loggers.stderr
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StderrAccessLog
                log_format:
                  text_format: '[%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%" "%DYNAMIC_METADATA(envoy.filters.http.jwt_authn)%"\n'
          http_filters:
          - name: envoy.filters.http.grpc_json_transcoder
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.grpc_json_transcoder.v3.GrpcJsonTranscoder
              proto_descriptor: "/config/unrelated_descriptor_set.protoset"
          - name: envoy.filters.http.jwt_authn
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.jwt_authn.v3.JwtAuthentication
              providers:
                provider:
                  local_jwks:
                    filename: /config/keys.json
                  payload_in_metadata: payload
              rules:
                - match:
                    prefix: /
                  requires:
                    provider_name: provider
          - name: envoy.filters.http.lua
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
              inline_code: |
                function envoy_on_request(request_handle)
                  -- request_handle:headers():add("a-header-that-doesnt-matter", "nop-value")
                end
          - name: envoy.filters.http.router

keys.json:

{
  "keys": [
    {
      "kty": "RSA",
      "n": "u1SU1LfVLPHCozMxH2Mo4lgOEePzNm0tRgeLezV6ffAt0gunVTLw7onLRnrq0_IzW7yWR7QkrmBL7jTKEn5u-qKhbwKfBstIs-bMY2Zkp18gnTxKLxoS2tFczGkPLPgizskuemMghRniWaoLcyehkd3qqGElvW_VDL5AaWTg0nLVkjRo9z-40RQzuVaE8AkAFmxZzow3x-VJYKdjykkJ0iT9wCS0DRTXu269V264Vf_3jvredZiKRkgwlL9xNAwxXFg0x_XFw005UWVRIkdgcKWTjpBP2dPwVZ4WWC-9aGVd-Gyn1o0CLelf4rEjGoXbAAEgAqeGUxrcIlbjXfbcmw",
      "e": "AQAB",
      "alg": "RS256",
      "use": "sig"
    }
  ]
}

Logs: (no relevant logs in output)