eventing: [BUG] {"error":"error sending cloudevent: 404 Not Found"}

Describe the bug

I followed this guide https://knative.dev/docs/eventing/broker-trigger/ and https://knative.dev/docs/eventing/samples/kafka/channel/index.html

with both I get:

$ kubectl run -i --tty --rm debug --image=yauritux/busybox-curl --restart=Never --

If you don't see a command prompt, try pressing enter.

/home # curl -v "http://default-broker.default.svc.cluster.local/" \
>   -X POST \
>   -H "X-B3-Flags: 1" \
>   -H "CE-SpecVersion: 0.2" \
>   -H "CE-Type: dev.knative.foo.bar" \
>   -H "CE-Time: 2018-04-05T03:56:24Z" \
>   -H "CE-ID: 45a8b444-3213-4758-be3f-540bf93f85ff" \
>   -H "CE-Source: dev.knative.example" \
>   -H 'Content-Type: application/json' \
>   -d '{ "much": "wow" }'
* About to connect() to default-broker.default.svc.cluster.local port 80 (#0)
*   Trying 10.4.11.119...
* Adding handle: conn: 0x1180eda8
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x1180eda8) send_pipe: 1, recv_pipe: 0
* Connected to default-broker.default.svc.cluster.local (10.4.11.119) port 80 (#0)
> POST / HTTP/1.1
> User-Agent: curl/7.30.0
> Host: default-broker.default.svc.cluster.local
> Accept: */*
> X-B3-Flags: 1
> CE-SpecVersion: 0.2
> CE-Type: dev.knative.foo.bar
> CE-Time: 2018-04-05T03:56:24Z
> CE-ID: 45a8b444-3213-4758-be3f-540bf93f85ff
> CE-Source: dev.knative.example
> Content-Type: application/json
> Content-Length: 17
>
* upload completely sent off: 17 out of 17 bytes
< HTTP/1.1 400 Bad Request
< date: Mon, 04 Nov 2019 13:54:33 GMT
< content-length: 51
< content-type: text/plain; charset=utf-8
< x-envoy-upstream-service-time: 24
* Server envoy is not blacklisted
< server: envoy
<
* Connection #0 to host default-broker.default.svc.cluster.local left intact
{"error":"error sending cloudevent: 404 Not Found"}

on the broker ingress:

$ kubectl logs default-broker-ingress-54d485746c-9wrmt ingress
2019/11/04 13:56:49 Registering 1 clients
2019/11/04 13:56:49 Registering 0 informer factories
2019/11/04 13:56:49 Registering 0 informers
{"level":"info","ts":"2019-11-04T13:56:49.537Z","caller":"logging/config.go:108","msg":"Successfully created the logger.","knative.dev/jsonconfig":"{\n  \"level\": \"info\",\n  \"development\": false,\n  \"outputPaths\": [\"stdout\"],\n  \"errorOutputPaths\": [\"stderr\"],\n  \"encoding\": \"json\",\n  \"encoderConfig\": {\n    \"timeKey\": \"ts\",\n    \"levelKey\": \"level\",\n    \"nameKey\": \"logger\",\n    \"callerKey\": \"caller\",\n    \"messageKey\": \"msg\",\n    \"stacktraceKey\": \"stacktrace\",\n    \"lineEnding\": \"\",\n    \"levelEncoder\": \"\",\n    \"timeEncoder\": \"iso8601\",\n    \"durationEncoder\": \"\",\n    \"callerEncoder\": \"\"\n  }\n}\n"}
{"level":"info","ts":"2019-11-04T13:56:49.537Z","caller":"logging/config.go:109","msg":"Logging level set to info"}
{"level":"info","ts":"2019-11-04T13:56:49.562Z","logger":"broker_ingress","caller":"ingress/main.go:103","msg":"Starting the Broker Ingress","commit":"5fe2b0a"}
{"level":"info","ts":"2019-11-04T13:56:49.569Z","logger":"broker_ingress","caller":"metrics/exporter.go:94","msg":"Flushing the existing exporter before setting up the new exporter.","commit":"5fe2b0a"}
{"level":"info","ts":"2019-11-04T13:56:49.569Z","logger":"broker_ingress","caller":"metrics/prometheus_exporter.go:37","msg":"Created Opencensus Prometheus exporter with config: &{knative.dev/eventing broker_ingress prometheus 5000000000 9090 false  false   {    }}. Start the server for Prometheus exporter.","commit":"5fe2b0a"}
{"level":"info","ts":"2019-11-04T13:56:49.570Z","logger":"broker_ingress","caller":"metrics/exporter.go:103","msg":"Successfully updated the metrics exporter; old config: <nil>; new config &{knative.dev/eventing broker_ingress prometheus 5000000000 9090 false  false   {    }}","commit":"5fe2b0a"}
{"level":"info","ts":"2019-11-04T13:56:49.570Z","logger":"broker_ingress","caller":"configmap/store.go:157","msg":"tracing-config config \"config-tracing\" config was added or updated: &config.Config{Backend:\"none\", ZipkinEndpoint:\"\", StackdriverProjectID:\"\", Debug:false, SampleRate:0.1}","commit":"5fe2b0a"}
{"level":"info","ts":"2019-11-04T13:56:49.663Z","logger":"broker_ingress","caller":"ingress/main.go:169","msg":"Starting informers.","commit":"5fe2b0a"}
{"level":"warn","ts":1572875810.3738859,"logger":"fallback","caller":"http/transport.go:502","msg":"got an error from receiver fn","error":"error sending cloudevent: 404 Not Found"}
{"level":"warn","ts":1572875810.3740385,"logger":"fallback","caller":"http/transport.go:594","msg":"error returned from invokeReceiver","error":"error sending cloudevent: 404 Not Found"}

based on this yaml;

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: config-autoscaler
  namespace: knative-serving
data:
  enable-scale-to-zero: "false"
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: default-ch-webhook
  namespace: knative-eventing
data:
  default-ch-config: |
    clusterDefault:
      apiVersion: messaging.knative.dev/v1alpha1
      kind: InMemoryChannel
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: config-domain
  namespace: knative-serving
data:
 svc.cluster.local: ""
---
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: my-service
  namespace: default
spec:
  template:
    spec:
      containers:
        -  # This corresponds to
          # https://github.com/knative/eventing-contrib/blob/v0.2.1/cmd/message_dumper/dumper.go.
          image: gcr.io/knative-releases/github.com/knative/eventing-sources/cmd/message_dumper@sha256:ab5391755f11a5821e7263686564b3c3cd5348522f5b31509963afb269ddcd63
---
apiVersion: eventing.knative.dev/v1alpha1
kind: Trigger
metadata:
  name: my-service-trigger
  namespace: default
spec:
  broker: default # Defaulted by the Webhook.
  filter:
    attributes:
      type: dev.knative.foo.bar
  subscriber:
    ref:
      apiVersion: serving.knative.dev/v1
      kind: Service
      name: my-service
---
apiVersion: eventing.knative.dev/v1alpha1
kind: Broker
metadata:
  name: default
spec:
  channelTemplateSpec:
    apiVersion: messaging.knative.dev/v1alpha1
    kind: InMemoryChannel
---
apiVersion: sources.eventing.knative.dev/v1alpha1
kind: ContainerSource
metadata:
  name: heartbeats-sender
spec:
  template:
    spec:
      containers:
        - image: index.docker.io/daisyycguo/heartbeats-6790335e994243a8d3f53b967cdd6398
          name: heartbeats-sender
          args:
            - --sink=http://default-broker.default.svc.cluster.local
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
  sink:
    apiVersion: eventing.knative.dev/v1alpha1
    kind: Broker
    name: default

Expected behavior

The cloudevent should pass through the broker to the service my-service

To Reproduce

Install v0.9.0 or v0.10.0 and apply the above yaml

Knative release version

Same issue with v0.9.0 or v0.10.0 and InMemory and Kafka