ingress-nginx: Default Backend Annotation Incompatible with custom-http-errors

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT, i think

NGINX Ingress controller version: 0.22.0 Kubernetes version (use kubectl version): 1.12.4 What happened: If you specify a default-backend override in an ingress resource while also including custom-http-errors, I believe the proxy-intercept-errors are not respecting the annotation default-backend.

What you expected to happen: That you can use custom-http-errors to proxy_intercept_errors to your custom default backend of your choosing

How to reproduce it (as minimally and precisely as possible): The yaml below has the service scaled to 0. This means the service WILL 503.
Using the yaml files referenced below, deploy it to a kube cluster (and adjust the ingress route to meet your cluster needs). Deploy it once with JUST the default-backend annotation. You will get your default backend 503 error as defined by this image. quay.io/kubernetes-ingress-controller/custom-error-pages-amd64:0.3. Now uncomment the #nginx.ingress.kubernetes.io/custom-http-errors: 404,503` annotation and it will no longer goto your default backend.

# ECHOSERVER EXAMPLE APP
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: echo-svc
spec:
  replicas: 0
  selector:
    matchLabels:
      app: echo-svc
  template:
    metadata:
      labels:
        app: echo-svc
    spec:
      containers:
      - name: echo-svc
        image: gcr.io/kubernetes-e2e-test-images/echoserver:2.1
        ports:
        - containerPort: 8080
        env:
          - name: NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
---
apiVersion: v1
kind: Service
metadata:
  name: echo-svc
  labels:
    app: echo-svc
spec:
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
    
    name: http
  selector:
    app: echo-svc
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/default-backend: nginx-errors-svc # This is referencing the SAME NAMESPACE that this resource is in
   #nginx.ingress.kubernetes.io/custom-http-errors: 404,503
  name: echo-app-ingress
spec:
  rules:
  - host: echo.domain.com
    http:
      paths:
      - backend:
          serviceName: echo-svc
          servicePort: http
        path: /
---
# ECHOSERVER CUSTOM DEFAULT BACKEND (ATTEMPTING TO OVERRIDE THE DEFAULT BACKEND DEFINED IN THE CONTROLLER INSTALLATION)
apiVersion: v1
kind: Service
metadata:
  name: nginx-errors-svc
  labels:
    app.kubernetes.io/name: nginx-errors
    app.kubernetes.io/part-of: ingress-nginx
spec:
  selector:
    app.kubernetes.io/name: nginx-errors
    app.kubernetes.io/part-of: ingress-nginx
  ports:
  - port: 80
    targetPort: 8080
    name: http
---
apiVersion: apps/v1beta2
kind: Deployment
apiVersion: apps/v1beta2
metadata:
  name: nginx-errors
  labels:
    app.kubernetes.io/name: nginx-errors
    app.kubernetes.io/part-of: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: nginx-errors
      app.kubernetes.io/part-of: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: nginx-errors
        app.kubernetes.io/part-of: ingress-nginx
    spec:
      containers:
      - name: nginx-error-server
        image: quay.io/kubernetes-ingress-controller/custom-error-pages-amd64:0.3
        ports:
        - containerPort: 8080

Anything else we need to know:

Since the default-backend annotation code only adds an override for 503 - there isn’t a real way to have custom-http-errors and a per-namespace default backend override with this bug