istio: when VirtualService Status Error，istiod can not start

istio version: 1.18.3 kubernetes:1.18.10

Bug description when VirtualService config error, istiod can not started

istiod error log:

{"level":"error","time":"2021-02-23T06:40:18.954868Z","scope":"klog","msg":"k8s.io/client-go@v0.19.3/tools/cache/reflector.go:156: Failed to watch *v1alpha3.VirtualService: failed to list *v1alpha3.VirtualService: v1alpha3.VirtualServiceList.Items: []v1alpha3.VirtualService: v1alpha3.VirtualService.v1alpha3.VirtualService.Status: unmarshalerDecoder: unknown value \"Error\" for enum istio.analysis.v1alpha1.AnalysisMessageBase_Level, error found in #10 byte of ...|l+v1\\\"\"}]}},{\"apiVer|..., bigger context ...|\"map-nginx.project-1925.svc.cluster.local+v1\\\"\"}]}},{\"apiVersion\":\"networking.istio.io/v1alpha3\",\"ki|...[]"}

error VirtualService config

spec:
exportTo:
- .
gateways:
- gateway-122-71
hosts:
- nginx-map.gw-wso2t-sy-in.earth.xcloud.lenovo.com
http:
- match:
- uri:
prefix: /
rewrite:
uri: /
route:
- destination:
host: map-nginx.project-1925.svc.cluster.local
port:
number: 80
subset: v1
weight: 100
status:
validationMessages:
- code: IST0101
documentation_url: https://istio.io/docs/reference/config/analysis/ist0101/?ref=status-controller
level: Error
message: 'Referenced host+subset in destinationrule not found: "map-nginx.project-1925.svc.cluster.local+v1"'

[ ] Docs [ ] Installation [ ] Networking [ ] Performance and Scalability [ ] Extensions and Telemetry [ ] Security [X] Test and Release [ ] User Experience [ ] Developer Infrastructure [ ] Upgrade

Expected behavior

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 20 (15 by maintainers)

Most upvoted comments

I am test in istio 1.6.8,1.6.13 1.7.4 ,1.7.6,1.8.3

public-system(the namespace publish service to outward) has gateway, virtualservice(exportTo: .)

project-1925(the namespace publish service to namespace`s service) has service, deploy, destinationrule, virtualservice(exportTo: .)

In <=1.7.4 public-system`s virtualservice has under error, but istod can restart

validationMessages:
- code: IST0101
documentation_url: https://istio.io/docs/reference/config/analysis/ist0101/?ref=status-controller
level: Error
message: 'Referenced host+subset in destinationrule not found: "map-nginx.project-1925.svc.cluster.local+v1"'

in 1.7.6 public-system`s virtualservice no error

upgrade from 1.7.4 to 1.8.3, istiod(1.8.3) canot not start

upgrade from 1.7.4 to 1.7.6, the public-system`s virtualservice error remain, but istiod(1.7.6) can started, and service can be connected.

upgrade from 1.7.4 to 1.7.6, then upgrade from 1.7.6 to 1.8.3, the public-system`s virtualservice error remain, but istiod(1.8.3) can not started,

{"level":"error","time":"2021-03-22T10:39:08.252656Z","scope":"klog","msg":"k8s.io/client-go@v0.19.3/tools/cache/reflector.go:156: Failed to watch *v1alpha3.VirtualService: failed to list 
*v1alpha3.VirtualService: v1alpha3.VirtualServiceList.Items: []v1alpha3.VirtualService: v1alpha3.VirtualService.v1alpha3.VirtualService.Status: unmarshalerDecoder: unknown value \"Error\" 
for enum istio.analysis.v1alpha1.AnalysisMessageBase_Level, error found in #10 byte of ...|l+v1\\\"\"}]}},{\"apiVer|..., bigger context ...|esh-b.project-1896.svc.frankfurtinp.local+v1\\\"
\"}]}},{\"apiVersion\":\"networking.istio.io/v1alpha3\",\"ki|...[]"}

install 1.7.6, then upgrade upgrade from 1.7.6 to 1.8.3 the public-system`s virtualservice no error, istiod(1.8.3) can started

I think This issues has two bug:

vs check has a bug, destination no support other namespaces destinationrule .

vs in public-system

http:                                                                                                                                                                                     
  - match:                                                                                                                                                                                  
    - uri:                                                                                                                                                                                  
        prefix: /                                                                                                                                                                           
    rewrite:                                                                                                                                                                                
      uri: /                                                                                                                                                                                
    route:                                                                                                                                                                                  
    - destination:                                                                                                                                                                          
        host: map-nginx.project-1896.svc.cluster.local                                                                                                                               
        port:                                                                                                                                                                               
          number: 80                                                                                                                                                                        
        subset: v1                                                                                                                                                                          
      weight: 10

istiod(1.8.3) has a bug for parse VirtualService that has Error(level: Error)

status:                                                                                                                                                                                   
    validationMessages:                                                                                                                                                                     
    - code: IST0101                                                                                                                                                                         
      documentation_url: https://istio.io/docs/reference/config/analysis/ist0101/?ref=status-controller                                                                                     
      level: Error      #this line  parse Error                                                                                                                                                                    
      message: 'Referenced host+subset in destinationrule not found: "map-nginx.project-1896.svc.cluster.local+v1"

zhengqisong on Mar 22, 2021

To manually put garbage into status, first install Istio with WITHOUT values.global.istiod.enableAnalysis=true and WITHOUT values.pilot.env.PILOT_ENABLE_STATUS=true. Then

kubectl proxy &
curl -k -s -X PATCH -H "Accept: application/json, */*" \
-H "Content-Type: application/merge-patch+json" \
127.0.0.1:8001/apis/networking.istio.io/v1beta1/namespaces/default/virtualservices/httpbin/status \
--data '{"status":{"monkey":"banana","conditions":"improving}}'

With master and daily builds, confirmed it logs

error	klog	k8s.io/client-go@v0.20.4/tools/cache/reflector.go:167: Failed to watch *v1alpha3.VirtualService: failed to list *v1alpha3.VirtualService: v1alpha3.VirtualServiceList.ListMeta: v1.ListMeta.TypeMeta: Kind: Items: []v1alpha3.VirtualService: v1alpha3.VirtualService.Status: unmarshalerDecoder: json: cannot unmarshal string into Go value of type []json.RawMessage, error found in #10 byte of ...|tion":"1"}}],"kind":|..., bigger context ...|oving","monkey":"banana","observedGeneration":"1"}}],"kind":"VirtualServiceList","metadata":{"contin|...
error	klog	k8s.io/client-go@v0.20.4/tools/cache/reflector.go:167: Failed to watch *v1alpha3.VirtualService: failed to list *v1alpha3.VirtualService: v1alpha3.VirtualServiceList.Items: []v1alpha3.VirtualService: v1alpha3.VirtualService.v1alpha3.VirtualService.Status: unmarshalerDecoder: json: cannot unmarshal string into Go value of type []json.RawMessage, error found in #10 byte of ...|tion":"1"}},{"apiVer|..., bigger context ...|oving","monkey":"banana","observedGeneration":"1"}},{"apiVersion":"networking.istio.io/v1alpha3","ki|...

esnible on Mar 19, 2021