ingress-nginx: ingress NGINX error curl: (92) HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)
NGINX Ingress controller version:
helm chart – stable/nginx-ingress:1.22.1 controller.image.tag – 0.26.1
Kubernetes version (use kubectl version):
v1.12.9-gke.15
Environment:
- Cloud provider or hardware configuration: GKE
- OS (e.g. from /etc/os-release):
- Kernel (e.g.
uname -a): - Install tools:
- Others:
What happened:
This is not curl problem, definitely problem in nginx-ingress and http2
When i’m trying to curl my endpoint
` Using HTTP2, server supports multi-use
- Connection state changed (HTTP/2 confirmed)
- Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
- Using Stream ID: 1 (easy handle 0x7febb1800400)
GET /api-console/api-model.json HTTP/2 Host: bs-tangerine-s2.tom.takeoff.com User-Agent: curl/7.54.0 Accept: /
- Connection state changed (MAX_CONCURRENT_STREAMS updated)! 0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0< HTTP/2 200 < server: nginx/1.15.9 < date: Tue, 01 Oct 2019 11:26:53 GMT < content-type: application/json < content-length: 596322 < vary: Accept-Encoding < last-modified: Fri, 27 Sep 2019 15:00:12 GMT < strict-transport-security: max-age=15724800; includeSubDomains < { [3907 bytes data] 85 582k 85 499k 0 0 186k 0 0:00:03 0:00:02 0:00:01 186k* HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1) 99 582k 99 579k 0 0 195k 0 0:00:02 0:00:02 --:–:-- 195k
- Closing connection 0
- TLSv1.2 (OUT), TLS alert, Client hello (1): } [2 bytes data] curl: (92) HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1) `
What you expected to happen:
Download JSON, works fine when disable http/2 and work with http1.1
How to reproduce it (as minimally and precisely as possible):
Create app and trying to download big JSON file
Anything else we need to know:
As workaround need to disable http/2
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 20 (6 by maintainers)
For the record, in my case it was a client bug in curl limited to Mac OS builds With curl built from https://github.com/curl/curl/commit/34a5400de113f2badfaa338b00cc361b4ab8355a the issue no longer occurs. curl never sent the whole request body but kept hanging consuming CPU time likely looping This caused 408 and 499 (timeout/Client Closed Request) on the NGINX side After NGINX’s proxy timeout was reached, I got
curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)See https://github.com/curl/curl/issues/1410
For me the issue was in the loadbalancer upstream that closed the connection as the idle time was reached, but it can be in several parts of your stack.
if you suspect the ingress controller (which for me wasn’t the culprit), try setting the debug logging to be very loud, you’d definately find your answer there 😃