traefik: Malformed/garbled HTML response when compress is enabled
What version of Traefik are you using (traefik version)?
1.1.2-a
What is your environment & configuration (arguments, toml…)?
# traefik.toml
logLevel = "INFO"
defaultEntryPoints = ["http"]
[entryPoints]
[entryPoints.http]
address = ":80"
compress = true
[kubernetes]
[web]
address = ":8081"
traefik service.yaml
# Source: traefik/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
name: rtraefik-traefik
labels:
app: rtraefik-traefik
chart: "traefik-1.1.2-a"
release: "rtraefik"
heritage: "Tiller"
spec:
type: LoadBalancer
selector:
app: rtraefik-traefik
ports:
- port: 80
name: http
- port: 443
name: https
traefik deployment.yaml
# Source: traefik/templates/deployment.yaml
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: rtraefik-traefik
labels:
app: rtraefik-traefik
chart: "traefik-1.1.2-a"
release: "rtraefik"
heritage: "Tiller"
spec:
template:
metadata:
labels:
app: rtraefik-traefik
chart: "traefik-1.1.2-a"
release: "rtraefik"
heritage: "Tiller"
spec:
terminationGracePeriodSeconds: 60
hostNetwork: true
containers:
- image: traefik:v1.1.2
name: rtraefik-traefik
resources:
requests:
cpu: "100m"
memory: "20Mi"
limits:
cpu: "100m"
memory: "30Mi"
readinessProbe:
tcpSocket:
port: 80
failureThreshold: 1
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 2
livenessProbe:
tcpSocket:
port: 80
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 2
volumeMounts:
- mountPath: /config
name: config
ports:
- containerPort: 80
- containerPort: 443
- containerPort: 8081
args:
- --configfile=/config/traefik.toml
volumes:
- name: config
configMap:
name: rtraefik-traefik
kubectl get ingress rjenkins-jenkins -o yaml
$ kubectl get ingress rjenkins-jenkins -o yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: '{"kind":"Ingress","apiVersion":"extensions/v1beta1","metadata":{"name":"rjenkins-jenkins","namespace":"default","selfLink":"/apis/extensions/v1beta1/namespaces/default/ingresses/rjenkins-jenkins","uid":"fd3bb42f-df57-11e6-bb42-0050568a2a2f","resourceVersion":"1052907","generation":2,"creationTimestamp":"2017-01-20T21:32:48Z","labels":{"app":"rjenkins-jenkins","chart":"traefik-1.1.2-a","heritage":"Tiller","release":"rtraefik"}},"spec":{"rules":[{"host":"jenkins.example.com","http":{"paths":[{"backend":{"serviceName":"rjenkins-jenkins","servicePort":8080}}]}}]},"status":{"loadBalancer":{}}}'
creationTimestamp: 2017-01-20T21:32:48Z
generation: 2
labels:
app: rjenkins-jenkins
chart: traefik-1.1.2-a
heritage: Tiller
release: rtraefik
name: rjenkins-jenkins
namespace: default
resourceVersion: "1053907"
selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/rjenkins-jenkins
uid: fd3bb42f-df57-11e6-bb42-0050568a2a2f
spec:
rules:
- host: jenkins.example.com
http:
paths:
- backend:
serviceName: rjenkins-jenkins
servicePort: 8080
status:
loadBalancer: {}
What did you do?
I’m using traefik in kubernetes, installed by helm. I have a jenkins instance that I’m trying to proxy to. I’m also using traefik to proxy traefik’s dashboard and prometheus server + alertmanager.
What did you expect to see?
I expected Jenkins page to load successfully. Other ingress resources like traefik’s dashboard and prometheus are successfully loading.
What did you see instead?
Jenkins gives malformed HTTP response. The headers look OK and in-tact but the HTML payload is malformed. This happens on all browsers tested including OSX Chrome and Firefox. My best guess is that browsers are trying to gzip decode the response when the response isn’t being compressed. See screenshots:
Jenkins will load OK if I hit the service IP and port directly:
The headers from the malformed response:
The raw response from the malformed response:
curl http://jenkins.example.com/login -v
$ curl http://jenkins.example.com/login -v
* Trying 10.163.148.196...
* TCP_NODELAY set
* Connected to jenkins.example.com (10.163.148.196) port 80 (#0)
> GET /login HTTP/1.1
> Host: jenkins.example.com
> User-Agent: curl/7.51.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Cache-Control: no-cache,no-store,must-revalidate
< Content-Type: text/html;charset=UTF-8
< Date: Mon, 23 Jan 2017 21:23:39 GMT
< Expires: 0
< Server: Jetty(9.2.z-SNAPSHOT)
< Set-Cookie: JSESSIONID.ab758541=tzum86vvk6es1iw413gtdcsql;Path=/;HttpOnly
< Vary: Accept-Encoding
< X-Content-Type-Options: nosniff
< X-Frame-Options: sameorigin
< X-Hudson: 1.395
< X-Hudson-Cli-Port: 50000
< X-Hudson-Theme: default
< X-Instance-Identity: MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAv0hMZsLNMdkrNHMnwFvR8Ks+xjLupJnNzofDhIJaswpRfcFN1LvrzahQatRmbka7AhDt3G90Syo+3gaCMUxAFI0Px61nhTJ8hE8hhgNf6yYUnfxg0HRJ4j3ddumCvnQoPeq8x/rd+wxkZW7u/IpJHoS6ShFUzrfTuZSWI8v4k3A9SwXm+4x9r7c0VduEl56wefaBnSRi8cJXbXxyjtZ4/v4rurh6rnSoorlsTAmx1R9M+jZ9JFedFRc1e54Mnob1FOAAyvQryAr+HAGRAANv/EGIJ4FZBxjGMhliBzo6oifvaY6C/St/7r2FHFcp1VD4mt7naKgbaTbKExcZfsn1vwIDAQAB
< X-Jenkins: 2.19.4
< X-Jenkins-Cli-Port: 50000
< X-Jenkins-Cli2-Port: 50000
< X-Jenkins-Session: d200f825
< X-Ssh-Endpoint: jenkins.example.com:35985
< Transfer-Encoding: chunked
<
<!DOCTYPE html><html><head resURL="/static/d200f825" data-rooturl="" data-resurl="/static/d200f825">
<title>Jenkins</title><link rel="stylesheet" href="/static/d200f825/css/layout-common.css" type="text/css" /><link rel="stylesheet" href="/static/d200f825/css/style.css" type="text/css" /><link rel="stylesheet" href="/static/d200f825/css/color.css" type="text/css" /><link rel="stylesheet" href="/static/d200f825/css/responsive-grid.css" type="text/css" /><link rel="shortcut icon" href="/static/d200f825/favicon.ico" type="image/vnd.microsoft.icon" /><link color="black" rel="mask-icon" href="/images/mask-icon.svg" /><script>var isRunAsTest=false; var rootURL=""; var resURL="/static/d200f825";</script><script src="/static/d200f825/scripts/prototype.js" type="text/javascript"></script><script src="/static/d200f825/scripts/behavior.js" type="text/javascript"></script><script src='/adjuncts/d200f825/org/kohsuke/stapler/bind.js' type='text/javascript'></script><script src="/static/d200f825/scripts/yui/yahoo/yahoo-min.js"></script><script src="/static/d200f825/scripts/yui/dom/dom-min.js"></script><script src="/static/d200f825/scripts/yui/event/event-min.js"></script><script src="/static/d200f825/scripts/yui/animation/animation-min.js"></script><script src="/static/d200f825/scripts/yui/dragdrop/dragdrop-min.js"></script><script src="/static/d200f825/scripts/yui/container/container-min.js"></script><script src="/static/d200f825/scripts/yui/connection/connection-min.js"></script><script src="/static/d200f825/scripts/yui/datasource/datasource-min.js"></script><script src="/static/d200f825/scripts/yui/autocomplete/autocomplete-min.js"></script><script src="/static/d200f825/scripts/yui/menu/menu-min.js"></script><script src="/static/d200f825/scripts/yui/element/element-min.js"></script><script src="/static/d200f825/scripts/yui/button/button-min.js"></script><script src="/static/d200f825/scripts/yui/storage/storage-min.js"></script><script src="/static/d200f825/scripts/hudson-behavior.js" type="text/javascript"></script><script src="/static/d200f825/scripts/sortable.js" type="text/javascript"></script><script>crumb.init("", "");</script><link rel="stylesheet" href="/static/d200f825/scripts/yui/container/assets/container.css" type="text/css" /><link rel="stylesheet" href="/static/d200f825/scripts/yui/assets/skins/sam/skin.css" type="text/css" /><link rel="stylesheet" href="/static/d200f825/scripts/yui/container/assets/skins/sam/container.css" type="text/css" /><link rel="stylesheet" href="/static/d200f825/scripts/yui/button/assets/skins/sam/button.css" type="text/css" /><link rel="stylesheet" href="/static/d200f825/scripts/yui/menu/assets/skins/sam/menu.css" type="text/css" /><meta name="ROBOTS" content="INDEX,NOFOLLOW" /><script src="/static/d200f825/jsbundles/page-init.js" type="text/javascript"></script></head><body data-model-type="hudson.model.Hudson" id="jenkins" class="yui-skin-sam jenkins-2.19.4 two-column" data-version="2.19.4"><a href="#skip2content" class="skiplink">Skip to content</a><div id="page-head"><div id="header"><div class="logo"><a id="jenkins-home-link" href="/"><img src="/static/d200f825/images/headshot.png" alt="title" id="jenkins-head-icon" /><img src="/static/d200f825/images/title.png" alt="title" width="139" id="jenkins-name-icon" height="34" /></a></div><div class="login"> <a href="/loginEntry?from=%2F"><b>log in</b></a></div><div class="searchbox hidden-xs"><form method="get" name="search" action="/search/" style="position:relative;" class="no-json"><div id="search-box-minWidth"></div><div id="search-box-sizer"></div><div id="searchform"><input name="q" placeholder="search" id="search-box" class="has-default-text" /> <a href="http://wiki.jenkins-ci.org/display/JENKINS/Search+Box"><img src="/static/d200f825/images/16x16/help.png" style="width: 16px; height: 16px; " class="icon-help icon-sm" /></a><div id="search-box-completion"></div><script>createSearchBox("/search/");</script></div></form></div></div><div id="breadcrumbBar"><tr id="top-nav"><td id="left-top-nav" colspan="2"><link rel='stylesheet' href='/adjuncts/d200f825/lib/layout/breadcrumbs.css' type='text/css' /><script src='/adjuncts/d200f825/lib/layout/breadcrumbs.js' type='text/javascript'></script><div class="top-sticker noedge"><div class="top-sticker-inner"><div id="right-top-nav"></div><ul id="breadcrumbs"><li class="item"><a href="/" class="model-link inside">Jenkins</a></li><li href="/" class="children"></li></ul><div id="breadcrumb-menu-target"></div></div></div></td></tr></div></div><div id="page-body" class="clear"><div id="side-panel"></div><div id="main-panel"><a name="skip2content"></a><div style="margin: 2em;"><form method="post" name="login" action="j_security_check" style="text-size:smaller"><table><tr><td>User:</td><td><input type="text" name="j_username" id="j_username" autocorrect="off" autocapitalize="off" /></td></tr><tr><td>Password:</td><td><input type="password" name="j_password" /></td></tr></table><input name="from" type="hidden" /><input name="Submit" type="submit" value="log in" class="submit-button primary" /><script>
$('j_username').focus();
* Curl_http_done: called premature == 0
* Connection #0 to host jenkins.example.com left intact
</script></form></div></div></div><footer><div class="container-fluid"><div class="row"><div class="col-md-6" id="footer"></div><div class="col-md-18"><span class="page_generated">Page generated: Jan 23, 2017 9:23:39 PM UTC</span><span class="rest_api"><a href="api/">REST API</a></span><span class="jenkins_ver"><a href="http://jenkins-ci.org/">Jenkins ver. 2.19.4</a></span></div></div></div></footer></body></html>
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Reactions: 1
- Comments: 16 (6 by maintainers)
Commits related to this issue
- Disable gzip in nginx See discussion at https://github.com/containous/traefik/issues/1060 — committed to data61/anonlink-entity-service by hardbyte 7 years ago
- Disable gzip in nginx See discussion at https://github.com/containous/traefik/issues/1060 — committed to data61/anonlink-entity-service by hardbyte 7 years ago
I opened a PR https://github.com/NYTimes/gziphandler/pull/29 which should fix this in the compression handler.
Hi gang! Maintainer of the gziphandler here.
I’d prefer our library to follow the principle of least surprise and simply compress any/all requests that flow through it.
If you’re running a mix of compressed and non-compressed endpoints on the same server, I would suggest using two separate middleware stacks and wrapping the endpoints accordingly. For a good example of this practice, take a look at how Go Kit composes endpoints and middleware in it’s examples: https://github.com/go-kit/kit/tree/master/examples
As for Traefik, I would suggest updating the service to move the compress configuration to individual back end services for a finer control.
I’m also using (or attempting to) use with helm, though turning off gzip doesn’t seem to resolve the issue for me. (v1.1.2-g) – not really sure what options I have in this state.
This can be looked at a little differently: traefik is forwarding an “Accept-Encoding: gzip” header, which depending on the fix (and current state), traefik does not support. It’s expecting to get a non-encoded/non-compressed response, but telling the service it accepts the encoding and promptly ignoring the Content-Encoding header sent back.
To add the feature of content-encoding (compression) at a proxy level, in HTTP, you can’t get away from handling & processing certain headers in some way or another (e.g. Transfer-Encoding, Content-Encoding, Content-Length, maybe others) – since the response from the service and response to the client differ.
I’m thinking we have three options to solve this without edge cases:
1 - side-steps the whole thing, but it’s a useful feature and feels a bit extreme
2 - should make things work, though not ideal since the service could be highly compressible, and the service could have cached the data. Meaning it would be far more efficient to let it respond with the encoded data directly. We would also lose support for any other content-encoding types the backend may support (like br) by going through traefik.
3 - IMO the ideal state: only enable gzip (at a traefik level) in one specific case:
I think “3” highlights the user’s expectation of what should happen. The key piece is only encoding the response to the client, if “Content-Encoding” is unset on the service response – meaning it’s not already encoded in whatever format (that might not be gzip anyway).
Ok before fixing this we should decide if we want behaviour like in @bakins https://github.com/NYTimes/gziphandler/pull/29, in which case we should fork gziphandler or write our own.
Or we make compression configurable at the backend level…
Or Both…
Personally I think pushing this down to the backend level is a reasonable thing to do, but then it would need to be supported on each provider…