traefik: Internal server error in buffering

Do you want to request a feature or report a bug?

Bug

What did you do?

I used labels in docker-compose.yml to configure the buffering of a backend:

  nginx:
    image: nginx
    labels:
      traefik.backend: 'nginx'
      traefik.backend.buffering.maxRequestBodyBytes: '8589934592'
      traefik.backend.buffering.maxResponseBodyBytes: '8589934592'
      traefik.backend.buffering.memRequestBodyBytes: '268436456'
      traefik.backend.buffering.memResponseBodyBytes: '268435456'
      traefik.backend.buffering.retryExpression: 'IsNetworkError() && Attempts() <= 2'
      traefik.enable: 'true'
      traefik.frontend.rule: 'Host: nginx.example.com'
      traefik.port: '80'
    networks:
    - web
    restart: always
    volumes:
    - ...

And I got an Internal Server Error when accessed the Host: nginx.example.com . ~~The docker-compose.yml works well in traefik v1.7.7.~~ And I’m sure that the backend works well too. When I removed the labels about buffering, the traefik works normally!

What did you expect to see?

Access the host normally.

What did you see instead?

500: Internal Server Error

Output of `traefik version`: (What version of Traefik are you using?)

Version:      v1.7.8
Codename:     maroilles
Go version:   go1.11.5
Built:        2019-01-29_04:33:36PM
OS/Arch:      linux/amd64

What is your environment & configuration (arguments, toml, provider, platform, …)?

debug = true
defaultEntryPoints = ['https', 'http']
logLevel = 'DEBUG'

[entryPoints]
  [entryPoints.http]
    address = ':80'
    [entryPoints.http.redirect]
      entryPoint = "https"
  [entryPoints.https]
    address = ':443'
    [entryPoints.https.tls]

[traefikLog]
  filePath = 'traefik.log'
  format = 'json'

[accessLog]
  format = 'json'

[docker]
  endpoint = 'unix:///var/run/docker.sock'
  exposedByDefault = false
  domain = 'example.com'
  network = 'web'
  watch = true

If applicable, please paste the log output in DEBUG level (`--logLevel=DEBUG` switch)

I found these messages in the debug log:

{
  "level": "debug",
  "msg": "Upstream ResponseWriter of type *pipelining.writerWithoutCloseNotify does not implement http.CloseNotifier. Returning dummy channel.",
  "time": "2019-02-03T11:38:05Z"
}
{
  "level": "error",
  "msg": "vulcand/oxy/buffer: failed to read response, err: no data ready",
  "time": "2019-02-03T11:38:06Z"
}
{
  "level": "debug",
  "msg": "'500 Internal Server Error' caused by: no data ready",
  "time": "2019-02-03T11:38:06Z"
}

And I think the vulcand/oxy/buffer caused the issue. The whole DEBUG log is debug.log. I think my report will help you!

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 1
Comments: 22

Most upvoted comments

Hey @mpl, hitting the same issue OP posted. Running traefik:2.1.2. Hopefully I could give some insight on repro steps. Run this python script to simulate the 302 redirect (as mentioned above, uncommenting the Content-Length header fixes the problem). $ python3 server.py 7777 http://localhost:8080/

#!/usr/bin/env python3

import sys
from http.server import HTTPServer, BaseHTTPRequestHandler

class Redirect(BaseHTTPRequestHandler):
   def do_GET(self):
       self.send_response(302)
       self.send_header('Location', sys.argv[2])
#       self.send_header('Content-Length', 0)
       self.end_headers()

HTTPServer(("", int(sys.argv[1])), Redirect).serve_forever()

dynamic_conf.yml (replace LOCAL_IP with something traefik can access)

http:
  routers:
    test:
      entryPoints:
        - web
      service: content
      rule: "PathPrefix(`/test`)"
      middlewares:
        - max_request_body_10_megabytes

  middlewares:
    max_request_body_10_megabytes:
      buffering:
        maxRequestBodyBytes: 1000000

  services:
    content:
      loadBalancer:
        passHostHeader: false
        servers:
          - url: "http://LOCAL_IP:7777/"

traefik.yml

providers:
  file:
    filename: /etc/traefik/dynamic_conf.yml
    watch: true

entryPoints:
  web:
    address: ":80"

$ curl -v localhost/test
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 80 (#0)
> GET /test HTTP/1.1
> Host: localhost
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 500 Internal Server Error
< Date: Thu, 23 Jan 2020 10:34:02 GMT
< Content-Length: 21
< Content-Type: text/plain; charset=utf-8

kbaldyga on Jan 23, 2020

However, RFC 7230 lists the few responses without a body, 302 is not one of them:

3.3 Message Body

[…] The presence of a message body in a response depends on both the request method to which it is responding and the response status code (Section 3.1.2). Responses to the HEAD request method (Section 4.3.2 of [RFC7231]) never include a message body because the associated response header fields (e.g., Transfer-Encoding, Content-Length, etc.), if present, indicate only what their values would have been if the request method had been GET (Section 4.3.1 of [RFC7231]). 2xx (Successful) responses to a CONNECT request method (Section 4.3.6 of [RFC7231]) switch to tunnel mode instead of having a message body. All 1xx (Informational), 204 (No Content), and 304 (Not Modified) responses do not include a message body. All other responses do include a message body, although the body might be of zero length.

It seems the link in the body is not required but the Content-Length: 0 is still a requirement.

kmichel2 on Mar 2, 2020

The RFC 7231, published in 2007 and obsoleting the RFC 2616, was reworded to remove the requirement to have a body with an hyperlink to the new location :

6.4.3 302 Found

The 302 (Found) status code indicates that the target resource resides temporarily under a different URI. Since the redirection might be altered on occasion, the client ought to continue to use the effective request URI for future requests.

The server SHOULD generate a Location header field in the response containing a URI reference for the different URI. The user agent MAY use the Location field value for automatic redirection. The server’s response payload usually contains a short hypertext note with a hyperlink to the different URI(s).