caddy: Running Caddy as a reverse proxy is causing JetBrains teamware to crash, issue with content-type headers?

We are trying to run JetBrains Teamware, a set of Java apps that have their own webservers (Jetty, it seems), through a Caddy reverse proxy. We’re having an issue, however, with the apps crashing under Caddy when they talk to JetBrains’ authentication service, which we think has something to do with the content-type header.

1. What version of Caddy are you running (caddy -version)?

Caddy 0.8.2

2. What are you trying to do?

Basically, each of the apps (a CD service, a code review service, etc) talks to a JetBrains “Hub,” or authentication service, that operates with oAuth and a few other things. Each of the apps has an internal “Hub” but in order to tie them together you need to be running an external instance of a hub. That’s fine, but when they boot up and try and connect to the Hub under Caddy, we encounter the issue specified here: https://youtrack.jetbrains.com/oauth?state=%2Fissue%2FJPS-3181 - with exactly the same error output.

We think the problem may be outlined in this comment:

Do you use some proxy for Hub? Looks like Content-type in response was changed to “text/plain”. My colleague told me that Hub may response only with “application/json”"

But through our poking around doesn’t actually appear that the content-type on that request is getting changed/added via Caddy – and nothing in the Caddy code looks to us like the content-types would be getting replaced.

Here’s our access log for those requests:

54.210.68.167 - [04/Apr/2016:14:31:30 +0000] GET /hub/api/rest/oauth2/target text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2 200 9
54.210.68.167 - [04/Apr/2016:14:31:30 +0000] GET /hub/api/rest/oauth2/target text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2 200 9
54.210.68.167 - [04/Apr/2016:14:31:30 +0000] POST /hub/api/rest/oauth2/token application/json 200 207
54.210.68.167 - [04/Apr/2016:14:31:30 +0000] GET /hub/api/rest/services/6af2617b-9215-46b8-bece-ab42dcf8a1ba application/json 200 77
54.210.68.167 - [04/Apr/2016:14:31:30 +0000] GET /hub/api/rest/oauth2/target text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2 200 9
54.210.68.167 - [04/Apr/2016:14:31:30 +0000] GET /hub/api/rest/oauth2/target text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2 200 9
54.210.68.167 - [04/Apr/2016:14:31:30 +0000] POST /hub/api/rest/oauth2/token application/json 200 207
54.210.68.167 - [04/Apr/2016:14:31:30 +0000] GET /hub/api/rest/usergroups/root application/json 200 64
54.210.68.167 - [04/Apr/2016:14:31:31 +0000] POST /hub/api/rest/oauth2/token application/json 200 207
54.210.68.167 - [04/Apr/2016:14:31:31 +0000] GET /hub/api/rest/services/6af2617b-9215-46b8-bece-ab42dcf8a1ba application/json 200 1268
54.210.68.167 - [04/Apr/2016:14:31:31 +0000] POST /hub/api/rest/services/6af2617b-9215-46b8-bece-ab42dcf8a1ba application/json 200 0

But we do not have a problem running the app without Caddy, so it must be an issue with Caddy. Our initial thought was to overwrite the headers anyway, but we can’t manually specify mimetypes due to #666 – they’d simply get appended, rather than replaced.

Curiously, according to the docs:

Normally, Content-Type is detected automatically, but this is not always possible. If you encounter responses with the wrong Content-Type, you can use this middleware to correct it.

but again, we don’t see where in Caddy the Content-Type would be getting inferred/set in the first place. Where in the Caddy code would that automatic detection be taking place?

The Hub docs suggest this as the setup for an Nginx/Apache proxy, and we think we emulated that with Caddy, except for the part about ‘DefaultType,’ directive which Caddy does not have. (And if I knew how the content-type headers were getting coerced, I’d just build one.)

3. What is your entire Caddyfile?

hub.tw.mydomain.com {
  proxy / localhost:8081 {
    proxy_header Host {host}
    proxy_header X-Real-IP {remote}
    proxy_header X-Forwarded-Proto {scheme}
    proxy_header X-Forwarded-Host {host}
    websocket
  }

  log /hub/api/rest /var/log/access.log "{remote} - [{when}] {method} {path} {>Accept} {status} {size}" 
}

upsource.tw.mydomain.com {
  proxy / localhost:8082 {
    proxy_header Host {host}
    proxy_header X-Real-IP {remote}
    proxy_header X-Forwarded-Proto {scheme}
    proxy_header X-Forwarded-Host {host}
    websocket
  }
}

youtrack.tw.mydomain.com {
  proxy / localhost:8083 {
    proxy_header Host {host}
    proxy_header X-Real-IP {remote}
    proxy_header X-Forwarded-Proto {scheme}
    proxy_header X-Forwarded-Host {host}
    websocket
  }
}

teamcity.tw.mydomain.com {
  proxy / localhost:8084 {
    proxy_header Host {host}
    proxy_header X-Real-IP {remote}
    proxy_header X-Forwarded-Proto {scheme}
    proxy_header X-Forwarded-Host {host}
    websocket
  }
}

4. How did you run Caddy (give the full command and describe the execution environment)?

We’re running it via a systemd service, in Centos 6, under a non-root centos user.

5. What did you expect to see?

We expect the JetBrains teamware not to crash. 😉

6. What did you see instead (give full error messages and/or log)?

The error linked above. Specifically, on our machine (sorry for the fugly Java stacktrace):

youtrack/bin/youtrack.sh run
Starting YouTrack...
* Configuring JetBrains YouTrack 6.5 
* Loading logging configuration from /opt/youtrack/lib/ext/log4j.xml 
* Redirecting JetBrains YouTrack 6.5 logging to /opt/youtrack/logs/internal/services/bundleProcess 
* Configuring Service-Container[bundleProcess] 
* Configuring Starting Page Service 
* Configuring YouTrack Configurator 
* Configuring Bundle Hub Configurator 
* Configuring YouTrack 
* Starting Service-Container[bundleProcess] 
* Starting Starting Page Service 
* JetBrains YouTrack 6.5 will be available on [https://youtrack.tw.mydomain.com/bundle/starting] after start 
* Starting YouTrack Configurator 
* Starting Bundle Hub Configurator 
[YouTrack Error] [2016-04-04 16:38:10,038]  ERROR - dle.util.hub.HubServiceUpdater - Cannot update services information in Hub 
[YouTrack Error] org.glassfish.jersey.message.internal.MessageBodyProviderNotFoundException: MessageBodyReader not found for media type=text/plain;charset=utf-8, type=class jetbrains.jetpass.rest.dto.ServiceJSON, genericType=class jetbrains.jetpass.rest.dto.ServiceJSON.
[YouTrack Error]    at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.aroundReadFrom(ReaderInterceptorExecutor.java:232)
[YouTrack Error]    at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:156)
[YouTrack Error]    at org.glassfish.jersey.message.internal.MessageBodyFactory.readFrom(MessageBodyFactory.java:1085)
[YouTrack Error]    at org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:853)
[YouTrack Error]    at org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:785)
[YouTrack Error]    at org.glassfish.jersey.client.ClientResponse.readEntity(ClientResponse.java:326)
[YouTrack Error]    at org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:790)
[YouTrack Error]    at org.glassfish.jersey.client.JerseyInvocation.access$500(JerseyInvocation.java:91)
[YouTrack Error]    at org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:687)
[YouTrack Error]    at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
[YouTrack Error]    at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
[YouTrack Error]    at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
[YouTrack Error]    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:444)
[YouTrack Error]    at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:683)
[YouTrack Error]    at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:439)
[YouTrack Error]    at org.glassfish.jersey.client.JerseyInvocation$Builder.post(JerseyInvocation.java:340)
[YouTrack Error]    at jetbrains.jetpass.client.accounts.ServiceClient.updateService(ServiceClient.java:89)
[YouTrack Error]    at com.jetbrains.bundle.util.hub.HubServiceUpdater.compareAndUpdateParameters(HubServiceUpdater.java:244)
[YouTrack Error]    at com.jetbrains.bundle.util.hub.HubServiceUpdater.updateServicesInfoInHub(HubServiceUpdater.java:267)
[YouTrack Error]    at com.jetbrains.bundle.util.hub.HubServiceUpdater.update(HubServiceUpdater.java:55)
[YouTrack Error]    at com.jetbrains.bundle.services.impl.HubConfiguratorService.doStart(HubConfiguratorService.java:65)
[YouTrack Error]    at com.jetbrains.bundle.services.impl.ServiceBase.start(ServiceBase.java:62)
[YouTrack Error]    at com.jetbrains.bundle.Services.startService(Services.java:518)
[YouTrack Error]    at com.jetbrains.bundle.Services.startAllServices(Services.java:490)
[YouTrack Error]    at com.jetbrains.bundle.Services.start(Services.java:468)
[YouTrack Error]    at com.jetbrains.bundle.BundleMain.start(BundleMain.java:248)
[YouTrack Error]    at com.jetbrains.bundle.BundleMain.start(BundleMain.java:208)
[YouTrack Error]    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[YouTrack Error]    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[YouTrack Error]    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[YouTrack Error]    at java.lang.reflect.Method.invoke(Method.java:497)
[YouTrack Error]    at com.jetbrains.bundle.bootstrap.Bundle$EntryPointProvider$1.invoke(Bundle.java:108)
[YouTrack Error]    at com.sun.proxy.$Proxy2.start(Unknown Source)
[YouTrack Error]    at com.jetbrains.bundle.bootstrap.Bundle.start(Bundle.java:54)
[YouTrack Error]    at com.jetbrains.launcher.AppProxy$4.call(AppProxy.java:83)
[YouTrack Error]    at com.jetbrains.launcher.AppProxy$4.call(AppProxy.java:81)
[YouTrack Error]    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[YouTrack Error]    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[YouTrack Error]    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[YouTrack Error]    at java.lang.Thread.run(Thread.java:745)
[YouTrack Error] [2016-04-04 16:38:10,042]  ERROR - es.impl.HubConfiguratorService - Cannot update Hub properties: MessageBodyReader not found for media type=text/plain;charset=utf-8, type=class jetbrains.jetpass.rest.dto.ServiceJSON, genericType=class jetbrains.jetpass.rest.dto.ServiceJSON. 
[YouTrack Error] org.glassfish.jersey.message.internal.MessageBodyProviderNotFoundException: MessageBodyReader not found for media type=text/plain;charset=utf-8, type=class jetbrains.jetpass.rest.dto.ServiceJSON, genericType=class jetbrains.jetpass.rest.dto.ServiceJSON.
[YouTrack Error]    at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.aroundReadFrom(ReaderInterceptorExecutor.java:232)
* Error while starting JetBrains YouTrack 6.5: MessageBodyReader not found for media type=text/plain;charset=utf-8, type=class jetbrains.jetpass.rest.dto.ServiceJSON, genericType=class jetbrains.jetpass.rest.dto.ServiceJSON. 
* Stopping YouTrack 
* Stopping Bundle Hub Configurator 
* Stopping YouTrack Configurator 
[YouTrack Error]    at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:156)
[YouTrack Error]    at org.glassfish.jersey.message.internal.MessageBodyFactory.readFrom(MessageBodyFactory.java:1085)
[YouTrack Error]    at org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:853)
[YouTrack Error]    at org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:785)
[YouTrack Error]    at org.glassfish.jersey.client.ClientResponse.readEntity(ClientResponse.java:326)
[YouTrack Error]    at org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:790)
[YouTrack Error]    at org.glassfish.jersey.client.JerseyInvocation.access$500(JerseyInvocation.java:91)
[YouTrack Error]    at org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:687)
[YouTrack Error]    at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
[YouTrack Error]    at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
[YouTrack Error]    at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
[YouTrack Error]    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:444)
[YouTrack Error]    at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:683)
[YouTrack Error]    at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:439)
[YouTrack Error]    at org.glassfish.jersey.client.JerseyInvocation$Builder.post(JerseyInvocation.java:340)
[YouTrack Error]    at jetbrains.jetpass.client.accounts.ServiceClient.updateService(ServiceClient.java:89)
[YouTrack Error]    at com.jetbrains.bundle.util.hub.HubServiceUpdater.compareAndUpdateParameters(HubServiceUpdater.java:244)
[YouTrack Error]    at com.jetbrains.bundle.util.hub.HubServiceUpdater.updateServicesInfoInHub(HubServiceUpdater.java:267)
[YouTrack Error]    at com.jetbrains.bundle.util.hub.HubServiceUpdater.update(HubServiceUpdater.java:55)
[YouTrack Error]    at com.jetbrains.bundle.services.impl.HubConfiguratorService.doStart(HubConfiguratorService.java:65)
[YouTrack Error]    at com.jetbrains.bundle.services.impl.ServiceBase.start(ServiceBase.java:62)
[YouTrack Error]    at com.jetbrains.bundle.Services.startService(Services.java:518)
[YouTrack Error]    at com.jetbrains.bundle.Services.startAllServices(Services.java:490)
[YouTrack Error]    at com.jetbrains.bundle.Services.start(Services.java:468)
[YouTrack Error]    at com.jetbrains.bundle.BundleMain.start(BundleMain.java:248)
[YouTrack Error]    at com.jetbrains.bundle.BundleMain.start(BundleMain.java:208)
[YouTrack Error]    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[YouTrack Error]    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[YouTrack Error]    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[YouTrack Error]    at java.lang.reflect.Method.invoke(Method.java:497)
[YouTrack Error]    at com.jetbrains.bundle.bootstrap.Bundle$EntryPointProvider$1.invoke(Bundle.java:108)
[YouTrack Error]    at com.sun.proxy.$Proxy2.start(Unknown Source)
[YouTrack Error]    at com.jetbrains.bundle.bootstrap.Bundle.start(Bundle.java:54)
[YouTrack Error]    at com.jetbrains.launcher.AppProxy$4.call(AppProxy.java:83)
[YouTrack Error]    at com.jetbrains.launcher.AppProxy$4.call(AppProxy.java:81)
[YouTrack Error]    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[YouTrack Error]    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[YouTrack Error]    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[YouTrack Error]    at java.lang.Thread.run(Thread.java:745)
[YouTrack Error] [2016-04-04 16:38:10,044]  ERROR -                 bundle.startup - Error while starting JetBrains YouTrack 6.5: MessageBodyReader not found for media type=text/plain;charset=utf-8, type=class jetbrains.jetpass.rest.dto.ServiceJSON, genericType=class jetbrains.jetpass.rest.dto.ServiceJSON. 
* Stopping Starting Page Service 
* Stopping Service-Container[bundleProcess] 
[YouTrack Error] [2016-04-04 16:38:10,080]   WARN - y.util.thread.QueuedThreadPool - qtp2042924422{STOPPING,8<=8<=200,i=6,q=6} Couldn't stop Thread[qtp2042924422-31,5,main] 
[YouTrack Error] [2016-04-04 16:38:10,080]   WARN - y.util.thread.QueuedThreadPool - qtp2042924422{STOPPING,8<=8<=200,i=6,q=6} Couldn't stop Thread[qtp2042924422-32,5,main] 
[YouTrack Error] [2016-04-04 16:38:10,080]   WARN - y.util.thread.QueuedThreadPool - qtp2042924422{STOPPING,8<=8<=200,i=6,q=6} Couldn't stop Thread[qtp2042924422-34,5,main] 
[YouTrack Error] [2016-04-04 16:38:10,080]   WARN - y.util.thread.QueuedThreadPool - qtp2042924422{STOPPING,8<=8<=200,i=6,q=6} Couldn't stop Thread[qtp2042924422-36,5,main] 
[YouTrack Error] [2016-04-04 16:38:10,081]   WARN - y.util.thread.QueuedThreadPool - qtp2042924422{STOPPING,8<=8<=200,i=6,q=6} Couldn't stop Thread[qtp2042924422-37,5,main] 
[YouTrack Error] [2016-04-04 16:38:10,081]   WARN - y.util.thread.QueuedThreadPool - qtp2042924422{STOPPING,8<=8<=200,i=4,q=5} Couldn't stop Thread[qtp2042924422-60,5,main] 
YouTrack process finished
Launcher is exiting

The full debug output is here.

Any suggestions? We’re also going to submit this issue to JetBrains, but we’re pretty sure it has something to do with Caddy because simply accessing the Hub on its port works just fine.

(Tagging my colleague, @mnaughto, who’s helping me with this.)