spring-boot: Spring 1.4.4 Embedded Tomcat Stops Servicing HTTP requests.

Spring Boot 1.4.4.

We have a production system with a five node cluster running REST endpoints and using the embedded Tomcat container. The cluster serves roughly 600 million REST calls a week.

We recently upgrade our Spring Boot version from 1.4.1.RELEASE to 1.4.4.RELEASE. We experienced an “event” in which all five nodes stopped servicing http requests. These are running in VMs, within a data center.

The only clues we really have are in the logs we see this:

Exception in thread "http-nio-8080-Acceptor-0" java.lang.NoClassDefFoundError: org/apache/tomcat/util/ExceptionUtils
	at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:470)
	at java.lang.Thread.run(Thread.java:745)

We are wondering if something may have changed with respect to how Spring Boot accesses the nested .jar files within the fat jar?

After we observe the error, a thread dump indicates that the acceptor thread has exited, but the other threads/pools are still active. (The service continues to send heartbeats to Eurkea, we can still get into the JMX port)

The servers give NO indication of the real problem within the logs. The New Relic reports look healthy on those servers.

The only thing we are seeing is that the clients (accessing the rest end points) are using a sidecar and that sidecar is correctly opening the circuit.

The real puzzle to us was that restarting, services resulted in the server serving up some http traffic and then again failing with the same error. This was a transient problem, as we rolled back to a previous version and then forward again and we are not seeing the errors.

This may be environmental, but I am just looking for help in where we might look?

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 23 (12 by maintainers)

Most upvoted comments

I’ve done some experimentation with this app that closes its ClassLoader as soon as its running:

package com.example;

import java.io.IOException;
import java.net.URLClassLoader;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@SpringBootApplication
@RestController
public class Gh8260Application {

	public static void main(String[] args) throws IOException {
		SpringApplication.run(Gh8260Application.class, args);
		((URLClassLoader)Gh8260Application.class.getClassLoader()).close();
	}

	@RequestMapping("/")
	public String hi() {
		return "Hi";
	}

}

Making a request to / will reproduce the failure described above:

Exception in thread "http-nio-8080-Acceptor-0" java.lang.NoClassDefFoundError: org/apache/tomcat/util/ExceptionUtils
	at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:470)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.tomcat.util.ExceptionUtils
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 2 more

Another way to reproduce this exact failure is to start an application, delete its jar file and then make a request to it.

@tkvangorder I can’t be sure that either of these scenarios are what happened in your case, but they do both produce the exact same failure as you suffered. We don’t have any code in Boot that would close the ClassLoader (or to delete a running jar!) but perhaps something else did it?

I tried changing my app to ensure that ExceptionUtils was loaded before closing the ClassLoader and I’m not convinced that it really made the diagnostics any better. Things may be different for you, of course, as my attempt at reproducing the problem is rather contrived but it means that I don’t think we should add some code to Boot to aggressively load ExceptionUtils.

To summarise, my working theory is that something closed the application’s ClassLoader or deleted that jar from underneath the application. At that point all bets are off and I don’t think there’s anything we could or should do to defend against either of them.

If we do encounter an issue again, I will make sure to add any additional information.

Thanks. I’m going to close this one for now. If the problem does occur again, please add any additional information to this issue and I’m more than happy to take another look.

wilkinsona on Feb 15, 2017

Perhaps we should initialize ExceptionUtils early ourselves when the server starts?

philwebb on Apr 4, 2017