prometheus-plugin: Stack overflow when scraping metrics

Hello,

I’m trying to scrape the prometheus metrics endpoint however always get a timeout when using a proxy. It only occurred to me today to try and poke the endpoint directly with a generous timeout for curl and here’s what I got:

java.lang.StackOverflowError
	at org.jenkinsci.plugins.workflow.graph.FlowNode.loadParents(FlowNode.java:149)
	at org.jenkinsci.plugins.workflow.graph.FlowNode.getParents(FlowNode.java:137)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.forEach(FlowNodes.java:44)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.forEach(FlowNodes.java:44)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.forEach(FlowNodes.java:44)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.forEach(FlowNodes.java:44)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.forEach(FlowNodes.java:44)
(...)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.forEach(FlowNodes.java:44)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.forEach(FlowNodes.java:44)
	at org.jenkinsci.plugins.prometheus.util.FlowNodes.forEach(FlowNodes.java:44)
Caused: javax.servlet.ServletException
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:765)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
	at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:374)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
	at org.jenkinsci.plugins.corsfilter.AccessControlsFilter.doFilter(AccessControlsFilter.java:79)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
	at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:225)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
	at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:51)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
	at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
	at com.smartcodeltd.jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:47)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
	at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
	at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
	at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:49)
	at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:44)
	at hudson.plugins.scm_sync_configuration.ScmSyncConfigurationDataProvider.provideRequestDuring(ScmSyncConfigurationDataProvider.java:106)
	at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter.doFilter(ScmSyncConfigurationFilter.java:44)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:138)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:92)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.Server.handle(Server.java:564)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
	at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128)
	at org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:199)
	at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run

I’m wondering if this could be related to #10? I do have dozens of folders (most of them multibranch pipelines) and jobs and thousands of builds (if not more) which I imagine doesn’t help!

edit: any subsequent GETs to /prometheus result in an empty 200 OK.

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 17

Most upvoted comments

@moertel I’ve not tried this for a while but from what I remember I’ve restarted Jenkins and used curl with huge timeout values directly against the Jenkins http port (avoiding load balancers and proxies to avoid a 504 interrupting the connection). If the connection gets interrupted or you get a stack overflow, every subsequent request to /prometheus will return an empty body with 200 OK.

choppedpork on Jun 20, 2018

@markyjackson-taulia thanks for looking into the issue. Sorry to read you weren’t able to reproduce it. From what I gather, there have to be quite some jobs in Jenkins (for me: 139 jobs and approx. 30 builds each) to cause this resource exhaustion. Having said that, while this makes producing a minimal example rather difficult, the issue is still real.

So are you closing the issue because you categorise it as an edge case not worth fixing? Or because you would want to fix it but there’s not enough information for you to start troubleshooting?

Either way, I’d be happy to assist; let me know what you need from me.

moertel on Nov 19, 2018