gocd: GoCD causing JVM restart due to OOM

Issue Type

Bug Report

Summary

GoCD causes: java.lang.OutOfMemoryError: Required array length 2147483644 + 4 is too large

See logs below for more details.

Environment

👋 Let me know if you need this and I can share this individually, as it might contain sensitive information.

Basic environment details

Go Version: 23.1.0 (16080-54a6971915ff8d402c9fea8cd2ceeb6e31c8cdc8)
JAVA Version: 17.0.6
OS: Linux 5.10.176+

Additional Environment Details

Steps to Reproduce

GoCD running

Expected Results

GoCD not OOM’ing itself with this error

Actual Results

See logs:

Possible Fix

Log snippets

INFO 2023-07-22T14:06:32.474893103Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at java.base/java.lang.Thread.run(Unknown Source)
INFO 2023-07-22T14:06:32.474870801Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
INFO 2023-07-22T14:06:32.474823547Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
INFO 2023-07-22T14:06:32.474802486Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
INFO 2023-07-22T14:06:32.474766980Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
INFO 2023-07-22T14:06:32.474722910Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
INFO 2023-07-22T14:06:32.474713196Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
INFO 2023-07-22T14:06:32.474702509Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
INFO 2023-07-22T14:06:32.474688272Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
INFO 2023-07-22T14:06:32.473940948Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
INFO 2023-07-22T14:06:32.473899125Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
INFO 2023-07-22T14:06:32.473860457Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
INFO 2023-07-22T14:06:32.473816795Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
INFO 2023-07-22T14:06:32.473809708Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
INFO 2023-07-22T14:06:32.473728424Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
INFO 2023-07-22T14:06:32.473686343Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.server.Server.handle(Server.java:516)
INFO 2023-07-22T14:06:32.473632621Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
INFO 2023-07-22T14:06:32.473615782Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
INFO 2023-07-22T14:06:32.473556542Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)
INFO 2023-07-22T14:06:32.473519362Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
INFO 2023-07-22T14:06:32.473481990Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
INFO 2023-07-22T14:06:32.473437657Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
INFO 2023-07-22T14:06:32.473401757Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
INFO 2023-07-22T14:06:32.473351218Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
INFO 2023-07-22T14:06:32.473304601Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
INFO 2023-07-22T14:06:32.473284950Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
INFO 2023-07-22T14:06:32.473228590Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
INFO 2023-07-22T14:06:32.473205257Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
INFO 2023-07-22T14:06:32.473135445Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
INFO 2023-07-22T14:06:32.473089525Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
INFO 2023-07-22T14:06:32.473065754Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
INFO 2023-07-22T14:06:32.473027405Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
INFO 2023-07-22T14:06:32.472995330Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
INFO 2023-07-22T14:06:32.472980170Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)
INFO 2023-07-22T14:06:32.472942550Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
INFO 2023-07-22T14:06:32.472928788Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
INFO 2023-07-22T14:06:32.472868125Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at com.thoughtworks.go.server.web.DefaultHeadersFilter.doFilter(DefaultHeadersFilter.java:51)
INFO 2023-07-22T14:06:32.472835997Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
INFO 2023-07-22T14:06:32.472819426Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
INFO 2023-07-22T14:06:32.472800736Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)
INFO 2023-07-22T14:06:32.472784523Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)
INFO 2023-07-22T14:06:32.472767561Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at com.thoughtworks.go.server.newsecurity.filterchains.MainFilterChain.doFilter(MainFilterChain.java:79)
INFO 2023-07-22T14:06:32.471981516Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
INFO 2023-07-22T14:06:32.471928980Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
INFO 2023-07-22T14:06:32.471883189Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
INFO 2023-07-22T14:06:32.471838516Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
INFO 2023-07-22T14:06:32.471800516Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at com.thoughtworks.go.server.web.BackupFilter.doFilterInternal(BackupFilter.java:79)
INFO 2023-07-22T14:06:32.471748309Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
INFO 2023-07-22T14:06:32.471706576Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at com.thoughtworks.go.server.newsecurity.filters.ModeAwareFilter.doFilter(ModeAwareFilter.java:76)
INFO 2023-07-22T14:06:32.471667268Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
INFO 2023-07-22T14:06:32.471619772Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185)
INFO 2023-07-22T14:06:32.471573087Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
INFO 2023-07-22T14:06:32.471542697Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
INFO 2023-07-22T14:06:32.471493225Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
INFO 2023-07-22T14:06:32.471455894Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at com.thoughtworks.go.server.newsecurity.filters.AlwaysCreateSessionFilter.doFilterInternal(AlwaysCreateSessionFilter.java:40)
INFO 2023-07-22T14:06:32.471422982Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
INFO 2023-07-22T14:06:32.471374664Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
INFO 2023-07-22T14:06:32.471363275Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185)
INFO 2023-07-22T14:06:32.471300832Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:208)
INFO 2023-07-22T14:06:32.471266158Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
INFO 2023-07-22T14:06:32.471243583Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185)
INFO 2023-07-22T14:06:32.471226648Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
INFO 2023-07-22T14:06:32.471154813Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
INFO 2023-07-22T14:06:32.471137077Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
INFO 2023-07-22T14:06:32.471124826Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at com.thoughtworks.go.server.newsecurity.filters.AgentAuthenticationFilter.doFilterInternal(AgentAuthenticationFilter.java:65)
INFO 2023-07-22T14:06:32.471058762Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at com.thoughtworks.go.server.newsecurity.filters.AgentAuthenticationFilter.tokenBasedFilter(AgentAuthenticationFilter.java:103)
INFO 2023-07-22T14:06:32.471048307Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
INFO 2023-07-22T14:06:32.471037732Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
INFO 2023-07-22T14:06:32.471017133Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
INFO 2023-07-22T14:06:32.471004679Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at com.thoughtworks.go.server.newsecurity.filters.ThreadLocalUserFilter.doFilterInternal(ThreadLocalUserFilter.java:41)
INFO 2023-07-22T14:06:32.470200972Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
INFO 2023-07-22T14:06:32.470183429Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185)
INFO 2023-07-22T14:06:32.470103505Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
INFO 2023-07-22T14:06:32.470055897Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
INFO 2023-07-22T14:06:32.469998461Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
INFO 2023-07-22T14:06:32.469973129Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at com.thoughtworks.go.server.newsecurity.filters.AbstractUserEnabledCheckFilter.doFilterInternal(AbstractUserEnabledCheckFilter.java:66)
INFO 2023-07-22T14:06:32.469909897Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
INFO 2023-07-22T14:06:32.469883269Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
INFO 2023-07-22T14:06:32.469862980Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185)
INFO 2023-07-22T14:06:32.469811549Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
INFO 2023-07-22T14:06:32.469758351Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
INFO 2023-07-22T14:06:32.469751131Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
INFO 2023-07-22T14:06:32.469679366Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at com.thoughtworks.go.server.newsecurity.filters.VerifyAuthorityFilter.doFilterInternal(VerifyAuthorityFilter.java:57)
INFO 2023-07-22T14:06:32.469632549Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
INFO 2023-07-22T14:06:32.469589387Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
INFO 2023-07-22T14:06:32.469557239Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185)
INFO 2023-07-22T14:06:32.469518936Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
INFO 2023-07-22T14:06:32.469507301Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
INFO 2023-07-22T14:06:32.469488578Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
INFO 2023-07-22T14:06:32.469405732Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at com.thoughtworks.go.server.newsecurity.filters.DenyIfRefererIsNotFilesFilter.doFilterInternal(DenyIfRefererIsNotFilesFilter.java:52)
INFO 2023-07-22T14:06:32.469371344Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
INFO 2023-07-22T14:06:32.469340399Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
INFO 2023-07-22T14:06:32.469304522Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185)
INFO 2023-07-22T14:06:32.469340399Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
INFO 2023-07-22T14:06:32.469304522Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185)
INFO 2023-07-22T14:06:32.469286653Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:208)
INFO 2023-07-22T14:06:32.469257592Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
INFO 2023-07-22T14:06:32.469246205Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
INFO 2023-07-22T14:06:32.469225276Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at com.thoughtworks.go.server.web.FlashLoadingFilter.doFilterInternal(FlashLoadingFilter.java:38)
INFO 2023-07-22T14:06:32.468479470Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
INFO 2023-07-22T14:06:32.468440627Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
INFO 2023-07-22T14:06:32.468430954Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
INFO 2023-07-22T14:06:32.468332642Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:381)
INFO 2023-07-22T14:06:32.468290117Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92)
INFO 2023-07-22T14:06:32.468272333Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145)
INFO 2023-07-22T14:06:32.468209449Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:171)
INFO 2023-07-22T14:06:32.468198950Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.tuckey.web.filters.urlrewrite.NormalRewrittenUrl.doRewrite(NormalRewrittenUrl.java:213)
INFO 2023-07-22T14:06:32.468081863Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:81)
INFO 2023-07-22T14:06:32.468043935Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:229)
INFO 2023-07-22T14:06:32.468028422Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
INFO 2023-07-22T14:06:32.468015988Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
INFO 2023-07-22T14:06:32.467945822Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
INFO 2023-07-22T14:06:32.467896665Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
INFO 2023-07-22T14:06:32.467856077Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
INFO 2023-07-22T14:06:32.467842253Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
INFO 2023-07-22T14:06:32.467777017Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
INFO 2023-07-22T14:06:32.467736725Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
INFO 2023-07-22T14:06:32.467716591Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
INFO 2023-07-22T14:06:32.467657598Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
INFO 2023-07-22T14:06:32.467575563Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
INFO 2023-07-22T14:06:32.467563961Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:618)
INFO 2023-07-22T14:06:32.467544539Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:626)
INFO 2023-07-22T14:06:32.467495609Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
INFO 2023-07-22T14:06:32.467473533Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)
INFO 2023-07-22T14:06:32.467444311Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
INFO 2023-07-22T14:06:32.467413537Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
INFO 2023-07-22T14:06:32.467368495Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at com.thoughtworks.go.spark.SparkPreFilter.doFilter(SparkPreFilter.java:63)
INFO 2023-07-22T14:06:32.467334641Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at spark.servlet.SparkFilter.doFilter(SparkFilter.java:173)
INFO 2023-07-22T14:06:32.467317505Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at spark.http.matching.MatcherFilter.doFilter(MatcherFilter.java:130)
INFO 2023-07-22T14:06:32.467305039Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at spark.http.matching.Routes.execute(Routes.java:61)
INFO 2023-07-22T14:06:32.467292523Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at spark.RouteImpl$1.handle(RouteImpl.java:72)
INFO 2023-07-22T14:06:32.467280756Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at spark.RouteImpl$1.handle(RouteImpl.java:72)
INFO 2023-07-22T14:06:32.467261360Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at com.thoughtworks.go.apiv1.internalagent.InternalAgentControllerV1.getWork(InternalAgentControllerV1.java:133)
INFO 2023-07-22T14:06:32.466511408Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at com.thoughtworks.go.apiv1.internalagent.representers.WorkRepresenter.toJSON(WorkRepresenter.java:27)
INFO 2023-07-22T14:06:32.466459976Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at com.google.gson.Gson.toJson(Gson.java:759)
INFO 2023-07-22T14:06:32.466424148Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at com.google.gson.Gson.toJson(Gson.java:812)
INFO 2023-07-22T14:06:32.466381834Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at com.google.gson.Gson.toJson(Gson.java:842)
INFO 2023-07-22T14:06:32.466337374Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at com.google.gson.TypeAdapter$1.write(TypeAdapter.java:196)
INFO 2023-07-22T14:06:32.466301694Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at com.thoughtworks.go.remote.adapter.RuntimeTypeAdapterFactory$1.write(RuntimeTypeAdapterFactory.java:288)
INFO 2023-07-22T14:06:32.466247612Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at com.google.gson.internal.Streams.write(Streams.java:73)
INFO 2023-07-22T14:06:32.466199103Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at com.google.gson.internal.bind.TypeAdapters$28.write(TypeAdapters.java:725)
INFO 2023-07-22T14:06:32.466157783Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at com.google.gson.internal.bind.TypeAdapters$28.write(TypeAdapters.java:846)
INFO 2023-07-22T14:06:32.466116116Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at com.google.gson.internal.bind.TypeAdapters$28.write(TypeAdapters.java:846)
INFO 2023-07-22T14:06:32.466067792Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at com.google.gson.internal.bind.TypeAdapters$28.write(TypeAdapters.java:846)
INFO 2023-07-22T14:06:32.466020703Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at com.google.gson.internal.bind.TypeAdapters$28.write(TypeAdapters.java:838)
INFO 2023-07-22T14:06:32.465995145Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at com.google.gson.internal.bind.TypeAdapters$28.write(TypeAdapters.java:846)
INFO 2023-07-22T14:06:32.465947202Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at com.google.gson.internal.bind.TypeAdapters$28.write(TypeAdapters.java:838)
INFO 2023-07-22T14:06:32.465923984Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at com.google.gson.internal.bind.TypeAdapters$28.write(TypeAdapters.java:846)
INFO 2023-07-22T14:06:32.465860946Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at com.google.gson.internal.bind.TypeAdapters$28.write(TypeAdapters.java:846)
INFO 2023-07-22T14:06:32.465845234Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at com.google.gson.internal.bind.TypeAdapters$28.write(TypeAdapters.java:832)
INFO 2023-07-22T14:06:32.465835478Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at com.google.gson.stream.JsonWriter.value(JsonWriter.java:419)
INFO 2023-07-22T14:06:32.465785010Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at com.google.gson.stream.JsonWriter.string(JsonWriter.java:638)
INFO 2023-07-22T14:06:32.465756421Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at java.base/java.io.StringWriter.write(Unknown Source)
INFO 2023-07-22T14:06:32.465748513Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at java.base/java.lang.StringBuffer.append(Unknown Source)
INFO 2023-07-22T14:06:32.465708254Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at java.base/java.lang.AbstractStringBuilder.append(Unknown Source)
INFO 2023-07-22T14:06:32.465665449Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at java.base/java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown Source)
INFO 2023-07-22T14:06:32.465659016Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at java.base/java.lang.AbstractStringBuilder.newCapacity(Unknown Source)
INFO 2023-07-22T14:06:32.465640968Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at java.base/jdk.internal.util.ArraysSupport.newLength(Unknown Source)
INFO 2023-07-22T14:06:32.465634401Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | at java.base/jdk.internal.util.ArraysSupport.hugeLength(Unknown Source)
INFO 2023-07-22T14:06:32.465611828Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] wrapper | The JVM has run out of memory. Restarting JVM.
INFO 2023-07-22T14:06:32.465589725Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | java.lang.OutOfMemoryError: Required array length 2147483644 + 4 is too large
INFO 2023-07-22T14:06:32.465550517Z [resource.labels.podName: gocd-server-857699cf6b-58rxt] jvm 1 | 2023-07-22 14:06:32,376 WARN [qtp445410010-33] HttpChannel:776 - /go/remoting/api/agent/get_work

Code snippets/Screenshots

Any other info

About this issue

Original URL
State: open
Created 10 months ago
Comments: 20 (8 by maintainers)

Most upvoted comments

I can also add that we have enabled creation of a heap dump when we get an OOM the next time. Hopefully we can find more detailed information there.

peterrosell on Aug 18, 2023

In addition to the above, the part of your server environment that includes the list of plugins and versions would be useful. Since artifact, package, task, secret plugins can affect serialized information as part of the “build work” for an agent to perform, there could be some aspect there. That may only “rule out” a particular possible cause rather than identify the issue, without knowing what was special about the job that was trying to run though.

This is the part from /go/api/support with the plugin details:

{
  "Plugin information": {
    "plugins": [
      {
        "id": "cd.go.authentication.passwordfile",
        "type": [
          "authorization"
        ],
        "version": "2.1.2-180",
        "bundled_plugin": true,
        "status": {
          "state": "ACTIVE",
          "messages": [],
          "rootCauseIfInvalid": null
        }
      },
      {
        "id": "github.pr",
        "type": [
          "scm"
        ],
        "version": "1.4.0-RC2",
        "bundled_plugin": false,
        "status": {
          "state": "ACTIVE",
          "messages": [],
          "rootCauseIfInvalid": null
        }
      },
      {
        "id": "maven-repo",
        "type": [
          "package-repository"
        ],
        "version": "1.2.1",
        "bundled_plugin": false,
        "status": {
          "state": "ACTIVE",
          "messages": [],
          "rootCauseIfInvalid": null
        }
      },
      {
        "id": "docker-registry",
        "type": [
          "package-repository"
        ],
        "version": "1.1.0",
        "bundled_plugin": false,
        "status": {
          "state": "ACTIVE",
          "messages": [],
          "rootCauseIfInvalid": null
        }
      },
      {
        "id": "cd.go.secrets.file-based-plugin",
        "type": [
          "secrets"
        ],
        "version": "1.1.1-146",
        "bundled_plugin": true,
        "status": {
          "state": "ACTIVE",
          "messages": [],
          "rootCauseIfInvalid": null
        }
      },
      {
        "id": "github.pr.status",
        "type": [
          "notification"
        ],
        "version": "1.6-73",
        "bundled_plugin": false,
        "status": {
          "state": "ACTIVE",
          "messages": [],
          "rootCauseIfInvalid": null
        }
      },
      {
        "id": "cd.go.authentication.ldap",
        "type": [
          "authorization"
        ],
        "version": "2.2.1-211",
        "bundled_plugin": true,
        "status": {
          "state": "ACTIVE",
          "messages": [],
          "rootCauseIfInvalid": null
        }
      },
      {
        "id": "yaml.config.plugin",
        "type": [
          "configrepo"
        ],
        "version": "0.14.1",
        "bundled_plugin": true,
        "status": {
          "state": "ACTIVE",
          "messages": [],
          "rootCauseIfInvalid": null
        }
      },
      {
        "id": "json.config.plugin",
        "type": [
          "configrepo"
        ],
        "version": "0.6.0",
        "bundled_plugin": true,
        "status": {
          "state": "ACTIVE",
          "messages": [],
          "rootCauseIfInvalid": null
        }
      }
    ]
  }
}

MPV on Aug 18, 2023