che: Workspace start failed!

Describe the bug

We deploy Che on Aliyun K8S,and starts the workspace,java or python all failed!!

Che version

we run docker pull eclipse/che-server:latest Actually che-server run info is 7.7.0, interesting？

Steps to reproduce

Expected behavior

Runtime

• kubernetes (include output of kubectl version) Client Version: version.Info{Major:“1”, Minor:“18”, GitVersion:“v1.18.3”, GitCommit:“2e7996e3e2712684bc73f0dec0200d64eec7fe40”, GitTreeState:“clean”, BuildDate:“2020-05-20T12:52:00Z”, GoVersion:“go1.13.9”, Compiler:“gc”, Platform:“linux/amd64”} Server Version: version.Info{Major:“1”, Minor:“16+”, GitVersion:“v1.16.9-aliyun.1”, GitCommit:“4f7ea78”, GitTreeState:“”, BuildDate:“2020-05-08T07:29:59Z”, GoVersion:“go1.13.9”, Compiler:“gc”, Platform:“linux/amd64”}

• docker version Client: Docker Engine - Community Version: 19.03.5 API version: 1.40 Go version: go1.12.12 Git commit: 633a0ea838 Built: Wed Nov 13 07:22:05 2019 OS/Arch: linux/amd64 Experimental: false

Screenshots

Installation method

helm init && helm dep update helm upgrade --install che --namespace che --set cheImage=eclipse/che-server:latest --set global.cheWorkspacesNamespace=“che” --set global.ingressDomain=172.26.1.208.nip.io ./

Environment

   Aliyun

Eclipse Che Logs

Using embedded assembly.
Listening for transport dt_socket at address: 8000
14-Jul-2020 07:01:10.517 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name:   Apache Tomcat/8.5.46
14-Jul-2020 07:01:10.521 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built:          Sep 16 2019 18:16:19 UTC
14-Jul-2020 07:01:10.521 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 8.5.46.0
14-Jul-2020 07:01:10.521 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name:               Linux
14-Jul-2020 07:01:10.522 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version:            3.10.0-1062.9.1.el7.x86_64
14-Jul-2020 07:01:10.522 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture:          amd64
14-Jul-2020 07:01:10.522 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home:             /usr/local/openjdk-8
14-Jul-2020 07:01:10.522 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version:           1.8.0_232-b09
14-Jul-2020 07:01:10.522 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:            Oracle Corporation
14-Jul-2020 07:01:10.522 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE:         /home/user/eclipse-che/tomcat
14-Jul-2020 07:01:10.522 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME:         /home/user/eclipse-che/tomcat
14-Jul-2020 07:01:10.522 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/home/user/eclipse-che//tomcat/conf/logging.properties
14-Jul-2020 07:01:10.522 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
14-Jul-2020 07:01:10.522 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:MaxRAMFraction=2
14-Jul-2020 07:01:10.522 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:+UseParallelGC
14-Jul-2020 07:01:10.522 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:MinHeapFreeRatio=10
14-Jul-2020 07:01:10.522 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:MaxHeapFreeRatio=20
14-Jul-2020 07:01:10.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:GCTimeRatio=4
14-Jul-2020 07:01:10.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:AdaptiveSizePolicyWeight=90
14-Jul-2020 07:01:10.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:+UnlockExperimentalVMOptions
14-Jul-2020 07:01:10.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:+UseCGroupMemoryLimitForHeap
14-Jul-2020 07:01:10.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dsun.zip.disableMemoryMapping=true
14-Jul-2020 07:01:10.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms20m
14-Jul-2020 07:01:10.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.docker.network=bridge
14-Jul-2020 07:01:10.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dport.http=8080
14-Jul-2020 07:01:10.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.home=/home/user/eclipse-che/
14-Jul-2020 07:01:10.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.logs.dir=/data/logs
14-Jul-2020 07:01:10.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.logs.level=INFO
14-Jul-2020 07:01:10.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djuli-logback.configurationFile=file:/home/user/eclipse-che//tomcat/conf/tomcat-logger.xml
14-Jul-2020 07:01:10.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djdk.tls.ephemeralDHKeySize=2048
14-Jul-2020 07:01:10.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
14-Jul-2020 07:01:10.523 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0022
14-Jul-2020 07:01:10.524 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -agentlib:jdwp=transport=dt_socket,address=8000,server=y,suspend=n
14-Jul-2020 07:01:10.524 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dche.local.conf.dir=/etc/conf
14-Jul-2020 07:01:10.524 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dignore.endorsed.dirs=
14-Jul-2020 07:01:10.524 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/home/user/eclipse-che//tomcat
14-Jul-2020 07:01:10.524 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/home/user/eclipse-che//tomcat
14-Jul-2020 07:01:10.524 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/home/user/eclipse-che//tomcat/temp
14-Jul-2020 07:01:10.604 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]
14-Jul-2020 07:01:10.611 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
14-Jul-2020 07:01:10.624 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 520 ms
14-Jul-2020 07:01:10.632 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
14-Jul-2020 07:01:10.632 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.5.46
14-Jul-2020 07:01:10.651 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/swagger.war]
14-Jul-2020 07:01:10.899 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/swagger.war] has finished in [248] ms
14-Jul-2020 07:01:10.903 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/api.war]
2020-07-14 07:01:14,712[ost-startStop-1]  [INFO ] [.e.c.c.d.JNDIDataSourceFactory 63]   - This=org.eclipse.che.core.db.h2.H2SQLJndiDataSourceFactory@1d6d44fb obj=ResourceRef[className=javax.sql.DataSource,factoryClassLocation=null,factoryClassName=org.apache.naming.factory.ResourceFactory,{type=scope,content=Shareable},{type=auth,content=Container},{type=singleton,content=true},{type=factory,content=org.eclipse.che.api.CommonJndiDataSourceFactory}] name=che Context=org.apache.naming.NamingContext@39cfdee5 environment={}
2020-07-14 07:01:17,658[ost-startStop-1]  [INFO ] [o.f.c.i.d.DbSupportFactory 44]       - Database: jdbc:h2:che (H2 1.4)
2020-07-14 07:01:17,710[ost-startStop-1]  [INFO ] [o.f.c.i.util.VersionPrinter 44]      - Flyway 4.2.0 by Boxfuse
2020-07-14 07:01:17,711[ost-startStop-1]  [INFO ] [o.f.c.i.d.DbSupportFactory 44]       - Database: jdbc:h2:che (H2 1.4)
2020-07-14 07:01:17,740[ost-startStop-1]  [INFO ] [i.f.CustomSqlMigrationResolver 158]  - Searching for sql scripts in locations [classpath:che-schema]
2020-07-14 07:01:17,794[ost-startStop-1]  [INFO ] [o.f.c.i.command.DbValidate 44]       - Successfully validated 55 migrations (execution time 00:00.054s)
2020-07-14 07:01:17,809[ost-startStop-1]  [INFO ] [o.f.c.i.command.DbMigrate 44]        - Current version of schema "PUBLIC": 7.6.0.1
2020-07-14 07:01:17,811[ost-startStop-1]  [INFO ] [o.f.c.i.command.DbMigrate 44]        - Schema "PUBLIC" is up to date. No migration necessary.
2020-07-14 07:01:19,359[ost-startStop-1]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 174]   - Configured factories for environments: '[kubernetes, no-environment]'
2020-07-14 07:01:19,359[ost-startStop-1]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 175]   - Registered infrastructure 'kubernetes'
2020-07-14 07:01:19,405[ost-startStop-1]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 651]   - Infrastructure is tracking 0 active runtimes
2020-07-14 07:01:19,423[ost-startStop-1]  [INFO ] [o.e.c.a.c.u.ApiInfoLogInformer 36]   - Eclipse Che Api Core: Build info '7.7.0' scmRevision '20c990bf70971f956ebe174da2d67f09d6ff9432' implementationVersion '7.7.0'
14-Jul-2020 07:01:19.616 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/api.war] has finished in [8,713] ms
14-Jul-2020 07:01:19.617 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/workspace-loader.war]
14-Jul-2020 07:01:19.734 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/workspace-loader.war] has finished in [116] ms
14-Jul-2020 07:01:19.736 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/dashboard.war]
14-Jul-2020 07:01:19.923 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
14-Jul-2020 07:01:19.930 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/dashboard.war] has finished in [194] ms
14-Jul-2020 07:01:19.931 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/docs.war]
14-Jul-2020 07:01:19.940 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/docs.war] has finished in [9] ms
14-Jul-2020 07:01:19.942 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/home/user/eclipse-che/tomcat/webapps/ROOT.war]
14-Jul-2020 07:01:21.319 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/home/user/eclipse-che/tomcat/webapps/ROOT.war] has finished in [1,377] ms
14-Jul-2020 07:01:21.322 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
14-Jul-2020 07:01:21.330 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 10705 ms
2020-07-14 07:02:10,509[nio-8080-exec-1]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 469]   - Starting workspace 'che/wksp-77xl' with id 'workspaceeq8elhr6jxecraue' by user 'che'
2020-07-14 07:02:18,269[/172.26.0.1/...]  [ERROR] [k.c.d.i.WatchConnectionManager 268]  - Invalid event type
java.lang.IllegalArgumentException: Pod event timestamp can not be blank
        at org.eclipse.che.workspace.infrastructure.kubernetes.util.PodEvents.convertEventTimestampToDate(PodEvents.java:35)
        at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments$4.happenedAfterWatcherInitialization(KubernetesDeployments.java:570)
        at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments$4.eventReceived(KubernetesDeployments.java:530)
        at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments$4.eventReceived(KubernetesDeployments.java:509)
        at io.fabric8.kubernetes.client.utils.WatcherToggle.eventReceived(WatcherToggle.java:49)
        at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$2.onMessage(WatchConnectionManager.java:232)
        at okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:310)
        at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:222)
        at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:101)
        at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:265)
        at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:204)
        at okhttp3.RealCall$AsyncCall.execute(RealCall.java:153)
        at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2020-07-14 07:02:18,280[/172.26.0.1/...]  [ERROR] [k.c.d.i.WatchConnectionManager 268]  - Invalid event type
java.lang.IllegalArgumentException: Pod event timestamp can not be blank
        at org.eclipse.che.workspace.infrastructure.kubernetes.util.PodEvents.convertEventTimestampToDate(PodEvents.java:35)
        at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments$4.happenedAfterWatcherInitialization(KubernetesDeployments.java:570)
        at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments$4.eventReceived(KubernetesDeployments.java:530)
        at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments$4.eventReceived(KubernetesDeployments.java:509)
        at io.fabric8.kubernetes.client.utils.WatcherToggle.eventReceived(WatcherToggle.java:49)
        at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$2.onMessage(WatchConnectionManager.java:232)
        at okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:310)

Additional context

More information

che container Log:

2020-07-14 08:10:16,502[/172.26.0.1/...]  [ERROR] [k.c.d.i.WatchConnectionManager 268]  - Invalid event type
java.lang.IllegalArgumentException: Pod event timestamp can not be blank
	at org.eclipse.che.workspace.infrastructure.kubernetes.util.PodEvents.convertEventTimestampToDate(PodEvents.java:35)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments$4.happenedAfterWatcherInitialization(KubernetesDeployments.java:570)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments$4.eventReceived(KubernetesDeployments.java:530)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments$4.eventReceived(KubernetesDeployments.java:509)
	at io.fabric8.kubernetes.client.utils.WatcherToggle.eventReceived(WatcherToggle.java:49)
	at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$2.onMessage(WatchConnectionManager.java:232)
	at okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:310)
	at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:222)
	at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:101)
	at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:265)
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:204)
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:153)
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2020-07-14 08:10:16,503[/172.26.0.1/...]  [ERROR] [k.c.d.i.WatchConnectionManager 268]  - Invalid event type
java.lang.IllegalArgumentException: Pod event timestamp can not be blank
	at org.eclipse.che.workspace.infrastructure.kubernetes.util.PodEvents.convertEventTimestampToDate(PodEvents.java:35)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments$4.happenedAfterWatcherInitialization(KubernetesDeployments.java:570)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments$4.eventReceived(KubernetesDeployments.java:530)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesDeployments$4.eventReceived(KubernetesDeployments.java:509)
	at io.fabric8.kubernetes.client.utils.WatcherToggle.eventReceived(WatcherToggle.java:49)
	at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$2.onMessage(WatchConnectionManager.java:232)
	at okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:310)
	at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:222)
	at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:101)
	at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:265)
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:204)
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:153)
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2020-07-14 08:13:16,604[aceSharedPool-4]  [WARN ] [.i.k.KubernetesInternalRuntime 252]  - Failed to start Kubernetes runtime of workspace workspaceeq8elhr6jxecraue. Cause: Plugins installation process timed out
2020-07-14 08:13:16,632[aceSharedPool-4]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 897]   - Workspace 'che:wksp-77xl' with id 'workspaceeq8elhr6jxecraue' start failed

workspace page: