che: Experiencing error when creating/starting a workspace.

Description

I had just installed Eclipe CHE on my Ubuntu Server, and was creating my first workspace, after a while I got an error mentioning that the workspace wasn’t running, and when I clicked the Start button I got the error mention below.

Reproduction Steps

On Web GUI I get a message saying “Working is not running” and when clicking Start I get the error: Could not start workspace gCyborg. Reason: Start of environment 'default' failed. Error: Error occurs while initializing command [/bin/sh, -c, trap '[ -z "$(jobs -p)" ] || kill $(jobs -p); [ -e /tmp/docker-exec-34.pid ] && rm /tmp/docker-exec-34.pid' EXIT; echo $$>/tmp/docker-exec-34.pid; command -v pidof >/dev/null 2>&1 && { pidof che-exec-agent >/dev/null 2>&1 && echo 0 || echo 1 } || { ps -fC che-exec-agent >/dev/null 2>&1 && echo 0 || echo 1 }] in docker container ebbdff02acb6214e845b9902f98799313aeee5b039531f60798910e1c3931ab4: Error response from docker API, status: 409, message: Container ebbdff02acb6214e845b9902f98799313aeee5b039531f60798910e1c3931ab4 is not running

OS and version:
Ubuntu 16.04.3 LTS

Docker Version:

Client:
 Version:       18.01.0-ce
 API version:   1.35
 Go version:    go1.9.2
 Git commit:    03596f5
 Built: Wed Jan 10 20:11:05 2018
 OS/Arch:       linux/amd64
 Experimental:  false
 Orchestrator:  swarm

Server:
 Engine:
  Version:      18.01.0-ce
  API version:  1.35 (minimum version 1.12)
  Go version:   go1.9.2
  Git commit:   03596f5
  Built:        Wed Jan 10 20:09:37 2018
  OS/Arch:      linux/amd64
  Experimental: false

Diagnostics:

---------------  S Y S T E M  ---------------

OS:NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.6.2
PRETTY_NAME="Alpine Linux v3.6"
HOME_URL="http://alpinelinux.org"
BUG_REPORT_URL="http://bugs.alpinelinux.org"

uname:Linux 4.4.0-109-generic #132-Ubuntu SMP Tue Jan 9 19:52:39 UTC 2018 x86_64
libc:glibc 2.9 NPTL
rlimit: STACK 8192k, CORE infinity, NPROC infinity, NOFILE 1048576, AS infinity
load average:0.02 0.09 0.05

/proc/meminfo:
MemTotal:        1014212 kB
MemFree:          105164 kB
MemAvailable:     356048 kB
Buffers:           33924 kB
Cached:           336484 kB
SwapCached:            0 kB
Active:           610400 kB
Inactive:         165428 kB
Active(anon):     411724 kB
Inactive(anon):    15016 kB
Active(file):     198676 kB
Inactive(file):   150412 kB
Unevictable:        3652 kB
Mlocked:            3652 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:             40704 kB
Writeback:          5148 kB
AnonPages:        409056 kB
Mapped:            68660 kB
Shmem:             18872 kB
Slab:              82728 kB
SReclaimable:      58844 kB
SUnreclaim:        23884 kB
KernelStack:        5008 kB
PageTables:        12408 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      507104 kB
Committed_AS:    1784828 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:    260096 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       86016 kB
DirectMap2M:      962560 kB


CPU:total 1 (initial active 1) (1 cores per cpu, 1 threads per core) family 6 model 63 stepping 2, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, lzcnt, tsc, bmi1, bmi2

/proc/cpuinfo:
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 63
model name      : Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz
stepping        : 2
microcode       : 0x3b
cpu MHz         : 2400.072
cache size      : 30720 KB
physical id     : 0
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single kaiser fsgsbase bmi1 avx2 smep bmi2 erms invpcid xsaveopt
bugs            :
bogomips        : 4800.14
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:



Memory: 4k page, physical 1014212k(105164k free), swap 0k(0k free)

vm_info: OpenJDK 64-Bit Server VM (25.131-b11) for linux-amd64 JRE (1.8.0_131-b11), built on Jun 16 2017 13:51:29 by "buildozer" with gcc 6.3.0

time: Mon Jan 29 20:23:40 2018
elapsed time: 0 seconds (0d 0h 0m 0s)

Using embedded assembly...
Found custom che.properties...
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000085330000, 449642496, 0) failed; error='Out of memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 449642496 bytes for committing reserved memory.
# Can not save log file, dump to screen..
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 449642496 bytes for committing reserved memory.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (os_linux.cpp:2658), pid=38, tid=0x00007fb9d53dfae8
#
# JRE version:  (8.0_131-b11) (build )
# Java VM: OpenJDK 64-Bit Server VM (25.131-b11 mixed mode linux-amd64 compressed oops)
# Derivative: IcedTea 3.4.0
# Distribution: Custom build (Fri Jun 16 13:41:54 GMT 2017)
# Core dump written. Default location: //core or core.38
#

---------------  T H R E A D  ---------------

Current thread (0x0000563914b51000):  JavaThread "Unknown thread" [_thread_in_vm, id=42, stack(0x00007fb9d52df000,0x00007fb9d53dfaa8)]

Stack: [0x00007fb9d52df000,0x00007fb9d53dfaa8]

---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )

Other Threads:

=>0x0000563914b51000 (exited) JavaThread "Unknown thread" [_thread_in_vm, id=42, stack(0x00007fb9d52df000,0x00007fb9d53dfaa8)]

VM state:not at safepoint (not fully initialized)

VM Mutex/Monitor currently owned by a thread: None

GC Heap History (0 events):
No events

Deoptimization events (0 events):
No events

Internal exceptions (0 events):
No events

Events (0 events):
No events


Dynamic libraries:
80000000-85330000 rw-p 00000000 00:00 0
a0000000-100000000 ---p 00000000 00:00 0
5639142e7000-5639142e8000 r-xp 00000000 00:2b 82                         /usr/lib/jvm/java-1.8-openjdk/jre/bin/java
5639144e7000-5639144e8000 r--p 00000000 00:2b 82                         /usr/lib/jvm/java-1.8-openjdk/jre/bin/java
5639144e8000-5639144e9000 rw-p 00001000 00:2b 82                         /usr/lib/jvm/java-1.8-openjdk/jre/bin/java
563914b39000-563914b68000 rw-p 00000000 00:00 0                          [heap]
7fb9c3cc0000-7fb9c3deb000 rw-p 00000000 00:00 0
7fb9c3deb000-7fb9c41c1000 ---p 00000000 00:00 0
7fb9c41c1000-7fb9c41c2000 rw-p 00000000 00:00 0
7fb9c41c2000-7fb9c41c3000 ---p 00000000 00:00 0
7fb9c41c3000-7fb9c42ce000 rw-p 00000000 00:00 0
7fb9c42ce000-7fb9c4684000 ---p 00000000 00:00 0
7fb9c4684000-7fb9c48f4000 rwxp 00000000 00:00 0
7fb9c48f4000-7fb9d3684000 ---p 00000000 00:00 0
7fb9d3684000-7fb9d388b000 r-xp 00000000 00:2b 103                        /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/libzip.so
7fb9d388b000-7fb9d388c000 r--p 00007000 00:2b 103                        /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/libzip.so
7fb9d388c000-7fb9d388d000 rw-p 00008000 00:2b 103                        /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/libzip.so
7fb9d388d000-7fb9d3ab9000 r-xp 00000000 00:2b 89                         /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/libjava.so
7fb9d3ab9000-7fb9d3aba000 r--p 0002c000 00:2b 89                         /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/libjava.so
7fb9d3aba000-7fb9d3abb000 rw-p 0002d000 00:2b 89                         /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/libjava.so
7fb9d3abb000-7fb9d3abc000 rw-p 00000000 00:00 0
7fb9d3abc000-7fb9d3cc8000 r-xp 00000000 00:2b 98                         /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/libverify.so
7fb9d3cc8000-7fb9d3cca000 r--p 0000c000 00:2b 98                         /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/libverify.so
7fb9d3cca000-7fb9d3ccb000 rw-p 0000e000 00:2b 98                         /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/libverify.so
7fb9d3ccb000-7fb9d3edb000 r-xp 00000000 00:2b 97                         /usr/lib/libgcc_s.so.1
7fb9d3edb000-7fb9d3edc000 r--p 00010000 00:2b 97                         /usr/lib/libgcc_s.so.1
7fb9d3edc000-7fb9d3edd000 rw-p 00011000 00:2b 97                         /usr/lib/libgcc_s.so.1
7fb9d3edd000-7fb9d421d000 r-xp 00000000 00:2b 96                         /usr/lib/libstdc++.so.6.0.22
7fb9d421d000-7fb9d422b000 r--p 00140000 00:2b 96                         /usr/lib/libstdc++.so.6.0.22
7fb9d422b000-7fb9d422c000 rw-p 0014e000 00:2b 96                         /usr/lib/libstdc++.so.6.0.22
7fb9d422c000-7fb9d422f000 rw-p 00000000 00:00 0
7fb9d422f000-7fb9d4c46000 r-xp 00000000 00:2b 92                         /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server/libjvm.so
7fb9d4c46000-7fb9d4cdb000 r--p 00817000 00:2b 92                         /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server/libjvm.so
7fb9d4cdb000-7fb9d4d03000 rw-p 008ac000 00:2b 92                         /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server/libjvm.so
7fb9d4d03000-7fb9d4d33000 rw-p 00000000 00:00 0
7fb9d4d33000-7fb9d4f48000 r-xp 00000000 00:2b 88                         /lib/libz.so.1.2.11
7fb9d4f48000-7fb9d4f49000 r--p 00015000 00:2b 88                         /lib/libz.so.1.2.11
7fb9d4f49000-7fb9d4f4a000 rw-p 00016000 00:2b 88                         /lib/libz.so.1.2.11
7fb9d4f4a000-7fb9d5156000 r-xp 00000000 00:2b 86                         /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/jli/libjli.so
7fb9d5156000-7fb9d5157000 r--p 0000c000 00:2b 86                         /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/jli/libjli.so
7fb9d5157000-7fb9d5158000 rw-p 0000d000 00:2b 86                         /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/jli/libjli.so
7fb9d5158000-7fb9d51e1000 r-xp 00000000 00:2b 30                         /lib/ld-musl-x86_64.so.1
7fb9d52d3000-7fb9d52db000 rw-s 00000000 00:2b 102                        /tmp/hsperfdata_root/38
7fb9d52db000-7fb9d52dc000 rw-p 00000000 00:00 0
7fb9d52dc000-7fb9d52dd000 r--p 00000000 00:00 0
7fb9d52dd000-7fb9d52df000 ---p 00000000 00:00 0
7fb9d52df000-7fb9d52e2000 ---p 00000000 00:00 0
7fb9d52e2000-7fb9d53e0000 rw-p 00000000 00:00 0
7fb9d53e0000-7fb9d53e1000 r--p 00088000 00:2b 30                         /lib/ld-musl-x86_64.so.1
7fb9d53e1000-7fb9d53e2000 rw-p 00089000 00:2b 30                         /lib/ld-musl-x86_64.so.1
7fb9d53e2000-7fb9d53e5000 rw-p 00000000 00:00 0
7ffe1560d000-7ffe1562f000 rw-p 00000000 00:00 0                          [stack]
7ffe15674000-7ffe15677000 r--p 00000000 00:00 0                          [vvar]
7ffe15677000-7ffe15679000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

VM Arguments:
jvm_args: -Djava.util.logging.config.file=/home/user/eclipse-che//tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms512m -Xmx2048m -Djava.security.egd=file:/dev/./urandom -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSInitiatingOccupancyOnly -XX:+ScavengeBeforeFullGC -XX:+CMSScavengeBeforeRemark -Dche.local.conf.dir=/conf -Dche.docker.network=bridge -Dport.http=6868 -Dche.home=/home/user/eclipse-che/ -Dche.logs.dir=/logs -Dche.logs.level=INFO -Djuli-logback.configurationFile=file:/home/user/eclipse-che//tomcat/conf/tomcat-logger.xml -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dche.local.conf.dir=/conf -Dcatalina.base=/home/user/eclipse-che//tomcat -Dcatalina.home=/home/user/eclipse-che//tomcat -Djava.io.tmpdir=/home/user/eclipse-che//tomcat/temp
java_command: org.apache.catalina.startup.Bootstrap start
java_class_path (initial): /home/user/eclipse-che//tomcat/conf/:/usr/lib/jvm/java-1.8-openjdk/jre/lib/tools.jar:/home/user/eclipse-che//tomcat/bin/bootstrap.jar:/home/user/eclipse-che//tomcat/bin/tomcat-juli.jar
Launcher Type: SUN_STANDARD

Environment Variables:
JAVA_HOME=/usr/lib/jvm/java-1.8-openjdk/jre
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/jvm/java-1.8-openjdk/jre/bin:/usr/lib/jvm/java-1.8-openjdk/bin
LD_LIBRARY_PATH=/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64:/usr/lib/jvm/java-1.8-openjdk/jre/../lib/amd64

Signal Handlers:
SIGSEGV: [libjvm.so+0x63094c], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGBUS: [libjvm.so+0x63094c], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGFPE: [libjvm.so+0x55ffac], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGPIPE: [libjvm.so+0x55ffac], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGXFSZ: [libjvm.so+0x55ffac], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGILL: [libjvm.so+0x55ffac], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGUSR1: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGUSR2: [libjvm.so+0x56014b], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
SIGHUP: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGINT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGTERM: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGQUIT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none


---------------  S Y S T E M  ---------------

OS:NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.6.2
PRETTY_NAME="Alpine Linux v3.6"
HOME_URL="http://alpinelinux.org"
BUG_REPORT_URL="http://bugs.alpinelinux.org"

uname:Linux 4.4.0-109-generic #132-Ubuntu SMP Tue Jan 9 19:52:39 UTC 2018 x86_64
libc:glibc 2.9 NPTL
rlimit: STACK 8192k, CORE infinity, NPROC infinity, NOFILE 1048576, AS infinity
load average:0.00 0.07 0.05

/proc/meminfo:
MemTotal:        1014212 kB
MemFree:          104672 kB
MemAvailable:     355896 kB
Buffers:           33992 kB
Cached:           336776 kB
SwapCached:            0 kB
Active:           611876 kB
Inactive:         164248 kB
Active(anon):     411680 kB
Inactive(anon):    15016 kB
Active(file):     200196 kB
Inactive(file):   149232 kB
Unevictable:        3652 kB
Mlocked:            3652 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:             40676 kB
Writeback:          5132 kB
AnonPages:        409020 kB
Mapped:            68468 kB
Shmem:             18872 kB
Slab:              82732 kB
SReclaimable:      58844 kB
SUnreclaim:        23888 kB
KernelStack:        4992 kB
PageTables:        12408 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      507104 kB
Committed_AS:    1783420 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:    268288 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       86016 kB
DirectMap2M:      962560 kB


CPU:total 1 (initial active 1) (1 cores per cpu, 1 threads per core) family 6 model 63 stepping 2, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, lzcnt, tsc, bmi1, bmi2

/proc/cpuinfo:
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 63
model name      : Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz
stepping        : 2
microcode       : 0x3b
cpu MHz         : 2400.072
cache size      : 30720 KB
physical id     : 0
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single kaiser fsgsbase bmi1 avx2 smep bmi2 erms invpcid xsaveopt
bugs            :
bogomips        : 4800.14
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:



Memory: 4k page, physical 1014212k(104672k free), swap 0k(0k free)

vm_info: OpenJDK 64-Bit Server VM (25.131-b11) for linux-amd64 JRE (1.8.0_131-b11), built on Jun 16 2017 13:51:29 by "buildozer" with gcc 6.3.0

time: Mon Jan 29 20:24:40 2018
elapsed time: 0 seconds (0d 0h 0m 0s)

Using embedded assembly...
Found custom che.properties...
2018-01-29 20:26:20,592[main]             [INFO ] [o.a.c.s.VersionLoggerListener 89]    - Server version:        Apache Tomcat/8.5.11
2018-01-29 20:26:20,599[main]             [INFO ] [o.a.c.s.VersionLoggerListener 91]    - Server built:          Jan 10 2017 21:02:52 UTC
2018-01-29 20:26:20,600[main]             [INFO ] [o.a.c.s.VersionLoggerListener 93]    - Server number:         8.5.11.0
2018-01-29 20:26:20,600[main]             [INFO ] [o.a.c.s.VersionLoggerListener 95]    - OS Name:               Linux
2018-01-29 20:26:20,601[main]             [INFO ] [o.a.c.s.VersionLoggerListener 97]    - OS Version:            4.4.0-112-generic
2018-01-29 20:26:20,602[main]             [INFO ] [o.a.c.s.VersionLoggerListener 99]    - Architecture:          amd64
2018-01-29 20:26:20,604[main]             [INFO ] [o.a.c.s.VersionLoggerListener 101]   - Java Home:             /usr/lib/jvm/java-1.8-openjdk/jre
2018-01-29 20:26:20,605[main]             [INFO ] [o.a.c.s.VersionLoggerListener 103]   - JVM Version:           1.8.0_131-b11
2018-01-29 20:26:20,605[main]             [INFO ] [o.a.c.s.VersionLoggerListener 105]   - JVM Vendor:            Oracle Corporation
2018-01-29 20:26:20,606[main]             [INFO ] [o.a.c.s.VersionLoggerListener 107]   - CATALINA_BASE:         /home/user/eclipse-che/tomcat
2018-01-29 20:26:20,609[main]             [INFO ] [o.a.c.s.VersionLoggerListener 109]   - CATALINA_HOME:         /home/user/eclipse-che/tomcat
2018-01-29 20:26:20,614[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Djava.util.logging.config.file=/home/user/eclipse-che//tomcat/conf/logging.properties
2018-01-29 20:26:20,614[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
2018-01-29 20:26:20,615[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Xms512m
2018-01-29 20:26:20,616[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Xmx2048m
2018-01-29 20:26:20,617[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Djava.security.egd=file:/dev/./urandom
2018-01-29 20:26:20,617[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -XX:+HeapDumpOnOutOfMemoryError
2018-01-29 20:26:20,617[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -XX:+UseConcMarkSweepGC
2018-01-29 20:26:20,620[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -XX:+CMSParallelRemarkEnabled
2018-01-29 20:26:20,620[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -XX:+UseCMSInitiatingOccupancyOnly
2018-01-29 20:26:20,621[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -XX:+ScavengeBeforeFullGC
2018-01-29 20:26:20,621[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -XX:+CMSScavengeBeforeRemark
2018-01-29 20:26:20,621[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dche.local.conf.dir=/conf
2018-01-29 20:26:20,622[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dche.docker.network=bridge
2018-01-29 20:26:20,623[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dport.http=6868
2018-01-29 20:26:20,624[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dche.home=/home/user/eclipse-che/
2018-01-29 20:26:20,628[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dche.logs.dir=/logs
2018-01-29 20:26:20,630[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dche.logs.level=INFO
2018-01-29 20:26:20,630[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Djuli-logback.configurationFile=file:/home/user/eclipse-che//tomcat/conf/tomcat-logger.xml
2018-01-29 20:26:20,630[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Djdk.tls.ephemeralDHKeySize=2048
2018-01-29 20:26:20,631[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
2018-01-29 20:26:20,631[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dcom.sun.management.jmxremote
2018-01-29 20:26:20,632[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dcom.sun.management.jmxremote.ssl=false
2018-01-29 20:26:20,633[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dcom.sun.management.jmxremote.authenticate=false
2018-01-29 20:26:20,634[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dche.local.conf.dir=/conf
2018-01-29 20:26:20,634[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dcatalina.base=/home/user/eclipse-che//tomcat
2018-01-29 20:26:20,634[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Dcatalina.home=/home/user/eclipse-che//tomcat
2018-01-29 20:26:20,635[main]             [INFO ] [o.a.c.s.VersionLoggerListener 115]   - Command line argument: -Djava.io.tmpdir=/home/user/eclipse-che//tomcat/temp
2018-01-29 20:26:20,841[main]             [INFO ] [o.a.c.http11.Http11NioProtocol 525]  - Initializing ProtocolHandler ["http-nio-6868"]
2018-01-29 20:26:20,880[main]             [INFO ] [o.a.t.util.net.NioSelectorPool 67]   - Using a shared selector for servlet write/read
2018-01-29 20:26:20,894[main]             [INFO ] [o.a.catalina.startup.Catalina 617]   - Initialization processed in 1616 ms
2018-01-29 20:26:20,931[main]             [INFO ] [c.m.JmxRemoteLifecycleListener 336]  - The JMX Remote Listener has configured the registry on port 32001 and the server on port 32101 for the Platform server
2018-01-29 20:26:20,932[main]             [INFO ] [o.a.c.core.StandardService 416]      - Starting service Catalina
2018-01-29 20:26:20,932[main]             [INFO ] [o.a.c.core.StandardEngine 259]       - Starting Servlet Engine: Apache Tomcat/8.5.11
2018-01-29 20:26:22,152[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 923]        - Deploying web application archive /home/user/eclipse-che/tomcat/webapps/dashboard.war
2018-01-29 20:26:22,663[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 987]        - Deployment of web application archive /home/user/eclipse-che/tomcat/webapps/dashboard.war has finished in 510 ms
2018-01-29 20:26:22,667[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 923]        - Deploying web application archive /home/user/eclipse-che/tomcat/webapps/docs.war
2018-01-29 20:26:23,528[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 987]        - Deployment of web application archive /home/user/eclipse-che/tomcat/webapps/docs.war has finished in 861 ms
2018-01-29 20:26:23,534[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 923]        - Deploying web application archive /home/user/eclipse-che/tomcat/webapps/ROOT.war
2018-01-29 20:26:24,305[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 987]        - Deployment of web application archive /home/user/eclipse-che/tomcat/webapps/ROOT.war has finished in 771 ms
2018-01-29 20:26:24,310[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 923]        - Deploying web application archive /home/user/eclipse-che/tomcat/webapps/wsmaster.war
2018-01-29 20:26:25,884[ost-startStop-1]  [INFO ] [.e.c.c.d.JNDIDataSourceFactory 59]   - This=org.eclipse.che.core.db.h2.H2SQLJndiDataSourceFactory@5b4c08bf obj=ResourceRef[className=javax.sql.DataSource,factoryClassLocation=null,factoryClassName=org.apache.naming.factory.ResourceFactory,{type=scope,content=Shareable},{type=auth,content=Container},{type=singleton,content=true},{type=factory,content=org.eclipse.che.core.db.h2.H2SQLJndiDataSourceFactory}] name=che Context=org.apache.naming.NamingContext@1179a32 environment={}
2018-01-29 20:26:30,253[ost-startStop-1]  [WARN ] [i.f.kubernetes.client.Config 338]    - Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
2018-01-29 20:26:30,579[ost-startStop-1]  [INFO ] [o.e.c.p.o.c.OpenShiftConnector 306]  - openshiftWorkspaceEnvironmentProvider = class org.eclipse.che.plugin.openshift.client.OpenshiftWorkspaceEnvironmentProvider
2018-01-29 20:26:31,521[ost-startStop-1]  [INFO ] [o.f.c.i.d.DbSupportFactory 44]       - Database: jdbc:h2:che (H2 1.4)
2018-01-29 20:26:31,620[ost-startStop-1]  [INFO ] [o.f.c.i.util.VersionPrinter 44]      - Flyway 4.0.3 by Boxfuse
2018-01-29 20:26:31,626[ost-startStop-1]  [INFO ] [o.f.c.i.d.DbSupportFactory 44]       - Database: jdbc:h2:che (H2 1.4)
2018-01-29 20:26:31,668[ost-startStop-1]  [INFO ] [i.f.CustomSqlMigrationResolver 157]  - Searching for sql scripts in locations [classpath:che-schema]
2018-01-29 20:26:31,704[ost-startStop-1]  [INFO ] [o.f.c.i.command.DbValidate 44]       - Successfully validated 11 migrations (execution time 00:00.046s)
2018-01-29 20:26:31,738[ost-startStop-1]  [INFO ] [o.f.c.i.m.MetaDataTableImpl 44]      - Creating Metadata table: "PUBLIC"."schema_version"
2018-01-29 20:26:31,777[ost-startStop-1]  [INFO ] [o.f.c.i.command.DbMigrate 44]        - Current version of schema "PUBLIC": << Empty Schema >>
2018-01-29 20:26:31,787[ost-startStop-1]  [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "PUBLIC" to version 5.0.0.8.1 - 1__init.sql
2018-01-29 20:26:32,169[ost-startStop-1]  [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "PUBLIC" to version 5.0.0.9.1 - 1__add_index_on_workspace_temporary.sql
2018-01-29 20:26:32,191[ost-startStop-1]  [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "PUBLIC" to version 5.0.0.9.2 - 2__update_local_links_in_environments.sql
2018-01-29 20:26:32,217[ost-startStop-1]  [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "PUBLIC" to version 5.2.0.1 - 1__increase_project_attributes_values_length.sql
2018-01-29 20:26:32,260[ost-startStop-1]  [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "PUBLIC" to version 5.4.0.1 - 1__drop_user_to_account_relation.sql
2018-01-29 20:26:32,344[ost-startStop-1]  [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "PUBLIC" to version 5.4.0.2 - 2__create_missed_account_indexes.sql
2018-01-29 20:26:32,372[ost-startStop-1]  [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "PUBLIC" to version 5.6.0.1 - 1__add_exec_agent_where_terminal_agent_is_present.sql
2018-01-29 20:26:32,387[ost-startStop-1]  [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "PUBLIC" to version 5.7.0.1 - 1__add_factory.sql
2018-01-29 20:26:32,478[ost-startStop-1]  [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "PUBLIC" to version 5.7.0.2 - 2__remove_match_policy.sql
2018-01-29 20:26:32,533[ost-startStop-1]  [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "PUBLIC" to version 5.8.0.1 - 1__add_foreigh_key_indexes.sql
2018-01-29 20:26:32,573[ost-startStop-1]  [INFO ] [o.f.c.i.command.DbMigrate 44]        - Migrating schema "PUBLIC" to version 5.11.0.1 - 1__optimize_user_search.sql
2018-01-29 20:26:32,593[ost-startStop-1]  [INFO ] [o.f.c.i.command.DbMigrate 44]        - Successfully applied 11 migrations to schema "PUBLIC" (execution time 00:00.860s).
2018-01-29 20:26:35,210[ost-startStop-1]  [INFO ] [o.e.c.a.m.s.r.RecipeLoader 74]       - Recipes initialization finished
2018-01-29 20:26:35,235[ost-startStop-1]  [INFO ] [ogDirSetterEnvVariableProvider 38]   - WsAgentLogDirProvider = org.eclipse.che.plugin.docker.machine.WsAgentLogDirProvider@54553196
2018-01-29 20:26:35,238[ost-startStop-1]  [INFO ] [ogDirSetterEnvVariableProvider 37]   - ExecAgentLogDirProvider = org.eclipse.che.plugin.docker.machine.ExecAgentLogDirProvider@7ac776e4
2018-01-29 20:26:35,378[ost-startStop-1]  [INFO ] [e.c.a.w.s.i.ServerIdleDetector 58]   - Idling che server scheduled [timeout=1800] seconds]
2018-01-29 20:26:35,500[ted-scheduler-2]  [INFO ] [ockerAbandonedResourcesCleaner 126]  - List containers registered in the api: []
2018-01-29 20:26:36,422[ost-startStop-1]  [WARN ] [o.e.c.a.w.s.stack.StackLoader 134]   - No configured image found for stack che-in-che
2018-01-29 20:26:36,440[ost-startStop-1]  [INFO ] [o.e.c.a.w.s.stack.StackLoader 103]   - Stacks initialization finished
2018-01-29 20:26:37,116[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 987]        - Deployment of web application archive /home/user/eclipse-che/tomcat/webapps/wsmaster.war has finished in 12,805 ms
2018-01-29 20:26:37,120[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 923]        - Deploying web application archive /home/user/eclipse-che/tomcat/webapps/swagger.war
2018-01-29 20:26:37,382[ost-startStop-1]  [INFO ] [o.a.c.startup.HostConfig 987]        - Deployment of web application archive /home/user/eclipse-che/tomcat/webapps/swagger.war has finished in 262 ms
2018-01-29 20:26:37,394[main]             [INFO ] [o.a.c.http11.Http11NioProtocol 570]  - Starting ProtocolHandler [http-nio-6868]
2018-01-29 20:26:37,410[main]             [INFO ] [o.a.catalina.startup.Catalina 668]   - Server startup in 16515 ms
2018-01-29 20:46:26,450[nio-6868-exec-3]  [INFO ] [o.a.c.http11.Http11Processor 710]    - Error parsing HTTP request header
 Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level.
java.lang.IllegalArgumentException: Invalid character found in method name. HTTP method names must be tokens
        at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:421)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:667)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:798)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1434)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)
2018-01-29 20:46:26,451[nio-6868-exec-4]  [INFO ] [o.a.c.http11.Http11Processor 710]    - Error parsing HTTP request header
 Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level.
java.lang.IllegalArgumentException: Invalid character found in method name. HTTP method names must be tokens
        at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:421)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:667)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:798)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1434)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)
2018-01-29 20:46:43,928[nio-6868-exec-2]  [INFO ] [o.e.c.a.f.s.jpa.JpaFactoryDao 111]   - FactoryDao#getByAttributes #maxItems: 15 #skipCount: 0, #attributes: [{first=creator.userId, second=che}]
2018-01-29 20:47:12,014[io-6868-exec-10]  [INFO ] [o.e.c.a.w.s.WorkspaceManager 807]    - Workspace 'che/gCyborg' with id 'workspacebsetwzqtonmg1guy' created by user 'che'
2018-01-29 20:47:32,019[aceSharedPool-0]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = null
2018-01-29 20:47:32,026[aceSharedPool-0]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = null
2018-01-29 20:47:32,027[aceSharedPool-0]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = org.eclipse.che.plugin.docker.client.WorkspacesRoutingSuffixProvider@2c549906
2018-01-29 20:47:32,033[aceSharedPool-0]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = org.eclipse.che.plugin.docker.client.WorkspacesRoutingSuffixProvider@2c549906
2018-01-29 20:47:32,036[aceSharedPool-0]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = org.eclipse.che.plugin.docker.client.WorkspacesRoutingSuffixProvider@2c549906
2018-01-29 20:47:32,224[aceSharedPool-0]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 676]   - Launching 'org.eclipse.che.exec' agent at workspace workspacebsetwzqtonmg1guy
2018-01-29 20:47:34,783[aceSharedPool-0]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 676]   - Launching 'org.eclipse.che.terminal' agent at workspace workspacebsetwzqtonmg1guy
2018-01-29 20:51:05,091[nio-6868-exec-8]  [INFO ] [o.e.c.a.f.s.jpa.JpaFactoryDao 111]   - FactoryDao#getByAttributes #maxItems: 15 #skipCount: 0, #attributes: [{first=creator.userId, second=che}]
2018-01-29 20:51:51,952[aceSharedPool-0]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = null
2018-01-29 20:51:51,960[aceSharedPool-0]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = null
2018-01-29 20:51:51,960[aceSharedPool-0]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = org.eclipse.che.plugin.docker.client.WorkspacesRoutingSuffixProvider@2c549906
2018-01-29 20:51:51,960[aceSharedPool-0]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = org.eclipse.che.plugin.docker.client.WorkspacesRoutingSuffixProvider@2c549906
2018-01-29 20:51:51,961[aceSharedPool-0]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = org.eclipse.che.plugin.docker.client.WorkspacesRoutingSuffixProvider@2c549906
2018-01-29 20:51:52,144[aceSharedPool-0]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 676]   - Launching 'org.eclipse.che.exec' agent at workspace workspacebsetwzqtonmg1guy
2018-01-29 20:51:54,516[aceSharedPool-0]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 676]   - Launching 'org.eclipse.che.terminal' agent at workspace workspacebsetwzqtonmg1guy
2018-01-29 20:51:56,750[aceSharedPool-0]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 676]   - Launching 'org.eclipse.che.ws-agent' agent at workspace workspacebsetwzqtonmg1guy
2018-01-29 20:54:57,094[aceSharedPool-0]  [ERROR] [o.e.c.a.agent.WsAgentLauncher 147]   - Fail pinging ws agent with http://172.31.1.176:32780/api/ url in workspacebsetwzqtonmg1guy workspace in machine2t9v7livavokgw3t machine on localhost node.
2018-01-29 20:54:57,921[aceSharedPool-0]  [ERROR] [o.e.c.a.w.s.WorkspaceManager 683]    - Timeout. The Che server is unable to ping your workspace. This implies a network configuration issue, workspace boot failure, or an unusually slow workspace boot.
org.eclipse.che.api.core.ServerException: Timeout. The Che server is unable to ping your workspace. This implies a network configuration issue, workspace boot failure, or an unusually slow workspace boot.
        at org.eclipse.che.api.agent.WsAgentLauncher.launch(WsAgentLauncher.java:153)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.launchAgents(WorkspaceRuntimes.java:681)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$MachineAgentsLauncher.started(WorkspaceRuntimes.java:1048)
        at org.eclipse.che.api.environment.server.CheEnvironmentEngine.startEnvironmentQueue(CheEnvironmentEngine.java:800)
        at org.eclipse.che.api.environment.server.CheEnvironmentEngine.start(CheEnvironmentEngine.java:263)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.startEnvironmentAndPublishEvents(WorkspaceRuntimes.java:694)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.access$100(WorkspaceRuntimes.java:103)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$StartTask.call(WorkspaceRuntimes.java:986)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$StartTask.call(WorkspaceRuntimes.java:948)
        at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalCallable.call(CopyThreadLocalCallable.java:31)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
2018-01-29 20:56:10,572[aceSharedPool-1]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = null
2018-01-29 20:56:10,573[aceSharedPool-1]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = null
2018-01-29 20:56:10,574[aceSharedPool-1]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = org.eclipse.che.plugin.docker.client.WorkspacesRoutingSuffixProvider@2c549906
2018-01-29 20:56:10,574[aceSharedPool-1]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = org.eclipse.che.plugin.docker.client.WorkspacesRoutingSuffixProvider@2c549906
2018-01-29 20:56:10,575[aceSharedPool-1]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = org.eclipse.che.plugin.docker.client.WorkspacesRoutingSuffixProvider@2c549906
2018-01-29 20:56:10,643[aceSharedPool-1]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 676]   - Launching 'org.eclipse.che.exec' agent at workspace workspacebsetwzqtonmg1guy
2018-01-29 20:56:12,849[aceSharedPool-1]  [ERROR] [.a.a.s.l.AbstractAgentLauncher 165]  - An error occurs while starting 'Exec' agent in 'workspacebsetwzqtonmg1guy' workspace in 'machinehkey58e3g59rqi2u' machine on 'localhost' node. Detailed log:
[STDOUT] 2018/01/29 20:56:11 Exec-agent configuration
[STDOUT] 2018/01/29 20:56:11   Server
[STDOUT] 2018/01/29 20:56:11     - Address: :4412
[STDOUT] 2018/01/29 20:56:11     - Base path: ''
[STDOUT] 2018/01/29 20:56:11   Process executor
[STDOUT] 2018/01/29 20:56:11     - Logs dir: /root/che/exec-agent/logs
[STDOUT] 2018/01/29 20:56:11
[STDOUT] 2018/01/29 20:56:11 ⇩ Registered HTTPRoutes:
[STDOUT]
[STDOUT] 2018/01/29 20:56:11 Process Routes:
[STDOUT] 2018/01/29 20:56:11 ✓ Start Process ........................... POST   /process
[STDOUT] 2018/01/29 20:56:11 ✓ Get Process ............................. GET    /process/:pid
[STDOUT] 2018/01/29 20:56:11 ✓ Kill Process ............................ DELETE /process/:pid
[STDOUT] 2018/01/29 20:56:11 ✓ Get Process Logs ........................ GET    /process/:pid/logs
[STDOUT] 2018/01/29 20:56:11 ✓ Get Processes ........................... GET    /process
[STDOUT] 2018/01/29 20:56:11
[STDOUT] 2018/01/29 20:56:11 Exec-Agent WebSocket routes:
[STDOUT] 2018/01/29 20:56:11 ✓ Connect to Exec-Agent(websocket) ........ GET    /connect
[STDOUT] 2018/01/29 20:56:11
[STDOUT] 2018/01/29 20:56:11 ⇩ Registered RPCRoutes:
[STDOUT]
[STDOUT] 2018/01/29 20:56:11 Process Routes:
[STDOUT] 2018/01/29 20:56:11 ✓ process.start
[STDOUT] 2018/01/29 20:56:11 ✓ process.kill
[STDOUT] 2018/01/29 20:56:11 ✓ process.subscribe
[STDOUT] 2018/01/29 20:56:11 ✓ process.unsubscribe
[STDOUT] 2018/01/29 20:56:11 ✓ process.updateSubscriber
[STDOUT] 2018/01/29 20:56:11 ✓ process.getLogs
[STDOUT] 2018/01/29 20:56:11 ✓ process.getProcess
[STDOUT] 2018/01/29 20:56:11 ✓ process.getProcesses
2018-01-29 20:56:12,934[aceSharedPool-1]  [ERROR] [o.e.c.a.w.s.WorkspaceManager 683]    - Error occurs while initializing command [/bin/sh, -c, trap '[ -z "$(jobs -p)" ] || kill $(jobs -p); [ -e /tmp/docker-exec-16.pid ] && rm /tmp/docker-exec-16.pid' EXIT; echo $$>/tmp/docker-exec-16.pid; command -v pidof >/dev/null 2>&1 && {
    pidof che-exec-agent >/dev/null 2>&1 && echo 0 || echo 1
} || {
    ps -fC che-exec-agent >/dev/null 2>&1 && echo 0 || echo 1
}] in docker container 8f877d704481bc68bd75b7d5f31140109f05437336e4e79382fa04695789cc7b: Error response from docker API, status: 409, message: Container 8f877d704481bc68bd75b7d5f31140109f05437336e4e79382fa04695789cc7b is not running

org.eclipse.che.api.core.ServerException: Error occurs while initializing command [/bin/sh, -c, trap '[ -z "$(jobs -p)" ] || kill $(jobs -p); [ -e /tmp/docker-exec-16.pid ] && rm /tmp/docker-exec-16.pid' EXIT; echo $$>/tmp/docker-exec-16.pid; command -v pidof >/dev/null 2>&1 && {
    pidof che-exec-agent >/dev/null 2>&1 && echo 0 || echo 1
} || {
    ps -fC che-exec-agent >/dev/null 2>&1 && echo 0 || echo 1
}] in docker container 8f877d704481bc68bd75b7d5f31140109f05437336e4e79382fa04695789cc7b: Error response from docker API, status: 409, message: Container 8f877d704481bc68bd75b7d5f31140109f05437336e4e79382fa04695789cc7b is not running

        at org.eclipse.che.api.agent.server.launcher.AbstractAgentLauncher.launch(AbstractAgentLauncher.java:115)
        at org.eclipse.che.api.agent.ExecAgentLauncher.launch(ExecAgentLauncher.java:46)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.launchAgents(WorkspaceRuntimes.java:681)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$MachineAgentsLauncher.started(WorkspaceRuntimes.java:1048)
        at org.eclipse.che.api.environment.server.CheEnvironmentEngine.startEnvironmentQueue(CheEnvironmentEngine.java:800)
        at org.eclipse.che.api.environment.server.CheEnvironmentEngine.start(CheEnvironmentEngine.java:263)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.startEnvironmentAndPublishEvents(WorkspaceRuntimes.java:694)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.access$100(WorkspaceRuntimes.java:103)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$StartTask.call(WorkspaceRuntimes.java:986)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$StartTask.call(WorkspaceRuntimes.java:948)
        at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalCallable.call(CopyThreadLocalCallable.java:31)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
2018-01-29 20:56:23,514[aceSharedPool-0]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = null
2018-01-29 20:56:23,514[aceSharedPool-0]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = null
2018-01-29 20:56:23,515[aceSharedPool-0]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = org.eclipse.che.plugin.docker.client.WorkspacesRoutingSuffixProvider@2c549906
2018-01-29 20:56:23,515[aceSharedPool-0]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = org.eclipse.che.plugin.docker.client.WorkspacesRoutingSuffixProvider@2c549906
2018-01-29 20:56:23,515[aceSharedPool-0]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = org.eclipse.che.plugin.docker.client.WorkspacesRoutingSuffixProvider@2c549906
2018-01-29 20:56:23,593[aceSharedPool-0]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 676]   - Launching 'org.eclipse.che.exec' agent at workspace workspacebsetwzqtonmg1guy
2018-01-29 20:56:25,835[aceSharedPool-0]  [ERROR] [.a.a.s.l.AbstractAgentLauncher 173]  - An error occurs while starting 'Exec' agent in 'workspacebsetwzqtonmg1guy' workspace in 'machinepzzxoc14b15kjtrk' machine on 'localhost' node. The agent didn't produce any logs.
2018-01-29 20:56:25,899[aceSharedPool-0]  [ERROR] [o.e.c.a.w.s.WorkspaceManager 683]    - Error occurs while initializing command [/bin/sh, -c, trap '[ -z "$(jobs -p)" ] || kill $(jobs -p); [ -e /tmp/docker-exec-19.pid ] && rm /tmp/docker-exec-19.pid' EXIT; echo $$>/tmp/docker-exec-19.pid; command -v pidof >/dev/null 2>&1 && {
    pidof che-exec-agent >/dev/null 2>&1 && echo 0 || echo 1
} || {
    ps -fC che-exec-agent >/dev/null 2>&1 && echo 0 || echo 1
}] in docker container 8c8c45b94296399a680afccb513301f7eaa2b84322955916e45439bd440a058e: Error response from docker API, status: 409, message: Container 8c8c45b94296399a680afccb513301f7eaa2b84322955916e45439bd440a058e is not running

org.eclipse.che.api.core.ServerException: Error occurs while initializing command [/bin/sh, -c, trap '[ -z "$(jobs -p)" ] || kill $(jobs -p); [ -e /tmp/docker-exec-19.pid ] && rm /tmp/docker-exec-19.pid' EXIT; echo $$>/tmp/docker-exec-19.pid; command -v pidof >/dev/null 2>&1 && {
    pidof che-exec-agent >/dev/null 2>&1 && echo 0 || echo 1
} || {
    ps -fC che-exec-agent >/dev/null 2>&1 && echo 0 || echo 1
}] in docker container 8c8c45b94296399a680afccb513301f7eaa2b84322955916e45439bd440a058e: Error response from docker API, status: 409, message: Container 8c8c45b94296399a680afccb513301f7eaa2b84322955916e45439bd440a058e is not running

        at org.eclipse.che.api.agent.server.launcher.AbstractAgentLauncher.launch(AbstractAgentLauncher.java:115)
        at org.eclipse.che.api.agent.ExecAgentLauncher.launch(ExecAgentLauncher.java:46)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.launchAgents(WorkspaceRuntimes.java:681)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$MachineAgentsLauncher.started(WorkspaceRuntimes.java:1048)
        at org.eclipse.che.api.environment.server.CheEnvironmentEngine.startEnvironmentQueue(CheEnvironmentEngine.java:800)
        at org.eclipse.che.api.environment.server.CheEnvironmentEngine.start(CheEnvironmentEngine.java:263)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.startEnvironmentAndPublishEvents(WorkspaceRuntimes.java:694)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.access$100(WorkspaceRuntimes.java:103)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$StartTask.call(WorkspaceRuntimes.java:986)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$StartTask.call(WorkspaceRuntimes.java:948)
        at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalCallable.call(CopyThreadLocalCallable.java:31)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
2018-01-29 20:56:35,968[ool-12-thread-1]  [WARN ] [i.f.kubernetes.client.Config 338]    - Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
2018-01-29 20:56:36,220[ool-12-thread-1]  [ERROR] [o.e.c.a.c.n.EventService 109]        - Operation: [get]  for kind: [DeploymentConfig]  with name: [che]  in namespace: [eclipse-che]  failed.
io.fabric8.kubernetes.client.KubernetesClientException: Operation: [get]  for kind: [DeploymentConfig]  with name: [che]  in namespace: [eclipse-che]  failed.
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62)
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:71)
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:217)
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:181)
        at org.eclipse.che.plugin.openshift.client.OpenShiftConnector.idleCheServer(OpenShiftConnector.java:391)
        at org.eclipse.che.plugin.openshift.client.OpenShiftConnector.access$000(OpenShiftConnector.java:179)
        at org.eclipse.che.plugin.openshift.client.OpenShiftConnector$1.onEvent(OpenShiftConnector.java:303)
        at org.eclipse.che.plugin.openshift.client.OpenShiftConnector$1.onEvent(OpenShiftConnector.java:299)
        at org.eclipse.che.api.core.notification.EventService.publish(EventService.java:107)
        at org.eclipse.che.api.workspace.server.idle.ServerIdleDetector.run(ServerIdleDetector.java:93)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.UnknownHostException: kubernetes.default.svc: Name does not resolve
        at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
        at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
        at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
        at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
        at java.net.InetAddress.getAllByName(InetAddress.java:1192)
        at java.net.InetAddress.getAllByName(InetAddress.java:1126)
        at okhttp3.Dns$1.lookup(Dns.java:39)
        at okhttp3.internal.connection.RouteSelector.resetNextInetSocketAddress(RouteSelector.java:170)
        at okhttp3.internal.connection.RouteSelector.nextProxy(RouteSelector.java:136)
        at okhttp3.internal.connection.RouteSelector.next(RouteSelector.java:81)
        at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:171)
        at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:121)
        at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:100)
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
        at io.fabric8.openshift.client.internal.OpenShiftOAuthInterceptor.intercept(OpenShiftOAuthInterceptor.java:64)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
        at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185)
        at okhttp3.RealCall.execute(RealCall.java:69)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:377)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:343)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:312)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:295)
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:753)
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:206)
        ... 14 common frames omitted
2018-01-29 20:56:45,314[aceSharedPool-1]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = null
2018-01-29 20:56:45,315[aceSharedPool-1]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = null
2018-01-29 20:56:45,316[aceSharedPool-1]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = org.eclipse.che.plugin.docker.client.WorkspacesRoutingSuffixProvider@2c549906
2018-01-29 20:56:45,317[aceSharedPool-1]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = org.eclipse.che.plugin.docker.client.WorkspacesRoutingSuffixProvider@2c549906
2018-01-29 20:56:45,318[aceSharedPool-1]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = org.eclipse.che.plugin.docker.client.WorkspacesRoutingSuffixProvider@2c549906
2018-01-29 20:56:45,418[aceSharedPool-1]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 676]   - Launching 'org.eclipse.che.exec' agent at workspace workspacebsetwzqtonmg1guy
2018-01-29 20:56:47,713[aceSharedPool-1]  [ERROR] [.a.a.s.l.AbstractAgentLauncher 173]  - An error occurs while starting 'Exec' agent in 'workspacebsetwzqtonmg1guy' workspace in 'machine0g7g1140bvhjnrxh' machine on 'localhost' node. The agent didn't produce any logs.
2018-01-29 20:56:47,776[aceSharedPool-1]  [ERROR] [o.e.c.a.w.s.WorkspaceManager 683]    - Error occurs while initializing command [/bin/sh, -c, trap '[ -z "$(jobs -p)" ] || kill $(jobs -p); [ -e /tmp/docker-exec-22.pid ] && rm /tmp/docker-exec-22.pid' EXIT; echo $$>/tmp/docker-exec-22.pid; command -v pidof >/dev/null 2>&1 && {
    pidof che-exec-agent >/dev/null 2>&1 && echo 0 || echo 1
} || {
    ps -fC che-exec-agent >/dev/null 2>&1 && echo 0 || echo 1
}] in docker container 0cd04ada71eff1004a38c77c41db63094cc95bef51cdfc2ad206dc54966592a0: Error response from docker API, status: 409, message: Container 0cd04ada71eff1004a38c77c41db63094cc95bef51cdfc2ad206dc54966592a0 is not running

org.eclipse.che.api.core.ServerException: Error occurs while initializing command [/bin/sh, -c, trap '[ -z "$(jobs -p)" ] || kill $(jobs -p); [ -e /tmp/docker-exec-22.pid ] && rm /tmp/docker-exec-22.pid' EXIT; echo $$>/tmp/docker-exec-22.pid; command -v pidof >/dev/null 2>&1 && {
    pidof che-exec-agent >/dev/null 2>&1 && echo 0 || echo 1
} || {
    ps -fC che-exec-agent >/dev/null 2>&1 && echo 0 || echo 1
}] in docker container 0cd04ada71eff1004a38c77c41db63094cc95bef51cdfc2ad206dc54966592a0: Error response from docker API, status: 409, message: Container 0cd04ada71eff1004a38c77c41db63094cc95bef51cdfc2ad206dc54966592a0 is not running

        at org.eclipse.che.api.agent.server.launcher.AbstractAgentLauncher.launch(AbstractAgentLauncher.java:115)
        at org.eclipse.che.api.agent.ExecAgentLauncher.launch(ExecAgentLauncher.java:46)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.launchAgents(WorkspaceRuntimes.java:681)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$MachineAgentsLauncher.started(WorkspaceRuntimes.java:1048)
        at org.eclipse.che.api.environment.server.CheEnvironmentEngine.startEnvironmentQueue(CheEnvironmentEngine.java:800)
        at org.eclipse.che.api.environment.server.CheEnvironmentEngine.start(CheEnvironmentEngine.java:263)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.startEnvironmentAndPublishEvents(WorkspaceRuntimes.java:694)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.access$100(WorkspaceRuntimes.java:103)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$StartTask.call(WorkspaceRuntimes.java:986)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$StartTask.call(WorkspaceRuntimes.java:948)
        at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalCallable.call(CopyThreadLocalCallable.java:31)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
2018-01-29 21:00:33,434[aceSharedPool-0]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = null
2018-01-29 21:00:33,435[aceSharedPool-0]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = null
2018-01-29 21:00:33,435[aceSharedPool-0]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = org.eclipse.che.plugin.docker.client.WorkspacesRoutingSuffixProvider@2c549906
2018-01-29 21:00:33,437[aceSharedPool-0]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = org.eclipse.che.plugin.docker.client.WorkspacesRoutingSuffixProvider@2c549906
2018-01-29 21:00:33,437[aceSharedPool-0]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = org.eclipse.che.plugin.docker.client.WorkspacesRoutingSuffixProvider@2c549906
2018-01-29 21:00:33,542[aceSharedPool-0]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 676]   - Launching 'org.eclipse.che.exec' agent at workspace workspacebsetwzqtonmg1guy
2018-01-29 21:00:35,881[aceSharedPool-0]  [ERROR] [.a.a.s.l.AbstractAgentLauncher 165]  - An error occurs while starting 'Exec' agent in 'workspacebsetwzqtonmg1guy' workspace in 'machineqek4xah9afrvkgpl' machine on 'localhost' node. Detailed log:
[STDOUT] 2018/01/29 21:00:34 Exec-agent configuration
[STDOUT] 2018/01/29 21:00:34   Server
[STDOUT] 2018/01/29 21:00:34     - Address: :4412
[STDOUT] 2018/01/29 21:00:34     - Base path: ''
[STDOUT] 2018/01/29 21:00:34   Process executor
[STDOUT] 2018/01/29 21:00:34     - Logs dir: /root/che/exec-agent/logs
[STDOUT] 2018/01/29 21:00:34
[STDOUT] 2018/01/29 21:00:34 ⇩ Registered HTTPRoutes:
[STDOUT]
[STDOUT] 2018/01/29 21:00:34 Process Routes:
[STDOUT] 2018/01/29 21:00:34 ✓ Start Process ........................... POST   /process
[STDOUT] 2018/01/29 21:00:34 ✓ Get Process ............................. GET    /process/:pid
[STDOUT] 2018/01/29 21:00:34 ✓ Kill Process ............................ DELETE /process/:pid
[STDOUT] 2018/01/29 21:00:34 ✓ Get Process Logs ........................ GET    /process/:pid/logs
[STDOUT] 2018/01/29 21:00:34 ✓ Get Processes ........................... GET    /process
[STDOUT] 2018/01/29 21:00:34
[STDOUT] 2018/01/29 21:00:34 Exec-Agent WebSocket routes:
[STDOUT] 2018/01/29 21:00:34 ✓ Connect to Exec-Agent(websocket) ........ GET    /connect
[STDOUT] 2018/01/29 21:00:34
[STDOUT] 2018/01/29 21:00:34 ⇩ Registered RPCRoutes:
[STDOUT]
[STDOUT] 2018/01/29 21:00:34 Process Routes:
[STDOUT] 2018/01/29 21:00:34 ✓ process.start
[STDOUT] 2018/01/29 21:00:34 ✓ process.kill
[STDOUT] 2018/01/29 21:00:34 ✓ process.subscribe
[STDOUT] 2018/01/29 21:00:34 ✓ process.unsubscribe
[STDOUT] 2018/01/29 21:00:34 ✓ process.updateSubscriber
[STDOUT] 2018/01/29 21:00:34 ✓ process.getLogs
[STDOUT] 2018/01/29 21:00:34 ✓ process.getProcess
[STDOUT] 2018/01/29 21:00:34 ✓ process.getProcesses
2018-01-29 21:00:35,965[aceSharedPool-0]  [ERROR] [o.e.c.a.w.s.WorkspaceManager 683]    - Error occurs while initializing command [/bin/sh, -c, trap '[ -z "$(jobs -p)" ] || kill $(jobs -p); [ -e /tmp/docker-exec-25.pid ] && rm /tmp/docker-exec-25.pid' EXIT; echo $$>/tmp/docker-exec-25.pid; command -v pidof >/dev/null 2>&1 && {
    pidof che-exec-agent >/dev/null 2>&1 && echo 0 || echo 1
} || {
    ps -fC che-exec-agent >/dev/null 2>&1 && echo 0 || echo 1
}] in docker container 9e59ff20ef9eaedbdfff60948328bc79ee17fb629e66e79abd3cba4471cf4f60: Error response from docker API, status: 409, message: Container 9e59ff20ef9eaedbdfff60948328bc79ee17fb629e66e79abd3cba4471cf4f60 is not running

org.eclipse.che.api.core.ServerException: Error occurs while initializing command [/bin/sh, -c, trap '[ -z "$(jobs -p)" ] || kill $(jobs -p); [ -e /tmp/docker-exec-25.pid ] && rm /tmp/docker-exec-25.pid' EXIT; echo $$>/tmp/docker-exec-25.pid; command -v pidof >/dev/null 2>&1 && {
    pidof che-exec-agent >/dev/null 2>&1 && echo 0 || echo 1
} || {
    ps -fC che-exec-agent >/dev/null 2>&1 && echo 0 || echo 1
}] in docker container 9e59ff20ef9eaedbdfff60948328bc79ee17fb629e66e79abd3cba4471cf4f60: Error response from docker API, status: 409, message: Container 9e59ff20ef9eaedbdfff60948328bc79ee17fb629e66e79abd3cba4471cf4f60 is not running

        at org.eclipse.che.api.agent.server.launcher.AbstractAgentLauncher.launch(AbstractAgentLauncher.java:115)
        at org.eclipse.che.api.agent.ExecAgentLauncher.launch(ExecAgentLauncher.java:46)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.launchAgents(WorkspaceRuntimes.java:681)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$MachineAgentsLauncher.started(WorkspaceRuntimes.java:1048)
        at org.eclipse.che.api.environment.server.CheEnvironmentEngine.startEnvironmentQueue(CheEnvironmentEngine.java:800)
        at org.eclipse.che.api.environment.server.CheEnvironmentEngine.start(CheEnvironmentEngine.java:263)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.startEnvironmentAndPublishEvents(WorkspaceRuntimes.java:694)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.access$100(WorkspaceRuntimes.java:103)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$StartTask.call(WorkspaceRuntimes.java:986)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$StartTask.call(WorkspaceRuntimes.java:948)
        at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalCallable.call(CopyThreadLocalCallable.java:31)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
2018-01-29 21:00:59,629[aceSharedPool-1]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = null
2018-01-29 21:00:59,630[aceSharedPool-1]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = null
2018-01-29 21:00:59,630[aceSharedPool-1]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = org.eclipse.che.plugin.docker.client.WorkspacesRoutingSuffixProvider@2c549906
2018-01-29 21:00:59,631[aceSharedPool-1]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = org.eclipse.che.plugin.docker.client.WorkspacesRoutingSuffixProvider@2c549906
2018-01-29 21:00:59,631[aceSharedPool-1]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = org.eclipse.che.plugin.docker.client.WorkspacesRoutingSuffixProvider@2c549906
2018-01-29 21:00:59,727[aceSharedPool-1]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 676]   - Launching 'org.eclipse.che.exec' agent at workspace workspacebsetwzqtonmg1guy
2018-01-29 21:01:02,023[aceSharedPool-1]  [ERROR] [.a.a.s.l.AbstractAgentLauncher 173]  - An error occurs while starting 'Exec' agent in 'workspacebsetwzqtonmg1guy' workspace in 'machinerncltl2vy8vh2kzq' machine on 'localhost' node. The agent didn't produce any logs.
2018-01-29 21:01:02,080[aceSharedPool-1]  [ERROR] [o.e.c.a.w.s.WorkspaceManager 683]    - Error occurs while initializing command [/bin/sh, -c, trap '[ -z "$(jobs -p)" ] || kill $(jobs -p); [ -e /tmp/docker-exec-28.pid ] && rm /tmp/docker-exec-28.pid' EXIT; echo $$>/tmp/docker-exec-28.pid; command -v pidof >/dev/null 2>&1 && {
    pidof che-exec-agent >/dev/null 2>&1 && echo 0 || echo 1
} || {
    ps -fC che-exec-agent >/dev/null 2>&1 && echo 0 || echo 1
}] in docker container 9a1f143521a958edc4e11664503b771dfea978324bed3b69fab456fdd5bc2850: Error response from docker API, status: 409, message: Container 9a1f143521a958edc4e11664503b771dfea978324bed3b69fab456fdd5bc2850 is not running

org.eclipse.che.api.core.ServerException: Error occurs while initializing command [/bin/sh, -c, trap '[ -z "$(jobs -p)" ] || kill $(jobs -p); [ -e /tmp/docker-exec-28.pid ] && rm /tmp/docker-exec-28.pid' EXIT; echo $$>/tmp/docker-exec-28.pid; command -v pidof >/dev/null 2>&1 && {
    pidof che-exec-agent >/dev/null 2>&1 && echo 0 || echo 1
} || {
    ps -fC che-exec-agent >/dev/null 2>&1 && echo 0 || echo 1
}] in docker container 9a1f143521a958edc4e11664503b771dfea978324bed3b69fab456fdd5bc2850: Error response from docker API, status: 409, message: Container 9a1f143521a958edc4e11664503b771dfea978324bed3b69fab456fdd5bc2850 is not running

        at org.eclipse.che.api.agent.server.launcher.AbstractAgentLauncher.launch(AbstractAgentLauncher.java:115)
        at org.eclipse.che.api.agent.ExecAgentLauncher.launch(ExecAgentLauncher.java:46)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.launchAgents(WorkspaceRuntimes.java:681)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$MachineAgentsLauncher.started(WorkspaceRuntimes.java:1048)
        at org.eclipse.che.api.environment.server.CheEnvironmentEngine.startEnvironmentQueue(CheEnvironmentEngine.java:800)
        at org.eclipse.che.api.environment.server.CheEnvironmentEngine.start(CheEnvironmentEngine.java:263)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.startEnvironmentAndPublishEvents(WorkspaceRuntimes.java:694)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.access$100(WorkspaceRuntimes.java:103)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$StartTask.call(WorkspaceRuntimes.java:986)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$StartTask.call(WorkspaceRuntimes.java:948)
        at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalCallable.call(CopyThreadLocalCallable.java:31)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
2018-01-29 21:01:07,606[io-6868-exec-10]  [INFO ] [o.e.c.a.f.s.jpa.JpaFactoryDao 111]   - FactoryDao#getByAttributes #maxItems: 15 #skipCount: 0, #attributes: [{first=creator.userId, second=che}]
2018-01-29 21:01:12,572[aceSharedPool-0]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = null
2018-01-29 21:01:12,574[aceSharedPool-0]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = null
2018-01-29 21:01:12,575[aceSharedPool-0]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = org.eclipse.che.plugin.docker.client.WorkspacesRoutingSuffixProvider@2c549906
2018-01-29 21:01:12,576[aceSharedPool-0]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = org.eclipse.che.plugin.docker.client.WorkspacesRoutingSuffixProvider@2c549906
2018-01-29 21:01:12,578[aceSharedPool-0]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = org.eclipse.che.plugin.docker.client.WorkspacesRoutingSuffixProvider@2c549906
2018-01-29 21:01:12,682[aceSharedPool-0]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 676]   - Launching 'org.eclipse.che.exec' agent at workspace workspacebsetwzqtonmg1guy
2018-01-29 21:01:14,927[aceSharedPool-0]  [ERROR] [.a.a.s.l.AbstractAgentLauncher 173]  - An error occurs while starting 'Exec' agent in 'workspacebsetwzqtonmg1guy' workspace in 'machinebxaboz4wn78o5n6r' machine on 'localhost' node. The agent didn't produce any logs.
2018-01-29 21:01:14,990[aceSharedPool-0]  [ERROR] [o.e.c.a.w.s.WorkspaceManager 683]    - Error occurs while initializing command [/bin/sh, -c, trap '[ -z "$(jobs -p)" ] || kill $(jobs -p); [ -e /tmp/docker-exec-31.pid ] && rm /tmp/docker-exec-31.pid' EXIT; echo $$>/tmp/docker-exec-31.pid; command -v pidof >/dev/null 2>&1 && {
    pidof che-exec-agent >/dev/null 2>&1 && echo 0 || echo 1
} || {
    ps -fC che-exec-agent >/dev/null 2>&1 && echo 0 || echo 1
}] in docker container 8b9f4344a0e3657b8c3efa2b61d31669d4162c9fe8a0390de65c564085f5159d: Error response from docker API, status: 409, message: Container 8b9f4344a0e3657b8c3efa2b61d31669d4162c9fe8a0390de65c564085f5159d is not running

org.eclipse.che.api.core.ServerException: Error occurs while initializing command [/bin/sh, -c, trap '[ -z "$(jobs -p)" ] || kill $(jobs -p); [ -e /tmp/docker-exec-31.pid ] && rm /tmp/docker-exec-31.pid' EXIT; echo $$>/tmp/docker-exec-31.pid; command -v pidof >/dev/null 2>&1 && {
    pidof che-exec-agent >/dev/null 2>&1 && echo 0 || echo 1
} || {
    ps -fC che-exec-agent >/dev/null 2>&1 && echo 0 || echo 1
}] in docker container 8b9f4344a0e3657b8c3efa2b61d31669d4162c9fe8a0390de65c564085f5159d: Error response from docker API, status: 409, message: Container 8b9f4344a0e3657b8c3efa2b61d31669d4162c9fe8a0390de65c564085f5159d is not running

        at org.eclipse.che.api.agent.server.launcher.AbstractAgentLauncher.launch(AbstractAgentLauncher.java:115)
        at org.eclipse.che.api.agent.ExecAgentLauncher.launch(ExecAgentLauncher.java:46)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.launchAgents(WorkspaceRuntimes.java:681)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$MachineAgentsLauncher.started(WorkspaceRuntimes.java:1048)
        at org.eclipse.che.api.environment.server.CheEnvironmentEngine.startEnvironmentQueue(CheEnvironmentEngine.java:800)
        at org.eclipse.che.api.environment.server.CheEnvironmentEngine.start(CheEnvironmentEngine.java:263)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.startEnvironmentAndPublishEvents(WorkspaceRuntimes.java:694)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.access$100(WorkspaceRuntimes.java:103)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$StartTask.call(WorkspaceRuntimes.java:986)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$StartTask.call(WorkspaceRuntimes.java:948)
        at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalCallable.call(CopyThreadLocalCallable.java:31)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
2018-01-29 21:01:19,712[aceSharedPool-1]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = null
2018-01-29 21:01:19,712[aceSharedPool-1]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = null
2018-01-29 21:01:19,712[aceSharedPool-1]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = org.eclipse.che.plugin.docker.client.WorkspacesRoutingSuffixProvider@2c549906
2018-01-29 21:01:19,719[aceSharedPool-1]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = org.eclipse.che.plugin.docker.client.WorkspacesRoutingSuffixProvider@2c549906
2018-01-29 21:01:19,720[aceSharedPool-1]  [INFO ] [m.BaseServerEvaluationStrategy 141]  - cheWorkspacesRoutingSuffixProvider = org.eclipse.che.plugin.docker.client.WorkspacesRoutingSuffixProvider@2c549906
2018-01-29 21:01:19,839[aceSharedPool-1]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 676]   - Launching 'org.eclipse.che.exec' agent at workspace workspacebsetwzqtonmg1guy
2018-01-29 21:01:21,987[aceSharedPool-1]  [ERROR] [.a.a.s.l.AbstractAgentLauncher 165]  - An error occurs while starting 'Exec' agent in 'workspacebsetwzqtonmg1guy' workspace in 'machinepes1c6xee84eo0j9' machine on 'localhost' node. Detailed log:
[STDOUT] OCI runtime exec failed: exec failed: container_linux.go:296: starting container process caused "process_linux.go:108: writing config to pipe caused \"write init-p: broken pipe\"": unknown
2018-01-29 21:01:22,041[aceSharedPool-1]  [ERROR] [o.e.c.a.w.s.WorkspaceManager 683]    - Error occurs while initializing command [/bin/sh, -c, trap '[ -z "$(jobs -p)" ] || kill $(jobs -p); [ -e /tmp/docker-exec-34.pid ] && rm /tmp/docker-exec-34.pid' EXIT; echo $$>/tmp/docker-exec-34.pid; command -v pidof >/dev/null 2>&1 && {
    pidof che-exec-agent >/dev/null 2>&1 && echo 0 || echo 1
} || {
    ps -fC che-exec-agent >/dev/null 2>&1 && echo 0 || echo 1
}] in docker container ebbdff02acb6214e845b9902f98799313aeee5b039531f60798910e1c3931ab4: Error response from docker API, status: 409, message: Container ebbdff02acb6214e845b9902f98799313aeee5b039531f60798910e1c3931ab4 is not running

org.eclipse.che.api.core.ServerException: Error occurs while initializing command [/bin/sh, -c, trap '[ -z "$(jobs -p)" ] || kill $(jobs -p); [ -e /tmp/docker-exec-34.pid ] && rm /tmp/docker-exec-34.pid' EXIT; echo $$>/tmp/docker-exec-34.pid; command -v pidof >/dev/null 2>&1 && {
    pidof che-exec-agent >/dev/null 2>&1 && echo 0 || echo 1
} || {
    ps -fC che-exec-agent >/dev/null 2>&1 && echo 0 || echo 1
}] in docker container ebbdff02acb6214e845b9902f98799313aeee5b039531f60798910e1c3931ab4: Error response from docker API, status: 409, message: Container ebbdff02acb6214e845b9902f98799313aeee5b039531f60798910e1c3931ab4 is not running

        at org.eclipse.che.api.agent.server.launcher.AbstractAgentLauncher.launch(AbstractAgentLauncher.java:115)
        at org.eclipse.che.api.agent.ExecAgentLauncher.launch(ExecAgentLauncher.java:46)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.launchAgents(WorkspaceRuntimes.java:681)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$MachineAgentsLauncher.started(WorkspaceRuntimes.java:1048)
        at org.eclipse.che.api.environment.server.CheEnvironmentEngine.startEnvironmentQueue(CheEnvironmentEngine.java:800)
        at org.eclipse.che.api.environment.server.CheEnvironmentEngine.start(CheEnvironmentEngine.java:263)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.startEnvironmentAndPublishEvents(WorkspaceRuntimes.java:694)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.access$100(WorkspaceRuntimes.java:103)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$StartTask.call(WorkspaceRuntimes.java:986)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$StartTask.call(WorkspaceRuntimes.java:948)
        at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalCallable.call(CopyThreadLocalCallable.java:31)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)