ray: AssignProcessToJobObject() failed in ray.init()

Search before asking

I searched the issues and found no similar issues.

Ray Component

Ray Core

What happened + What you expected to happen

When starting ray.init() from terminal - both in Run and Debug mode, the following exception is thrown:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\python_venvs\gem_dl\lib\site-packages\ray\_private\client_mode_hook.py", line 105, in wrapper
    return func(*args, **kwargs)
  File "C:\Users\python_venvs\gem_dl\lib\site-packages\ray\worker.py", line 908, in init
    _global_node = ray.node.Node(
  File "C:\Users\python_venvs\gem_dl\lib\site-packages\ray\node.py", line 235, in __init__
    self.start_ray_processes()
  File "C:\Users\python_venvs\gem_dl\lib\site-packages\ray\node.py", line 947, in start_ray_processes
    self.start_raylet(plasma_directory, object_store_memory)
  File "C:\Users\python_venvs\gem_dl\lib\site-packages\ray\node.py", line 808, in start_raylet
    process_info = ray._private.services.start_raylet(
  File "C:\Users\python_venvs\gem_dl\lib\site-packages\ray\_private\services.py", line 1642, in start_raylet
    process_info = start_ray_process(
  File "C:\Users\python_venvs\gem_dl\lib\site-packages\ray\_private\services.py", line 665, in start_ray_process
    ray._private.utils.set_kill_child_on_death_win32(process)
  File "C:\Users\python_venvs\gem_dl\lib\site-packages\ray\_private\utils.py", line 796, in set_kill_child_on_death_win32
    raise OSError(ctypes.get_last_error(),
OSError: [Errno 0] AssignProcessToJobObject() failed

Versions / Dependencies

ray==1.9.2
Python 3.8.10
Windows 10

Reproduction script

Reproduction (REQUIRED) In a python terminal run this:

import ray
ray.init()

Anything else

No response

Are you willing to submit a PR?

Yes I am willing to submit a PR!

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 25 (12 by maintainers)

Most upvoted comments

I can also confirm that I faced the mentioned issue when using Python 3.10 from the Microsoft Store.

By downloading Python 3.10 from the official website, I was able to use Ray without any problem (so far).

thomasbbrunner on Feb 19, 2024

I’m running on a local install.

I checked again on the logs. They are located in (with different logs for various datetimes, but they’re all the same in content): ‘C:\Users_________\AppData\Local\Temp\ray\session_2023-08-23_12-47-44_328823_34212\logs.…’

All the log files are empty apart from gcs_server.out. Here’s the content:

\AppData\Local\Temp\ray\session_2023-08-23_12-47-44_328823_34212\logs\gcs_server.out

[2023-08-23 12:47:44,370 I 41356 40156] (gcs_server.exe) io_service_pool.cc:35: IOServicePool is running with 1 io_service. [2023-08-23 12:47:44,378 I 41356 40156] (gcs_server.exe) event.cc:234: Set ray event level to warning [2023-08-23 12:47:44,378 I 41356 40156] (gcs_server.exe) event.cc:342: Ray Event initialized for GCS [2023-08-23 12:47:44,378 I 41356 40156] (gcs_server.exe) gcs_server.cc:74: GCS storage type is StorageType::IN_MEMORY [2023-08-23 12:47:44,379 I 41356 40156] (gcs_server.exe) gcs_init_data.cc:44: Loading job table data. [2023-08-23 12:47:44,379 I 41356 40156] (gcs_server.exe) gcs_init_data.cc:56: Loading node table data. [2023-08-23 12:47:44,379 I 41356 40156] (gcs_server.exe) gcs_init_data.cc:68: Loading cluster resources table data. [2023-08-23 12:47:44,379 I 41356 40156] (gcs_server.exe) gcs_init_data.cc:95: Loading actor table data. [2023-08-23 12:47:44,380 I 41356 40156] (gcs_server.exe) gcs_init_data.cc:108: Loading actor task spec table data. [2023-08-23 12:47:44,380 I 41356 40156] (gcs_server.exe) gcs_init_data.cc:81: Loading placement group table data. [2023-08-23 12:47:44,380 I 41356 40156] (gcs_server.exe) gcs_init_data.cc:48: Finished loading job table data, size = 0 [2023-08-23 12:47:44,380 I 41356 40156] (gcs_server.exe) gcs_init_data.cc:60: Finished loading node table data, size = 0 [2023-08-23 12:47:44,380 I 41356 40156] (gcs_server.exe) gcs_init_data.cc:72: Finished loading cluster resources table data, size = 0 [2023-08-23 12:47:44,380 I 41356 40156] (gcs_server.exe) gcs_init_data.cc:99: Finished loading actor table data, size = 0 [2023-08-23 12:47:44,380 I 41356 40156] (gcs_server.exe) gcs_init_data.cc:112: Finished loading actor task spec table data, size = 0 [2023-08-23 12:47:44,380 I 41356 40156] (gcs_server.exe) gcs_init_data.cc:86: Finished loading placement group table data, size = 0 [2023-08-23 12:47:44,380 I 41356 40156] (gcs_server.exe) gcs_server.cc:164: No existing server cluster ID found. Generating new ID: a1d4cb0f4085e98a3f7e5ff3904e3fd0b747d5766d8861605bf34d4a [2023-08-23 12:47:44,380 I 41356 40156] (gcs_server.exe) gcs_server.cc:658: Autoscaler V2 enabled: 0 [2023-08-23 12:47:44,381 I 41356 40156] (gcs_server.exe) grpc_server.cc:129: GcsServer server started, listening on port 64673. [2023-08-23 12:47:44,426 I 41356 40156] (gcs_server.exe) gcs_server.cc:255: GcsNodeManager: - RegisterNode request count: 0 - DrainNode request count: 0 - GetAllNodeInfo request count: 0 - GetInternalConfig request count: 0 GcsActorManager: RegisterActor request count: 0 CreateActor request count: 0 GetActorInfo request count: 0 GetNamedActorInfo request count: 0 GetAllActorInfo request count: 0 KillActor request count: 0 ListNamedActors request count: 0 Registered actors count: 0 Destroyed actors count: 0 Named actors count: 0 Unresolved actors count: 0 Pending actors count: 0 Created actors count: 0 owners_: 0 actor_to_register_callbacks_: 0 actor_to_create_callbacks_: 0 sorted_destroyed_actor_list_: 0 GcsResourceManager: GetResources request count: 0 GetAllAvailableResources request count0 ReportResourceUsage request count: 0 GetAllResourceUsage request count: 0 GcsPlacementGroupManager: CreatePlacementGroup request count: 0 RemovePlacementGroup request count: 0 GetPlacementGroup request count: 0 GetAllPlacementGroup request count: 0 WaitPlacementGroupUntilReady request count: 0 GetNamedPlacementGroup request count: 0 Scheduling pending placement group count: 0 Registered placement groups count: 0 Named placement group count: 0 Pending placement groups count: 0 Infeasible placement groups count: 0 GcsPublisher {} [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 0 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Total num bytes of task event stored: 0MiB -Current num of task events stored: 0 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 0 -Total num of driver tasks: 0 [2023-08-23 12:47:44,426 I 41356 40156] (gcs_server.exe) gcs_server.cc:872: Event stats: Global stats: 30 total (16 active) Queueing time: mean = 4.585 ms, max = 45.625 ms, min = 2.500 us, total = 137.562 ms Execution time: mean = 1.532 ms, total = 45.947 ms Event stats: InternalKVGcsService.grpc_client.InternalKVPut - 6 total (6 active), CPU time: mean = 0.000 s, total = 0.000 s GcsInMemoryStore.GetAll - 6 total (0 active), CPU time: mean = 14.767 us, total = 88.600 us InternalKVGcsService.grpc_server.InternalKVPut - 6 total (4 active), CPU time: mean = 2.050 us, total = 12.300 us GcsInMemoryStore.Put - 5 total (2 active), CPU time: mean = 9.164 ms, total = 45.819 ms PeriodicalRunner.RunFnPeriodically - 4 total (2 active, 1 running), CPU time: mean = 1.325 us, total = 5.300 us RayletLoadPulled - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s UNKNOWN - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s GcsInMemoryStore.Get - 1 total (0 active), CPU time: mean = 22.600 us, total = 22.600 us [2023-08-23 12:47:44,426 I 41356 40156] (gcs_server.exe) gcs_server.cc:873: GcsTaskManager Event stats: Global stats: 0 total (0 active) Queueing time: mean = -nan(ind) s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Execution time: mean = -nan(ind) s, total = 0.000 s Event stats: [2023-08-23 12:47:56,476 W 41356 6664] (gcs_server.exe) metric_exporter.cc:212: [1] Export metrics to agent failed: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details: . This won’t affect Ray, but you can lose metrics from the cluster. [2023-08-23 12:48:44,438 I 41356 40156] (gcs_server.exe) gcs_server.cc:255: GcsNodeManager: RegisterNode request count: 0 DrainNode request count: 0 GetAllNodeInfo request count: 0 GetInternalConfig request count: 0 GcsActorManager: RegisterActor request count: 0 CreateActor request count: 0 GetActorInfo request count: 0 GetNamedActorInfo request count: 0 GetAllActorInfo request count: 0 KillActor request count: 0 ListNamedActors request count: 0 Registered actors count: 0 Destroyed actors count: 0 Named actors count: 0 Unresolved actors count: 0 Pending actors count: 0 Created actors count: 0 owners_: 0 actor_to_register_callbacks_: 0 actor_to_create_callbacks_: 0 sorted_destroyed_actor_list_: 0 GcsResourceManager: GetResources request count: 0 GetAllAvailableResources request count0 ReportResourceUsage request count: 0 GetAllResourceUsage request count: 0 GcsPlacementGroupManager: CreatePlacementGroup request count: 0 RemovePlacementGroup request count: 0 GetPlacementGroup request count: 0 GetAllPlacementGroup request count: 0 WaitPlacementGroupUntilReady request count: 0 GetNamedPlacementGroup request count: 0 Scheduling pending placement group count: 0 Registered placement groups count: 0 Named placement group count: 0 Pending placement groups count: 0 Infeasible placement groups count: 0 GcsPublisher {} [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 0 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Total num bytes of task event stored: 0MiB -Current num of task events stored: 0 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 0 -Total num of driver tasks: 0 [2023-08-23 12:48:44,438 I 41356 40156] (gcs_server.exe) gcs_server.cc:872: Event stats: Global stats: 327 total (4 active) Queueing time: mean = 2.524 ms, max = 45.625 ms, min = 2.030 us, total = 825.445 ms Execution time: mean = 193.377 us, total = 63.234 ms Event stats: GcsInMemoryStore.Put - 79 total (0 active), CPU time: mean = 596.677 us, total = 47.137 ms InternalKVGcsService.grpc_server.InternalKVPut - 76 total (0 active), CPU time: mean = 10.321 us, total = 784.416 us InternalKVGcsService.grpc_client.InternalKVPut - 72 total (0 active), CPU time: mean = 5.984 us, total = 430.849 us RayletLoadPulled - 60 total (1 active), CPU time: mean = 4.130 us, total = 247.828 us UNKNOWN - 20 total (1 active), CPU time: mean = 6.982 us, total = 139.636 us GcsInMemoryStore.GetAll - 6 total (0 active), CPU time: mean = 14.767 us, total = 88.600 us GCSServer.deadline_timer.debug_state_dump - 6 total (1 active), CPU time: mean = 1.941 ms, total = 11.647 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), CPU time: mean = 678.900 us, total = 2.716 ms GcsInMemoryStore.Get - 2 total (0 active), CPU time: mean = 14.127 us, total = 28.253 us InternalKVGcsService.grpc_server.InternalKVGet - 1 total (0 active), CPU time: mean = 14.884 us, total = 14.884 us GCSServer.deadline_timer.debug_state_event_stats_print - 1 total (1 active, 1 running), CPU time: mean = 0.000 s, total = 0.000 s [2023-08-23 12:48:44,438 I 41356 40156] (gcs_server.exe) gcs_server.cc:873: GcsTaskManager Event stats: Global stats: 0 total (0 active) Queueing time: mean = -nan(ind) s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Execution time: mean = -nan(ind) s, total = 0.000 s Event stats: [2023-08-23 12:49:44,452 I 41356 40156] (gcs_server.exe) gcs_server.cc:255: GcsNodeManager: RegisterNode request count: 0 DrainNode request count: 0 GetAllNodeInfo request count: 0 GetInternalConfig request count: 0 GcsActorManager: RegisterActor request count: 0 CreateActor request count: 0 GetActorInfo request count: 0 GetNamedActorInfo request count: 0 GetAllActorInfo request count: 0 KillActor request count: 0 ListNamedActors request count: 0 Registered actors count: 0 Destroyed actors count: 0 Named actors count: 0 Unresolved actors count: 0 Pending actors count: 0 Created actors count: 0 owners_: 0 actor_to_register_callbacks_: 0 actor_to_create_callbacks_: 0 sorted_destroyed_actor_list_: 0 GcsResourceManager: GetResources request count: 0 GetAllAvailableResources request count0 ReportResourceUsage request count: 0 GetAllResourceUsage request count: 0 GcsPlacementGroupManager: CreatePlacementGroup request count: 0 RemovePlacementGroup request count: 0 GetPlacementGroup request count: 0 GetAllPlacementGroup request count: 0 WaitPlacementGroupUntilReady request count: 0 GetNamedPlacementGroup request count: 0 Scheduling pending placement group count: 0 Registered placement groups count: 0 Named placement group count: 0 Pending placement groups count: 0 Infeasible placement groups count: 0 GcsPublisher {} [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 0 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Total num bytes of task event stored: 0MiB -Current num of task events stored: 0 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 0 -Total num of driver tasks: 0 [2023-08-23 12:49:44,452 I 41356 40156] (gcs_server.exe) gcs_server.cc:872: Event stats: Global stats: 630 total (4 active) Queueing time: mean = 2.401 ms, max = 45.625 ms, min = 2.030 us, total = 1.512 s Execution time: mean = 125.330 us, total = 78.958 ms Event stats: GcsInMemoryStore.Put - 151 total (0 active), CPU time: mean = 313.577 us, total = 47.350 ms InternalKVGcsService.grpc_server.InternalKVPut - 148 total (0 active), CPU time: mean = 12.783 us, total = 1.892 ms InternalKVGcsService.grpc_client.InternalKVPut - 144 total (0 active), CPU time: mean = 6.774 us, total = 975.435 us RayletLoadPulled - 120 total (1 active), CPU time: mean = 4.320 us, total = 518.408 us UNKNOWN - 40 total (1 active), CPU time: mean = 6.561 us, total = 262.457 us GCSServer.deadline_timer.debug_state_dump - 12 total (1 active), CPU time: mean = 2.069 ms, total = 24.831 ms GcsInMemoryStore.GetAll - 6 total (0 active), CPU time: mean = 14.767 us, total = 88.600 us PeriodicalRunner.RunFnPeriodically - 4 total (0 active), CPU time: mean = 678.900 us, total = 2.716 ms GcsInMemoryStore.Get - 2 total (0 active), CPU time: mean = 14.127 us, total = 28.253 us GCSServer.deadline_timer.debug_state_event_stats_print - 2 total (1 active, 1 running), CPU time: mean = 140.755 us, total = 281.510 us InternalKVGcsService.grpc_server.InternalKVGet - 1 total (0 active), CPU time: mean = 14.884 us, total = 14.884 us [2023-08-23 12:49:44,452 I 41356 40156] (gcs_server.exe) gcs_server.cc:873: GcsTaskManager Event stats: Global stats: 0 total (0 active) Queueing time: mean = -nan(ind) s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Execution time: mean = -nan(ind) s, total = 0.000 s Event stats: [2023-08-23 12:50:44,453 I 41356 40156] (gcs_server.exe) gcs_server.cc:255: GcsNodeManager: RegisterNode request count: 0 DrainNode request count: 0 GetAllNodeInfo request count: 0 GetInternalConfig request count: 0 GcsActorManager: RegisterActor request count: 0 CreateActor request count: 0 GetActorInfo request count: 0 GetNamedActorInfo request count: 0 GetAllActorInfo request count: 0 KillActor request count: 0 ListNamedActors request count: 0 Registered actors count: 0 Destroyed actors count: 0 Named actors count: 0 Unresolved actors count: 0 Pending actors count: 0 Created actors count: 0 owners_: 0 actor_to_register_callbacks_: 0 actor_to_create_callbacks_: 0 sorted_destroyed_actor_list_: 0 GcsResourceManager: GetResources request count: 0 GetAllAvailableResources request count0 ReportResourceUsage request count: 0 GetAllResourceUsage request count: 0 GcsPlacementGroupManager: CreatePlacementGroup request count: 0 RemovePlacementGroup request count: 0 GetPlacementGroup request count: 0 GetAllPlacementGroup request count: 0 WaitPlacementGroupUntilReady request count: 0 GetNamedPlacementGroup request count: 0 Scheduling pending placement group count: 0 Registered placement groups count: 0 Named placement group count: 0 Pending placement groups count: 0 Infeasible placement groups count: 0 GcsPublisher {} [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 0 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Total num bytes of task event stored: 0MiB -Current num of task events stored: 0 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 0 -Total num of driver tasks: 0 [2023-08-23 12:50:44,453 I 41356 40156] (gcs_server.exe) gcs_server.cc:872: Event stats: Global stats: 932 total (4 active) Queueing time: mean = 2.255 ms, max = 45.625 ms, min = 2.030 us, total = 2.102 s Execution time: mean = 101.100 us, total = 94.225 ms Event stats: GcsInMemoryStore.Put - 223 total (0 active), CPU time: mean = 213.287 us, total = 47.563 ms InternalKVGcsService.grpc_server.InternalKVPut - 220 total (0 active), CPU time: mean = 12.331 us, total = 2.713 ms InternalKVGcsService.grpc_client.InternalKVPut - 216 total (0 active), CPU time: mean = 6.376 us, total = 1.377 ms RayletLoadPulled - 179 total (1 active), CPU time: mean = 4.219 us, total = 755.142 us UNKNOWN - 60 total (1 active), CPU time: mean = 6.083 us, total = 364.977 us GCSServer.deadline_timer.debug_state_dump - 18 total (1 active), CPU time: mean = 2.113 ms, total = 38.035 ms GcsInMemoryStore.GetAll - 6 total (0 active), CPU time: mean = 14.767 us, total = 88.600 us PeriodicalRunner.RunFnPeriodically - 4 total (0 active), CPU time: mean = 678.900 us, total = 2.716 ms GCSServer.deadline_timer.debug_state_event_stats_print - 3 total (1 active, 1 running), CPU time: mean = 189.673 us, total = 569.020 us GcsInMemoryStore.Get - 2 total (0 active), CPU time: mean = 14.127 us, total = 28.253 us InternalKVGcsService.grpc_server.InternalKVGet - 1 total (0 active), CPU time: mean = 14.884 us, total = 14.884 us [2023-08-23 12:50:44,453 I 41356 40156] (gcs_server.exe) gcs_server.cc:873: GcsTaskManager Event stats: Global stats: 0 total (0 active) Queueing time: mean = -nan(ind) s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Execution time: mean = -nan(ind) s, total = 0.000 s Event stats: [2023-08-23 12:51:44,460 I 41356 40156] (gcs_server.exe) gcs_server.cc:255: GcsNodeManager: RegisterNode request count: 0 DrainNode request count: 0 GetAllNodeInfo request count: 0 GetInternalConfig request count: 0 GcsActorManager: RegisterActor request count: 0 CreateActor request count: 0 GetActorInfo request count: 0 GetNamedActorInfo request count: 0 GetAllActorInfo request count: 0 KillActor request count: 0 ListNamedActors request count: 0 Registered actors count: 0 Destroyed actors count: 0 Named actors count: 0 Unresolved actors count: 0 Pending actors count: 0 Created actors count: 0 owners_: 0 actor_to_register_callbacks_: 0 actor_to_create_callbacks_: 0 sorted_destroyed_actor_list_: 0 GcsResourceManager: GetResources request count: 0 GetAllAvailableResources request count0 ReportResourceUsage request count: 0 GetAllResourceUsage request count: 0 GcsPlacementGroupManager: CreatePlacementGroup request count: 0 RemovePlacementGroup request count: 0 GetPlacementGroup request count: 0 GetAllPlacementGroup request count: 0 WaitPlacementGroupUntilReady request count: 0 GetNamedPlacementGroup request count: 0 Scheduling pending placement group count: 0 Registered placement groups count: 0 Named placement group count: 0 Pending placement groups count: 0 Infeasible placement groups count: 0 GcsPublisher {} [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 0 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Total num bytes of task event stored: 0MiB -Current num of task events stored: 0 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 0 -Total num of driver tasks: 0 [2023-08-23 12:51:44,460 I 41356 40156] (gcs_server.exe) gcs_server.cc:872: Event stats: Global stats: 1234 total (4 active) Queueing time: mean = 3.752 ms, max = 947.978 ms, min = 1.525 us, total = 4.631 s Execution time: mean = 89.193 us, total = 110.064 ms Event stats: GcsInMemoryStore.Put - 295 total (0 active), CPU time: mean = 162.026 us, total = 47.798 ms InternalKVGcsService.grpc_server.InternalKVPut - 292 total (0 active), CPU time: mean = 11.896 us, total = 3.474 ms InternalKVGcsService.grpc_client.InternalKVPut - 288 total (0 active), CPU time: mean = 6.330 us, total = 1.823 ms RayletLoadPulled - 238 total (1 active), CPU time: mean = 4.866 us, total = 1.158 ms UNKNOWN - 80 total (1 active), CPU time: mean = 6.067 us, total = 485.393 us GCSServer.deadline_timer.debug_state_dump - 24 total (1 active), CPU time: mean = 2.151 ms, total = 51.625 ms GcsInMemoryStore.GetAll - 6 total (0 active), CPU time: mean = 14.767 us, total = 88.600 us PeriodicalRunner.RunFnPeriodically - 4 total (0 active), CPU time: mean = 678.900 us, total = 2.716 ms GCSServer.deadline_timer.debug_state_event_stats_print - 4 total (1 active, 1 running), CPU time: mean = 213.333 us, total = 853.332 us GcsInMemoryStore.Get - 2 total (0 active), CPU time: mean = 14.127 us, total = 28.253 us InternalKVGcsService.grpc_server.InternalKVGet - 1 total (0 active), CPU time: mean = 14.884 us, total = 14.884 us [2023-08-23 12:51:44,460 I 41356 40156] (gcs_server.exe) gcs_server.cc:873: GcsTaskManager Event stats: Global stats: 0 total (0 active) Queueing time: mean = -nan(ind) s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Execution time: mean = -nan(ind) s, total = 0.000 s Event stats: [2023-08-23 12:52:44,469 I 41356 40156] (gcs_server.exe) gcs_server.cc:255: GcsNodeManager: RegisterNode request count: 0 DrainNode request count: 0 GetAllNodeInfo request count: 0 GetInternalConfig request count: 0 GcsActorManager: RegisterActor request count: 0 CreateActor request count: 0 GetActorInfo request count: 0 GetNamedActorInfo request count: 0 GetAllActorInfo request count: 0 KillActor request count: 0 ListNamedActors request count: 0 Registered actors count: 0 Destroyed actors count: 0 Named actors count: 0 Unresolved actors count: 0 Pending actors count: 0 Created actors count: 0 owners_: 0 actor_to_register_callbacks_: 0 actor_to_create_callbacks_: 0 sorted_destroyed_actor_list_: 0 GcsResourceManager: GetResources request count: 0 GetAllAvailableResources request count0 ReportResourceUsage request count: 0 GetAllResourceUsage request count: 0 GcsPlacementGroupManager: CreatePlacementGroup request count: 0 RemovePlacementGroup request count: 0 GetPlacementGroup request count: 0 GetAllPlacementGroup request count: 0 WaitPlacementGroupUntilReady request count: 0 GetNamedPlacementGroup request count: 0 Scheduling pending placement group count: 0 Registered placement groups count: 0 Named placement group count: 0 Pending placement groups count: 0 Infeasible placement groups count: 0 GcsPublisher {} [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 0 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Total num bytes of task event stored: 0MiB -Current num of task events stored: 0 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 0 -Total num of driver tasks: 0 [2023-08-23 12:52:44,469 I 41356 40156] (gcs_server.exe) gcs_server.cc:872: Event stats: Global stats: 1536 total (4 active) Queueing time: mean = 3.433 ms, max = 947.978 ms, min = 1.525 us, total = 5.273 s Execution time: mean = 81.996 us, total = 125.946 ms Event stats: GcsInMemoryStore.Put - 367 total (0 active), CPU time: mean = 130.799 us, total = 48.003 ms InternalKVGcsService.grpc_server.InternalKVPut - 364 total (0 active), CPU time: mean = 11.959 us, total = 4.353 ms InternalKVGcsService.grpc_client.InternalKVPut - 360 total (0 active), CPU time: mean = 6.328 us, total = 2.278 ms RayletLoadPulled - 297 total (1 active), CPU time: mean = 5.624 us, total = 1.670 ms UNKNOWN - 100 total (1 active), CPU time: mean = 6.063 us, total = 606.347 us GCSServer.deadline_timer.debug_state_dump - 30 total (1 active), CPU time: mean = 2.170 ms, total = 65.088 ms GcsInMemoryStore.GetAll - 6 total (0 active), CPU time: mean = 14.767 us, total = 88.600 us GCSServer.deadline_timer.debug_state_event_stats_print - 5 total (1 active, 1 running), CPU time: mean = 219.841 us, total = 1.099 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), CPU time: mean = 678.900 us, total = 2.716 ms GcsInMemoryStore.Get - 2 total (0 active), CPU time: mean = 14.127 us, total = 28.253 us InternalKVGcsService.grpc_server.InternalKVGet - 1 total (0 active), CPU time: mean = 14.884 us, total = 14.884 us [2023-08-23 12:52:44,469 I 41356 40156] (gcs_server.exe) gcs_server.cc:873: GcsTaskManager Event stats: Global stats: 0 total (0 active) Queueing time: mean = -nan(ind) s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Execution time: mean = -nan(ind) s, total = 0.000 s Event stats: [2023-08-23 12:53:44,471 I 41356 40156] (gcs_server.exe) gcs_server.cc:255: GcsNodeManager: RegisterNode request count: 0 DrainNode request count: 0 GetAllNodeInfo request count: 0 GetInternalConfig request count: 0 GcsActorManager: RegisterActor request count: 0 CreateActor request count: 0 GetActorInfo request count: 0 GetNamedActorInfo request count: 0 GetAllActorInfo request count: 0 KillActor request count: 0 ListNamedActors request count: 0 Registered actors count: 0 Destroyed actors count: 0 Named actors count: 0 Unresolved actors count: 0 Pending actors count: 0 Created actors count: 0 owners_: 0 actor_to_register_callbacks_: 0 actor_to_create_callbacks_: 0 sorted_destroyed_actor_list_: 0 GcsResourceManager: GetResources request count: 0 GetAllAvailableResources request count0 ReportResourceUsage request count: 0 GetAllResourceUsage request count: 0 GcsPlacementGroupManager: CreatePlacementGroup request count: 0 RemovePlacementGroup request count: 0 GetPlacementGroup request count: 0 GetAllPlacementGroup request count: 0 WaitPlacementGroupUntilReady request count: 0 GetNamedPlacementGroup request count: 0 Scheduling pending placement group count: 0 Registered placement groups count: 0 Named placement group count: 0 Pending placement groups count: 0 Infeasible placement groups count: 0 GcsPublisher {} [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 0 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Total num bytes of task event stored: 0MiB -Current num of task events stored: 0 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 0 -Total num of driver tasks: 0 [2023-08-23 12:53:44,471 I 41356 40156] (gcs_server.exe) gcs_server.cc:872: Event stats: Global stats: 1839 total (4 active) Queueing time: mean = 3.255 ms, max = 947.978 ms, min = -0.000 s, total = 5.986 s Execution time: mean = 77.704 us, total = 142.897 ms Event stats: GcsInMemoryStore.Put - 439 total (0 active), CPU time: mean = 109.881 us, total = 48.238 ms InternalKVGcsService.grpc_server.InternalKVPut - 436 total (0 active), CPU time: mean = 12.038 us, total = 5.249 ms InternalKVGcsService.grpc_client.InternalKVPut - 432 total (0 active), CPU time: mean = 6.283 us, total = 2.714 ms RayletLoadPulled - 357 total (1 active), CPU time: mean = 5.926 us, total = 2.115 ms UNKNOWN - 120 total (1 active), CPU time: mean = 6.057 us, total = 726.807 us GCSServer.deadline_timer.debug_state_dump - 36 total (1 active), CPU time: mean = 2.209 ms, total = 79.537 ms GcsInMemoryStore.GetAll - 6 total (0 active), CPU time: mean = 14.767 us, total = 88.600 us GCSServer.deadline_timer.debug_state_event_stats_print - 6 total (1 active, 1 running), CPU time: mean = 244.923 us, total = 1.470 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), CPU time: mean = 678.900 us, total = 2.716 ms GcsInMemoryStore.Get - 2 total (0 active), CPU time: mean = 14.127 us, total = 28.253 us InternalKVGcsService.grpc_server.InternalKVGet - 1 total (0 active), CPU time: mean = 14.884 us, total = 14.884 us [2023-08-23 12:53:44,471 I 41356 40156] (gcs_server.exe) gcs_server.cc:873: GcsTaskManager Event stats: Global stats: 0 total (0 active) Queueing time: mean = -nan(ind) s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Execution time: mean = -nan(ind) s, total = 0.000 s Event stats: [2023-08-23 12:54:44,480 I 41356 40156] (gcs_server.exe) gcs_server.cc:255: GcsNodeManager: RegisterNode request count: 0 DrainNode request count: 0 GetAllNodeInfo request count: 0 GetInternalConfig request count: 0 GcsActorManager: RegisterActor request count: 0 CreateActor request count: 0 GetActorInfo request count: 0 GetNamedActorInfo request count: 0 GetAllActorInfo request count: 0 KillActor request count: 0 ListNamedActors request count: 0 Registered actors count: 0 Destroyed actors count: 0 Named actors count: 0 Unresolved actors count: 0 Pending actors count: 0 Created actors count: 0 owners_: 0 actor_to_register_callbacks_: 0 actor_to_create_callbacks_: 0 sorted_destroyed_actor_list_: 0 GcsResourceManager: GetResources request count: 0 GetAllAvailableResources request count0 ReportResourceUsage request count: 0 GetAllResourceUsage request count: 0 GcsPlacementGroupManager: CreatePlacementGroup request count: 0 RemovePlacementGroup request count: 0 GetPlacementGroup request count: 0 GetAllPlacementGroup request count: 0 WaitPlacementGroupUntilReady request count: 0 GetNamedPlacementGroup request count: 0 Scheduling pending placement group count: 0 Registered placement groups count: 0 Named placement group count: 0 Pending placement groups count: 0 Infeasible placement groups count: 0 GcsPublisher {} [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 0 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Total num bytes of task event stored: 0MiB -Current num of task events stored: 0 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 0 -Total num of driver tasks: 0 [2023-08-23 12:54:44,480 I 41356 40156] (gcs_server.exe) gcs_server.cc:872: Event stats: Global stats: 2141 total (4 active) Queueing time: mean = 3.438 ms, max = 947.978 ms, min = -0.000 s, total = 7.360 s Execution time: mean = 74.130 us, total = 158.713 ms Event stats: GcsInMemoryStore.Put - 511 total (0 active), CPU time: mean = 94.979 us, total = 48.534 ms InternalKVGcsService.grpc_server.InternalKVPut - 508 total (0 active), CPU time: mean = 12.276 us, total = 6.236 ms InternalKVGcsService.grpc_client.InternalKVPut - 504 total (0 active), CPU time: mean = 6.236 us, total = 3.143 ms RayletLoadPulled - 416 total (1 active), CPU time: mean = 6.363 us, total = 2.647 ms UNKNOWN - 140 total (1 active), CPU time: mean = 6.114 us, total = 856.024 us GCSServer.deadline_timer.debug_state_dump - 42 total (1 active), CPU time: mean = 2.208 ms, total = 92.735 ms GCSServer.deadline_timer.debug_state_event_stats_print - 7 total (1 active, 1 running), CPU time: mean = 245.003 us, total = 1.715 ms GcsInMemoryStore.GetAll - 6 total (0 active), CPU time: mean = 14.767 us, total = 88.600 us PeriodicalRunner.RunFnPeriodically - 4 total (0 active), CPU time: mean = 678.900 us, total = 2.716 ms GcsInMemoryStore.Get - 2 total (0 active), CPU time: mean = 14.127 us, total = 28.253 us InternalKVGcsService.grpc_server.InternalKVGet - 1 total (0 active), CPU time: mean = 14.884 us, total = 14.884 us [2023-08-23 12:54:44,480 I 41356 40156] (gcs_server.exe) gcs_server.cc:873: GcsTaskManager Event stats: Global stats: 0 total (0 active) Queueing time: mean = -nan(ind) s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Execution time: mean = -nan(ind) s, total = 0.000 s Event stats: [2023-08-23 12:55:44,481 I 41356 40156] (gcs_server.exe) gcs_server.cc:255: GcsNodeManager: RegisterNode request count: 0 DrainNode request count: 0 GetAllNodeInfo request count: 0 GetInternalConfig request count: 0 GcsActorManager: RegisterActor request count: 0 CreateActor request count: 0 GetActorInfo request count: 0 GetNamedActorInfo request count: 0 GetAllActorInfo request count: 0 KillActor request count: 0 ListNamedActors request count: 0 Registered actors count: 0 Destroyed actors count: 0 Named actors count: 0 Unresolved actors count: 0 Pending actors count: 0 Created actors count: 0 owners_: 0 actor_to_register_callbacks_: 0 actor_to_create_callbacks_: 0 sorted_destroyed_actor_list_: 0 GcsResourceManager: GetResources request count: 0 GetAllAvailableResources request count0 ReportResourceUsage request count: 0 GetAllResourceUsage request count: 0 GcsPlacementGroupManager: CreatePlacementGroup request count: 0 RemovePlacementGroup request count: 0 GetPlacementGroup request count: 0 GetAllPlacementGroup request count: 0 WaitPlacementGroupUntilReady request count: 0 GetNamedPlacementGroup request count: 0 Scheduling pending placement group count: 0 Registered placement groups count: 0 Named placement group count: 0 Pending placement groups count: 0 Infeasible placement groups count: 0 GcsPublisher {} [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 0 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Total num bytes of task event stored: 0MiB -Current num of task events stored: 0 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 0 -Total num of driver tasks: 0 [2023-08-23 12:55:44,481 I 41356 40156] (gcs_server.exe) gcs_server.cc:872: Event stats: Global stats: 2442 total (4 active) Queueing time: mean = 3.878 ms, max = 947.978 ms, min = -0.000 s, total = 9.469 s Execution time: mean = 71.971 us, total = 175.753 ms Event stats: GcsInMemoryStore.Put - 583 total (0 active), CPU time: mean = 83.684 us, total = 48.788 ms InternalKVGcsService.grpc_server.InternalKVPut - 580 total (0 active), CPU time: mean = 12.577 us, total = 7.295 ms InternalKVGcsService.grpc_client.InternalKVPut - 576 total (0 active), CPU time: mean = 6.164 us, total = 3.551 ms RayletLoadPulled - 475 total (1 active), CPU time: mean = 6.485 us, total = 3.081 ms UNKNOWN - 159 total (1 active), CPU time: mean = 6.154 us, total = 978.420 us GCSServer.deadline_timer.debug_state_dump - 48 total (1 active), CPU time: mean = 2.232 ms, total = 107.138 ms GCSServer.deadline_timer.debug_state_event_stats_print - 8 total (1 active, 1 running), CPU time: mean = 259.461 us, total = 2.076 ms GcsInMemoryStore.GetAll - 6 total (0 active), CPU time: mean = 14.767 us, total = 88.600 us PeriodicalRunner.RunFnPeriodically - 4 total (0 active), CPU time: mean = 678.900 us, total = 2.716 ms GcsInMemoryStore.Get - 2 total (0 active), CPU time: mean = 14.127 us, total = 28.253 us InternalKVGcsService.grpc_server.InternalKVGet - 1 total (0 active), CPU time: mean = 14.884 us, total = 14.884 us [2023-08-23 12:55:44,481 I 41356 40156] (gcs_server.exe) gcs_server.cc:873: GcsTaskManager Event stats: Global stats: 0 total (0 active) Queueing time: mean = -nan(ind) s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Execution time: mean = -nan(ind) s, total = 0.000 s Event stats: [2023-08-23 12:56:44,487 I 41356 40156] (gcs_server.exe) gcs_server.cc:255: GcsNodeManager: RegisterNode request count: 0 DrainNode request count: 0 GetAllNodeInfo request count: 0 GetInternalConfig request count: 0 GcsActorManager: RegisterActor request count: 0 CreateActor request count: 0 GetActorInfo request count: 0 GetNamedActorInfo request count: 0 GetAllActorInfo request count: 0 KillActor request count: 0 ListNamedActors request count: 0 Registered actors count: 0 Destroyed actors count: 0 Named actors count: 0 Unresolved actors count: 0 Pending actors count: 0 Created actors count: 0 owners_: 0 actor_to_register_callbacks_: 0 actor_to_create_callbacks_: 0 sorted_destroyed_actor_list_: 0 GcsResourceManager: GetResources request count: 0 GetAllAvailableResources request count0 ReportResourceUsage request count: 0 GetAllResourceUsage request count: 0 GcsPlacementGroupManager: CreatePlacementGroup request count: 0 RemovePlacementGroup request count: 0 GetPlacementGroup request count: 0 GetAllPlacementGroup request count: 0 WaitPlacementGroupUntilReady request count: 0 GetNamedPlacementGroup request count: 0 Scheduling pending placement group count: 0 Registered placement groups count: 0 Named placement group count: 0 Pending placement groups count: 0 Infeasible placement groups count: 0 GcsPublisher {} [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 0 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Total num bytes of task event stored: 0MiB -Current num of task events stored: 0 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 0 -Total num of driver tasks: 0 [2023-08-23 12:56:44,487 I 41356 40156] (gcs_server.exe) gcs_server.cc:872: Event stats: Global stats: 2744 total (4 active) Queueing time: mean = 3.676 ms, max = 947.978 ms, min = -0.000 s, total = 10.087 s Execution time: mean = 70.326 us, total = 192.975 ms Event stats: GcsInMemoryStore.Put - 655 total (0 active), CPU time: mean = 74.797 us, total = 48.992 ms InternalKVGcsService.grpc_server.InternalKVPut - 652 total (0 active), CPU time: mean = 12.812 us, total = 8.353 ms InternalKVGcsService.grpc_client.InternalKVPut - 648 total (0 active), CPU time: mean = 6.124 us, total = 3.969 ms RayletLoadPulled - 534 total (1 active), CPU time: mean = 6.752 us, total = 3.606 ms UNKNOWN - 179 total (1 active), CPU time: mean = 5.946 us, total = 1.064 ms GCSServer.deadline_timer.debug_state_dump - 54 total (1 active), CPU time: mean = 2.254 ms, total = 121.707 ms GCSServer.deadline_timer.debug_state_event_stats_print - 9 total (1 active, 1 running), CPU time: mean = 270.683 us, total = 2.436 ms GcsInMemoryStore.GetAll - 6 total (0 active), CPU time: mean = 14.767 us, total = 88.600 us PeriodicalRunner.RunFnPeriodically - 4 total (0 active), CPU time: mean = 678.900 us, total = 2.716 ms GcsInMemoryStore.Get - 2 total (0 active), CPU time: mean = 14.127 us, total = 28.253 us InternalKVGcsService.grpc_server.InternalKVGet - 1 total (0 active), CPU time: mean = 14.884 us, total = 14.884 us [2023-08-23 12:56:44,487 I 41356 40156] (gcs_server.exe) gcs_server.cc:873: GcsTaskManager Event stats: Global stats: 0 total (0 active) Queueing time: mean = -nan(ind) s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Execution time: mean = -nan(ind) s, total = 0.000 s Event stats: [2023-08-23 12:57:44,494 I 41356 40156] (gcs_server.exe) gcs_server.cc:255: GcsNodeManager: RegisterNode request count: 0 DrainNode request count: 0 GetAllNodeInfo request count: 0 GetInternalConfig request count: 0 GcsActorManager: RegisterActor request count: 0 CreateActor request count: 0 GetActorInfo request count: 0 GetNamedActorInfo request count: 0 GetAllActorInfo request count: 0 KillActor request count: 0 ListNamedActors request count: 0 Registered actors count: 0 Destroyed actors count: 0 Named actors count: 0 Unresolved actors count: 0 Pending actors count: 0 Created actors count: 0 owners_: 0 actor_to_register_callbacks_: 0 actor_to_create_callbacks_: 0 sorted_destroyed_actor_list_: 0 GcsResourceManager: GetResources request count: 0 GetAllAvailableResources request count0 ReportResourceUsage request count: 0 GetAllResourceUsage request count: 0 GcsPlacementGroupManager: CreatePlacementGroup request count: 0 RemovePlacementGroup request count: 0 GetPlacementGroup request count: 0 GetAllPlacementGroup request count: 0 WaitPlacementGroupUntilReady request count: 0 GetNamedPlacementGroup request count: 0 Scheduling pending placement group count: 0 Registered placement groups count: 0 Named placement group count: 0 Pending placement groups count: 0 Infeasible placement groups count: 0 GcsPublisher {} [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 0 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Total num bytes of task event stored: 0MiB -Current num of task events stored: 0 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 0 -Total num of driver tasks: 0 [2023-08-23 12:57:44,494 I 41356 40156] (gcs_server.exe) gcs_server.cc:872: Event stats: Global stats: 3047 total (4 active) Queueing time: mean = 3.504 ms, max = 947.978 ms, min = -0.000 s, total = 10.677 s Execution time: mean = 68.486 us, total = 208.677 ms Event stats: GcsInMemoryStore.Put - 727 total (0 active), CPU time: mean = 67.757 us, total = 49.259 ms InternalKVGcsService.grpc_server.InternalKVPut - 724 total (0 active), CPU time: mean = 12.601 us, total = 9.123 ms InternalKVGcsService.grpc_client.InternalKVPut - 720 total (0 active), CPU time: mean = 6.199 us, total = 4.463 ms RayletLoadPulled - 594 total (1 active), CPU time: mean = 7.002 us, total = 4.159 ms UNKNOWN - 199 total (1 active), CPU time: mean = 5.849 us, total = 1.164 ms GCSServer.deadline_timer.debug_state_dump - 60 total (1 active), CPU time: mean = 2.249 ms, total = 134.948 ms GCSServer.deadline_timer.debug_state_event_stats_print - 10 total (1 active, 1 running), CPU time: mean = 271.241 us, total = 2.712 ms GcsInMemoryStore.GetAll - 6 total (0 active), CPU time: mean = 14.767 us, total = 88.600 us PeriodicalRunner.RunFnPeriodically - 4 total (0 active), CPU time: mean = 678.900 us, total = 2.716 ms GcsInMemoryStore.Get - 2 total (0 active), CPU time: mean = 14.127 us, total = 28.253 us InternalKVGcsService.grpc_server.InternalKVGet - 1 total (0 active), CPU time: mean = 14.884 us, total = 14.884 us [2023-08-23 12:57:44,494 I 41356 40156] (gcs_server.exe) gcs_server.cc:873: GcsTaskManager Event stats: Global stats: 0 total (0 active) Queueing time: mean = -nan(ind) s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Execution time: mean = -nan(ind) s, total = 0.000 s Event stats: [2023-08-23 12:58:44,506 I 41356 40156] (gcs_server.exe) gcs_server.cc:255: GcsNodeManager: RegisterNode request count: 0 DrainNode request count: 0 GetAllNodeInfo request count: 0 GetInternalConfig request count: 0 GcsActorManager: RegisterActor request count: 0 CreateActor request count: 0 GetActorInfo request count: 0 GetNamedActorInfo request count: 0 GetAllActorInfo request count: 0 KillActor request count: 0 ListNamedActors request count: 0 Registered actors count: 0 Destroyed actors count: 0 Named actors count: 0 Unresolved actors count: 0 Pending actors count: 0 Created actors count: 0 owners_: 0 actor_to_register_callbacks_: 0 actor_to_create_callbacks_: 0 sorted_destroyed_actor_list_: 0 GcsResourceManager: GetResources request count: 0 GetAllAvailableResources request count0 ReportResourceUsage request count: 0 GetAllResourceUsage request count: 0 GcsPlacementGroupManager: CreatePlacementGroup request count: 0 RemovePlacementGroup request count: 0 GetPlacementGroup request count: 0 GetAllPlacementGroup request count: 0 WaitPlacementGroupUntilReady request count: 0 GetNamedPlacementGroup request count: 0 Scheduling pending placement group count: 0 Registered placement groups count: 0 Named placement group count: 0 Pending placement groups count: 0 Infeasible placement groups count: 0 GcsPublisher {} [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 0 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Total num bytes of task event stored: 0MiB -Current num of task events stored: 0 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 0 -Total num of driver tasks: 0 [2023-08-23 12:58:44,507 I 41356 40156] (gcs_server.exe) gcs_server.cc:872: Event stats: Global stats: 3350 total (4 active) Queueing time: mean = 3.393 ms, max = 947.978 ms, min = -0.000 s, total = 11.368 s Execution time: mean = 67.045 us, total = 224.602 ms Event stats: GcsInMemoryStore.Put - 799 total (0 active), CPU time: mean = 61.880 us, total = 49.442 ms InternalKVGcsService.grpc_server.InternalKVPut - 796 total (0 active), CPU time: mean = 12.332 us, total = 9.816 ms InternalKVGcsService.grpc_client.InternalKVPut - 792 total (0 active), CPU time: mean = 6.123 us, total = 4.850 ms RayletLoadPulled - 654 total (1 active), CPU time: mean = 7.114 us, total = 4.652 ms UNKNOWN - 219 total (1 active), CPU time: mean = 5.949 us, total = 1.303 ms GCSServer.deadline_timer.debug_state_dump - 66 total (1 active), CPU time: mean = 2.252 ms, total = 148.634 ms GCSServer.deadline_timer.debug_state_event_stats_print - 11 total (1 active, 1 running), CPU time: mean = 278.006 us, total = 3.058 ms GcsInMemoryStore.GetAll - 6 total (0 active), CPU time: mean = 14.767 us, total = 88.600 us PeriodicalRunner.RunFnPeriodically - 4 total (0 active), CPU time: mean = 678.900 us, total = 2.716 ms GcsInMemoryStore.Get - 2 total (0 active), CPU time: mean = 14.127 us, total = 28.253 us InternalKVGcsService.grpc_server.InternalKVGet - 1 total (0 active), CPU time: mean = 14.884 us, total = 14.884 us [2023-08-23 12:58:44,507 I 41356 40156] (gcs_server.exe) gcs_server.cc:873: GcsTaskManager Event stats: Global stats: 0 total (0 active) Queueing time: mean = -nan(ind) s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Execution time: mean = -nan(ind) s, total = 0.000 s Event stats: [2023-08-23 12:59:44,520 I 41356 40156] (gcs_server.exe) gcs_server.cc:255: GcsNodeManager: RegisterNode request count: 0 DrainNode request count: 0 GetAllNodeInfo request count: 0 GetInternalConfig request count: 0 GcsActorManager: RegisterActor request count: 0 CreateActor request count: 0 GetActorInfo request count: 0 GetNamedActorInfo request count: 0 GetAllActorInfo request count: 0 KillActor request count: 0 ListNamedActors request count: 0 Registered actors count: 0 Destroyed actors count: 0 Named actors count: 0 Unresolved actors count: 0 Pending actors count: 0 Created actors count: 0 owners_: 0 actor_to_register_callbacks_: 0 actor_to_create_callbacks_: 0 sorted_destroyed_actor_list_: 0 GcsResourceManager: GetResources request count: 0 GetAllAvailableResources request count0 ReportResourceUsage request count: 0 GetAllResourceUsage request count: 0 GcsPlacementGroupManager: CreatePlacementGroup request count: 0 RemovePlacementGroup request count: 0 GetPlacementGroup request count: 0 GetAllPlacementGroup request count: 0 WaitPlacementGroupUntilReady request count: 0 GetNamedPlacementGroup request count: 0 Scheduling pending placement group count: 0 Registered placement groups count: 0 Named placement group count: 0 Pending placement groups count: 0 Infeasible placement groups count: 0 GcsPublisher {} [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 0 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Total num bytes of task event stored: 0MiB -Current num of task events stored: 0 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 0 -Total num of driver tasks: 0 [2023-08-23 12:59:44,520 I 41356 40156] (gcs_server.exe) gcs_server.cc:872: Event stats: Global stats: 3652 total (4 active) Queueing time: mean = 3.297 ms, max = 947.978 ms, min = -0.000 s, total = 12.042 s Execution time: mean = 66.080 us, total = 241.323 ms Event stats: GcsInMemoryStore.Put - 871 total (0 active), CPU time: mean = 57.015 us, total = 49.660 ms InternalKVGcsService.grpc_server.InternalKVPut - 868 total (0 active), CPU time: mean = 12.301 us, total = 10.677 ms InternalKVGcsService.grpc_client.InternalKVPut - 864 total (0 active), CPU time: mean = 6.180 us, total = 5.340 ms RayletLoadPulled - 713 total (1 active), CPU time: mean = 7.296 us, total = 5.202 ms UNKNOWN - 239 total (1 active), CPU time: mean = 5.855 us, total = 1.399 ms GCSServer.deadline_timer.debug_state_dump - 72 total (1 active), CPU time: mean = 2.262 ms, total = 162.888 ms GCSServer.deadline_timer.debug_state_event_stats_print - 12 total (1 active, 1 running), CPU time: mean = 275.748 us, total = 3.309 ms GcsInMemoryStore.GetAll - 6 total (0 active), CPU time: mean = 14.767 us, total = 88.600 us PeriodicalRunner.RunFnPeriodically - 4 total (0 active), CPU time: mean = 678.900 us, total = 2.716 ms GcsInMemoryStore.Get - 2 total (0 active), CPU time: mean = 14.127 us, total = 28.253 us InternalKVGcsService.grpc_server.InternalKVGet - 1 total (0 active), CPU time: mean = 14.884 us, total = 14.884 us [2023-08-23 12:59:44,520 I 41356 40156] (gcs_server.exe) gcs_server.cc:873: GcsTaskManager Event stats: Global stats: 0 total (0 active) Queueing time: mean = -nan(ind) s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Execution time: mean = -nan(ind) s, total = 0.000 s Event stats: [2023-08-23 13:00:44,535 I 41356 40156] (gcs_server.exe) gcs_server.cc:255: GcsNodeManager: RegisterNode request count: 0 DrainNode request count: 0 GetAllNodeInfo request count: 0 GetInternalConfig request count: 0 GcsActorManager: RegisterActor request count: 0 CreateActor request count: 0 GetActorInfo request count: 0 GetNamedActorInfo request count: 0 GetAllActorInfo request count: 0 KillActor request count: 0 ListNamedActors request count: 0 Registered actors count: 0 Destroyed actors count: 0 Named actors count: 0 Unresolved actors count: 0 Pending actors count: 0 Created actors count: 0 owners_: 0 actor_to_register_callbacks_: 0 actor_to_create_callbacks_: 0 sorted_destroyed_actor_list_: 0 GcsResourceManager: GetResources request count: 0 GetAllAvailableResources request count0 ReportResourceUsage request count: 0 GetAllResourceUsage request count: 0 GcsPlacementGroupManager: CreatePlacementGroup request count: 0 RemovePlacementGroup request count: 0 GetPlacementGroup request count: 0 GetAllPlacementGroup request count: 0 WaitPlacementGroupUntilReady request count: 0 GetNamedPlacementGroup request count: 0 Scheduling pending placement group count: 0 Registered placement groups count: 0 Named placement group count: 0 Pending placement groups count: 0 Infeasible placement groups count: 0 GcsPublisher {} [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 0 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Total num bytes of task event stored: 0MiB -Current num of task events stored: 0 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 0 -Total num of driver tasks: 0 [2023-08-23 13:00:44,535 I 41356 40156] (gcs_server.exe) gcs_server.cc:872: Event stats: Global stats: 3955 total (4 active) Queueing time: mean = 3.211 ms, max = 947.978 ms, min = -0.000 s, total = 12.700 s Execution time: mean = 65.336 us, total = 258.405 ms Event stats: GcsInMemoryStore.Put - 943 total (0 active), CPU time: mean = 52.896 us, total = 49.881 ms InternalKVGcsService.grpc_server.InternalKVPut - 940 total (0 active), CPU time: mean = 12.316 us, total = 11.577 ms InternalKVGcsService.grpc_client.InternalKVPut - 936 total (0 active), CPU time: mean = 6.166 us, total = 5.771 ms RayletLoadPulled - 773 total (1 active), CPU time: mean = 7.344 us, total = 5.677 ms UNKNOWN - 259 total (1 active), CPU time: mean = 5.746 us, total = 1.488 ms GCSServer.deadline_timer.debug_state_dump - 78 total (1 active), CPU time: mean = 2.275 ms, total = 177.477 ms GCSServer.deadline_timer.debug_state_event_stats_print - 13 total (1 active, 1 running), CPU time: mean = 283.577 us, total = 3.687 ms GcsInMemoryStore.GetAll - 6 total (0 active), CPU time: mean = 14.767 us, total = 88.600 us PeriodicalRunner.RunFnPeriodically - 4 total (0 active), CPU time: mean = 678.900 us, total = 2.716 ms GcsInMemoryStore.Get - 2 total (0 active), CPU time: mean = 14.127 us, total = 28.253 us InternalKVGcsService.grpc_server.InternalKVGet - 1 total (0 active), CPU time: mean = 14.884 us, total = 14.884 us [2023-08-23 13:00:44,535 I 41356 40156] (gcs_server.exe) gcs_server.cc:873: GcsTaskManager Event stats: Global stats: 0 total (0 active) Queueing time: mean = -nan(ind) s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Execution time: mean = -nan(ind) s, total = 0.000 s Event stats: [2023-08-23 13:01:44,537 I 41356 40156] (gcs_server.exe) gcs_server.cc:255: GcsNodeManager: RegisterNode request count: 0 DrainNode request count: 0 GetAllNodeInfo request count: 0 GetInternalConfig request count: 0 GcsActorManager: RegisterActor request count: 0 CreateActor request count: 0 GetActorInfo request count: 0 GetNamedActorInfo request count: 0 GetAllActorInfo request count: 0 KillActor request count: 0 ListNamedActors request count: 0 Registered actors count: 0 Destroyed actors count: 0 Named actors count: 0 Unresolved actors count: 0 Pending actors count: 0 Created actors count: 0 owners_: 0 actor_to_register_callbacks_: 0 actor_to_create_callbacks_: 0 sorted_destroyed_actor_list_: 0 GcsResourceManager: GetResources request count: 0 GetAllAvailableResources request count0 ReportResourceUsage request count: 0 GetAllResourceUsage request count: 0 GcsPlacementGroupManager: CreatePlacementGroup request count: 0 RemovePlacementGroup request count: 0 GetPlacementGroup request count: 0 GetAllPlacementGroup request count: 0 WaitPlacementGroupUntilReady request count: 0 GetNamedPlacementGroup request count: 0 Scheduling pending placement group count: 0 Registered placement groups count: 0 Named placement group count: 0 Pending placement groups count: 0 Infeasible placement groups count: 0 GcsPublisher {} [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 0 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Total num bytes of task event stored: 0MiB -Current num of task events stored: 0 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 0 -Total num of driver tasks: 0 [2023-08-23 13:01:44,537 I 41356 40156] (gcs_server.exe) gcs_server.cc:872: Event stats: Global stats: 4257 total (4 active) Queueing time: mean = 3.137 ms, max = 947.978 ms, min = -0.000 s, total = 13.353 s Execution time: mean = 64.408 us, total = 274.186 ms Event stats: GcsInMemoryStore.Put - 1015 total (0 active), CPU time: mean = 49.350 us, total = 50.090 ms InternalKVGcsService.grpc_server.InternalKVPut - 1012 total (0 active), CPU time: mean = 12.318 us, total = 12.466 ms InternalKVGcsService.grpc_client.InternalKVPut - 1008 total (0 active), CPU time: mean = 6.144 us, total = 6.194 ms RayletLoadPulled - 832 total (1 active), CPU time: mean = 7.333 us, total = 6.101 ms UNKNOWN - 279 total (1 active), CPU time: mean = 5.758 us, total = 1.606 ms GCSServer.deadline_timer.debug_state_dump - 84 total (1 active), CPU time: mean = 2.270 ms, total = 190.686 ms GCSServer.deadline_timer.debug_state_event_stats_print - 14 total (1 active, 1 running), CPU time: mean = 299.661 us, total = 4.195 ms GcsInMemoryStore.GetAll - 6 total (0 active), CPU time: mean = 14.767 us, total = 88.600 us PeriodicalRunner.RunFnPeriodically - 4 total (0 active), CPU time: mean = 678.900 us, total = 2.716 ms GcsInMemoryStore.Get - 2 total (0 active), CPU time: mean = 14.127 us, total = 28.253 us InternalKVGcsService.grpc_server.InternalKVGet - 1 total (0 active), CPU time: mean = 14.884 us, total = 14.884 us [2023-08-23 13:01:44,537 I 41356 40156] (gcs_server.exe) gcs_server.cc:873: GcsTaskManager Event stats: Global stats: 0 total (0 active) Queueing time: mean = -nan(ind) s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Execution time: mean = -nan(ind) s, total = 0.000 s Event stats: [2023-08-23 13:02:44,553 I 41356 40156] (gcs_server.exe) gcs_server.cc:255: GcsNodeManager: RegisterNode request count: 0 DrainNode request count: 0 GetAllNodeInfo request count: 0 GetInternalConfig request count: 0 GcsActorManager: RegisterActor request count: 0 CreateActor request count: 0 GetActorInfo request count: 0 GetNamedActorInfo request count: 0 GetAllActorInfo request count: 0 KillActor request count: 0 ListNamedActors request count: 0 Registered actors count: 0 Destroyed actors count: 0 Named actors count: 0 Unresolved actors count: 0 Pending actors count: 0 Created actors count: 0 owners_: 0 actor_to_register_callbacks_: 0 actor_to_create_callbacks_: 0 sorted_destroyed_actor_list_: 0 GcsResourceManager: GetResources request count: 0 GetAllAvailableResources request count0 ReportResourceUsage request count: 0 GetAllResourceUsage request count: 0 GcsPlacementGroupManager: CreatePlacementGroup request count: 0 RemovePlacementGroup request count: 0 GetPlacementGroup request count: 0 GetAllPlacementGroup request count: 0 WaitPlacementGroupUntilReady request count: 0 GetNamedPlacementGroup request count: 0 Scheduling pending placement group count: 0 Registered placement groups count: 0 Named placement group count: 0 Pending placement groups count: 0 Infeasible placement groups count: 0 GcsPublisher {} [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 0 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Total num bytes of task event stored: 0MiB -Current num of task events stored: 0 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 0 -Total num of driver tasks: 0 [2023-08-23 13:02:44,553 I 41356 40156] (gcs_server.exe) gcs_server.cc:872: Event stats: Global stats: 4560 total (4 active) Queueing time: mean = 3.076 ms, max = 947.978 ms, min = -0.000 s, total = 14.025 s Execution time: mean = 63.817 us, total = 291.008 ms Event stats: GcsInMemoryStore.Put - 1087 total (0 active), CPU time: mean = 46.262 us, total = 50.287 ms InternalKVGcsService.grpc_server.InternalKVPut - 1084 total (0 active), CPU time: mean = 12.337 us, total = 13.374 ms InternalKVGcsService.grpc_client.InternalKVPut - 1080 total (0 active), CPU time: mean = 6.116 us, total = 6.605 ms RayletLoadPulled - 892 total (1 active), CPU time: mean = 7.261 us, total = 6.477 ms UNKNOWN - 299 total (1 active), CPU time: mean = 5.696 us, total = 1.703 ms GCSServer.deadline_timer.debug_state_dump - 90 total (1 active), CPU time: mean = 2.280 ms, total = 205.184 ms GCSServer.deadline_timer.debug_state_event_stats_print - 15 total (1 active, 1 running), CPU time: mean = 302.035 us, total = 4.531 ms GcsInMemoryStore.GetAll - 6 total (0 active), CPU time: mean = 14.767 us, total = 88.600 us PeriodicalRunner.RunFnPeriodically - 4 total (0 active), CPU time: mean = 678.900 us, total = 2.716 ms GcsInMemoryStore.Get - 2 total (0 active), CPU time: mean = 14.127 us, total = 28.253 us InternalKVGcsService.grpc_server.InternalKVGet - 1 total (0 active), CPU time: mean = 14.884 us, total = 14.884 us [2023-08-23 13:02:44,553 I 41356 40156] (gcs_server.exe) gcs_server.cc:873: GcsTaskManager Event stats: Global stats: 0 total (0 active) Queueing time: mean = -nan(ind) s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Execution time: mean = -nan(ind) s, total = 0.000 s Event stats: [2023-08-23 13:03:44,559 I 41356 40156] (gcs_server.exe) gcs_server.cc:255: GcsNodeManager: RegisterNode request count: 0 DrainNode request count: 0 GetAllNodeInfo request count: 0 GetInternalConfig request count: 0 GcsActorManager: RegisterActor request count: 0 CreateActor request count: 0 GetActorInfo request count: 0 GetNamedActorInfo request count: 0 GetAllActorInfo request count: 0 KillActor request count: 0 ListNamedActors request count: 0 Registered actors count: 0 Destroyed actors count: 0 Named actors count: 0 Unresolved actors count: 0 Pending actors count: 0 Created actors count: 0 owners_: 0 actor_to_register_callbacks_: 0 actor_to_create_callbacks_: 0 sorted_destroyed_actor_list_: 0 GcsResourceManager: GetResources request count: 0 GetAllAvailableResources request count0 ReportResourceUsage request count: 0 GetAllResourceUsage request count: 0 GcsPlacementGroupManager: CreatePlacementGroup request count: 0 RemovePlacementGroup request count: 0 GetPlacementGroup request count: 0 GetAllPlacementGroup request count: 0 WaitPlacementGroupUntilReady request count: 0 GetNamedPlacementGroup request count: 0 Scheduling pending placement group count: 0 Registered placement groups count: 0 Named placement group count: 0 Pending placement groups count: 0 Infeasible placement groups count: 0 GcsPublisher {} [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 0 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Total num bytes of task event stored: 0MiB -Current num of task events stored: 0 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 0 -Total num of driver tasks: 0 [2023-08-23 13:03:44,560 I 41356 40156] (gcs_server.exe) gcs_server.cc:872: Event stats: Global stats: 4862 total (4 active) Queueing time: mean = 3.027 ms, max = 947.978 ms, min = -0.000 s, total = 14.715 s Execution time: mean = 63.140 us, total = 306.987 ms Event stats: GcsInMemoryStore.Put - 1159 total (0 active), CPU time: mean = 43.564 us, total = 50.490 ms InternalKVGcsService.grpc_server.InternalKVPut - 1156 total (0 active), CPU time: mean = 12.455 us, total = 14.398 ms InternalKVGcsService.grpc_client.InternalKVPut - 1152 total (0 active), CPU time: mean = 6.098 us, total = 7.025 ms RayletLoadPulled - 951 total (1 active), CPU time: mean = 7.204 us, total = 6.851 ms UNKNOWN - 319 total (1 active), CPU time: mean = 5.675 us, total = 1.810 ms GCSServer.deadline_timer.debug_state_dump - 96 total (1 active), CPU time: mean = 2.278 ms, total = 218.732 ms GCSServer.deadline_timer.debug_state_event_stats_print - 16 total (1 active, 1 running), CPU time: mean = 302.052 us, total = 4.833 ms GcsInMemoryStore.GetAll - 6 total (0 active), CPU time: mean = 14.767 us, total = 88.600 us PeriodicalRunner.RunFnPeriodically - 4 total (0 active), CPU time: mean = 678.900 us, total = 2.716 ms GcsInMemoryStore.Get - 2 total (0 active), CPU time: mean = 14.127 us, total = 28.253 us InternalKVGcsService.grpc_server.InternalKVGet - 1 total (0 active), CPU time: mean = 14.884 us, total = 14.884 us [2023-08-23 13:03:44,560 I 41356 40156] (gcs_server.exe) gcs_server.cc:873: GcsTaskManager Event stats: Global stats: 0 total (0 active) Queueing time: mean = -nan(ind) s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Execution time: mean = -nan(ind) s, total = 0.000 s Event stats: [2023-08-23 13:04:44,573 I 41356 40156] (gcs_server.exe) gcs_server.cc:255: GcsNodeManager: RegisterNode request count: 0 DrainNode request count: 0 GetAllNodeInfo request count: 0 GetInternalConfig request count: 0 GcsActorManager: RegisterActor request count: 0 CreateActor request count: 0 GetActorInfo request count: 0 GetNamedActorInfo request count: 0 GetAllActorInfo request count: 0 KillActor request count: 0 ListNamedActors request count: 0 Registered actors count: 0 Destroyed actors count: 0 Named actors count: 0 Unresolved actors count: 0 Pending actors count: 0 Created actors count: 0 owners_: 0 actor_to_register_callbacks_: 0 actor_to_create_callbacks_: 0 sorted_destroyed_actor_list_: 0 GcsResourceManager: GetResources request count: 0 GetAllAvailableResources request count0 ReportResourceUsage request count: 0 GetAllResourceUsage request count: 0 GcsPlacementGroupManager: CreatePlacementGroup request count: 0 RemovePlacementGroup request count: 0 GetPlacementGroup request count: 0 GetAllPlacementGroup request count: 0 WaitPlacementGroupUntilReady request count: 0 GetNamedPlacementGroup request count: 0 Scheduling pending placement group count: 0 Registered placement groups count: 0 Named placement group count: 0 Pending placement groups count: 0 Infeasible placement groups count: 0 GcsPublisher {} [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 0 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Total num bytes of task event stored: 0MiB -Current num of task events stored: 0 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 0 -Total num of driver tasks: 0 [2023-08-23 13:04:44,573 I 41356 40156] (gcs_server.exe) gcs_server.cc:872: Event stats: Global stats: 5165 total (4 active) Queueing time: mean = 2.972 ms, max = 947.978 ms, min = -0.000 s, total = 15.350 s Execution time: mean = 62.718 us, total = 323.936 ms Event stats: GcsInMemoryStore.Put - 1231 total (0 active), CPU time: mean = 41.218 us, total = 50.740 ms InternalKVGcsService.grpc_server.InternalKVPut - 1228 total (0 active), CPU time: mean = 12.535 us, total = 15.393 ms InternalKVGcsService.grpc_client.InternalKVPut - 1224 total (0 active), CPU time: mean = 6.060 us, total = 7.417 ms RayletLoadPulled - 1011 total (1 active), CPU time: mean = 7.141 us, total = 7.220 ms UNKNOWN - 339 total (1 active), CPU time: mean = 5.631 us, total = 1.909 ms GCSServer.deadline_timer.debug_state_dump - 102 total (1 active), CPU time: mean = 2.286 ms, total = 233.182 ms GCSServer.deadline_timer.debug_state_event_stats_print - 17 total (1 active, 1 running), CPU time: mean = 307.563 us, total = 5.229 ms GcsInMemoryStore.GetAll - 6 total (0 active), CPU time: mean = 14.767 us, total = 88.600 us PeriodicalRunner.RunFnPeriodically - 4 total (0 active), CPU time: mean = 678.900 us, total = 2.716 ms GcsInMemoryStore.Get - 2 total (0 active), CPU time: mean = 14.127 us, total = 28.253 us InternalKVGcsService.grpc_server.InternalKVGet - 1 total (0 active), CPU time: mean = 14.884 us, total = 14.884 us [2023-08-23 13:04:44,573 I 41356 40156] (gcs_server.exe) gcs_server.cc:873: GcsTaskManager Event stats: Global stats: 0 total (0 active) Queueing time: mean = -nan(ind) s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Execution time: mean = -nan(ind) s, total = 0.000 s Event stats: [2023-08-23 13:05:44,581 I 41356 40156] (gcs_server.exe) gcs_server.cc:255: GcsNodeManager: RegisterNode request count: 0 DrainNode request count: 0 GetAllNodeInfo request count: 0 GetInternalConfig request count: 0 GcsActorManager: RegisterActor request count: 0 CreateActor request count: 0 GetActorInfo request count: 0 GetNamedActorInfo request count: 0 GetAllActorInfo request count: 0 KillActor request count: 0 ListNamedActors request count: 0 Registered actors count: 0 Destroyed actors count: 0 Named actors count: 0 Unresolved actors count: 0 Pending actors count: 0 Created actors count: 0 owners_: 0 actor_to_register_callbacks_: 0 actor_to_create_callbacks_: 0 sorted_destroyed_actor_list_: 0 GcsResourceManager: GetResources request count: 0 GetAllAvailableResources request count0 ReportResourceUsage request count: 0 GetAllResourceUsage request count: 0 GcsPlacementGroupManager: CreatePlacementGroup request count: 0 RemovePlacementGroup request count: 0 GetPlacementGroup request count: 0 GetAllPlacementGroup request count: 0 WaitPlacementGroupUntilReady request count: 0 GetNamedPlacementGroup request count: 0 Scheduling pending placement group count: 0 Registered placement groups count: 0 Named placement group count: 0 Pending placement groups count: 0 Infeasible placement groups count: 0 GcsPublisher {} [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 0 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Total num bytes of task event stored: 0MiB -Current num of task events stored: 0 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 0 -Total num of driver tasks: 0 [2023-08-23 13:05:44,581 I 41356 40156] (gcs_server.exe) gcs_server.cc:872: Event stats: Global stats: 5468 total (4 active) Queueing time: mean = 2.934 ms, max = 947.978 ms, min = -0.000 s, total = 16.042 s Execution time: mean = 62.177 us, total = 339.986 ms Event stats: GcsInMemoryStore.Put - 1303 total (0 active), CPU time: mean = 39.127 us, total = 50.982 ms InternalKVGcsService.grpc_server.InternalKVPut - 1300 total (0 active), CPU time: mean = 12.550 us, total = 16.314 ms InternalKVGcsService.grpc_client.InternalKVPut - 1296 total (0 active), CPU time: mean = 6.093 us, total = 7.897 ms RayletLoadPulled - 1071 total (1 active), CPU time: mean = 7.089 us, total = 7.593 ms UNKNOWN - 359 total (1 active), CPU time: mean = 5.625 us, total = 2.019 ms GCSServer.deadline_timer.debug_state_dump - 108 total (1 active), CPU time: mean = 2.285 ms, total = 246.815 ms GCSServer.deadline_timer.debug_state_event_stats_print - 18 total (1 active, 1 running), CPU time: mean = 306.596 us, total = 5.519 ms GcsInMemoryStore.GetAll - 6 total (0 active), CPU time: mean = 14.767 us, total = 88.600 us PeriodicalRunner.RunFnPeriodically - 4 total (0 active), CPU time: mean = 678.900 us, total = 2.716 ms GcsInMemoryStore.Get - 2 total (0 active), CPU time: mean = 14.127 us, total = 28.253 us InternalKVGcsService.grpc_server.InternalKVGet - 1 total (0 active), CPU time: mean = 14.884 us, total = 14.884 us [2023-08-23 13:05:44,581 I 41356 40156] (gcs_server.exe) gcs_server.cc:873: GcsTaskManager Event stats: Global stats: 0 total (0 active) Queueing time: mean = -nan(ind) s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Execution time: mean = -nan(ind) s, total = 0.000 s Event stats: [2023-08-23 13:06:44,582 I 41356 40156] (gcs_server.exe) gcs_server.cc:255: GcsNodeManager: RegisterNode request count: 0 DrainNode request count: 0 GetAllNodeInfo request count: 0 GetInternalConfig request count: 0 GcsActorManager: RegisterActor request count: 0 CreateActor request count: 0 GetActorInfo request count: 0 GetNamedActorInfo request count: 0 GetAllActorInfo request count: 0 KillActor request count: 0 ListNamedActors request count: 0 Registered actors count: 0 Destroyed actors count: 0 Named actors count: 0 Unresolved actors count: 0 Pending actors count: 0 Created actors count: 0 owners_: 0 actor_to_register_callbacks_: 0 actor_to_create_callbacks_: 0 sorted_destroyed_actor_list_: 0 GcsResourceManager: GetResources request count: 0 GetAllAvailableResources request count0 ReportResourceUsage request count: 0 GetAllResourceUsage request count: 0 GcsPlacementGroupManager: CreatePlacementGroup request count: 0 RemovePlacementGroup request count: 0 GetPlacementGroup request count: 0 GetAllPlacementGroup request count: 0 WaitPlacementGroupUntilReady request count: 0 GetNamedPlacementGroup request count: 0 Scheduling pending placement group count: 0 Registered placement groups count: 0 Named placement group count: 0 Pending placement groups count: 0 Infeasible placement groups count: 0 GcsPublisher {} [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 0 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Total num bytes of task event stored: 0MiB -Current num of task events stored: 0 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 0 -Total num of driver tasks: 0 [2023-08-23 13:06:44,582 I 41356 40156] (gcs_server.exe) gcs_server.cc:872: Event stats: Global stats: 5770 total (4 active) Queueing time: mean = 2.952 ms, max = 947.978 ms, min = -0.000 s, total = 17.033 s Execution time: mean = 61.787 us, total = 356.512 ms Event stats: GcsInMemoryStore.Put - 1375 total (0 active), CPU time: mean = 37.300 us, total = 51.287 ms InternalKVGcsService.grpc_server.InternalKVPut - 1372 total (0 active), CPU time: mean = 12.647 us, total = 17.352 ms InternalKVGcsService.grpc_client.InternalKVPut - 1368 total (0 active), CPU time: mean = 6.136 us, total = 8.394 ms RayletLoadPulled - 1130 total (1 active), CPU time: mean = 7.159 us, total = 8.090 ms UNKNOWN - 379 total (1 active), CPU time: mean = 5.664 us, total = 2.147 ms GCSServer.deadline_timer.debug_state_dump - 114 total (1 active), CPU time: mean = 2.285 ms, total = 260.532 ms GCSServer.deadline_timer.debug_state_event_stats_print - 19 total (1 active, 1 running), CPU time: mean = 308.602 us, total = 5.863 ms GcsInMemoryStore.GetAll - 6 total (0 active), CPU time: mean = 14.767 us, total = 88.600 us PeriodicalRunner.RunFnPeriodically - 4 total (0 active), CPU time: mean = 678.900 us, total = 2.716 ms GcsInMemoryStore.Get - 2 total (0 active), CPU time: mean = 14.127 us, total = 28.253 us InternalKVGcsService.grpc_server.InternalKVGet - 1 total (0 active), CPU time: mean = 14.884 us, total = 14.884 us [2023-08-23 13:06:44,582 I 41356 40156] (gcs_server.exe) gcs_server.cc:873: GcsTaskManager Event stats: Global stats: 0 total (0 active) Queueing time: mean = -nan(ind) s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Execution time: mean = -nan(ind) s, total = 0.000 s Event stats: [2023-08-23 13:07:44,583 I 41356 40156] (gcs_server.exe) gcs_server.cc:255: GcsNodeManager: RegisterNode request count: 0 DrainNode request count: 0 GetAllNodeInfo request count: 0 GetInternalConfig request count: 0 GcsActorManager: RegisterActor request count: 0 CreateActor request count: 0 GetActorInfo request count: 0 GetNamedActorInfo request count: 0 GetAllActorInfo request count: 0 KillActor request count: 0 ListNamedActors request count: 0 Registered actors count: 0 Destroyed actors count: 0 Named actors count: 0 Unresolved actors count: 0 Pending actors count: 0 Created actors count: 0 owners_: 0 actor_to_register_callbacks_: 0 actor_to_create_callbacks_: 0 sorted_destroyed_actor_list_: 0 GcsResourceManager: GetResources request count: 0 GetAllAvailableResources request count0 ReportResourceUsage request count: 0 GetAllResourceUsage request count: 0 GcsPlacementGroupManager: CreatePlacementGroup request count: 0 RemovePlacementGroup request count: 0 GetPlacementGroup request count: 0 GetAllPlacementGroup request count: 0 WaitPlacementGroupUntilReady request count: 0 GetNamedPlacementGroup request count: 0 Scheduling pending placement group count: 0 Registered placement groups count: 0 Named placement group count: 0 Pending placement groups count: 0 Infeasible placement groups count: 0 GcsPublisher {} [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 0 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Total num bytes of task event stored: 0MiB -Current num of task events stored: 0 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 0 -Total num of driver tasks: 0 [2023-08-23 13:07:44,583 I 41356 40156] (gcs_server.exe) gcs_server.cc:872: Event stats: Global stats: 6072 total (4 active) Queueing time: mean = 3.045 ms, max = 947.978 ms, min = -0.000 s, total = 18.490 s Execution time: mean = 61.234 us, total = 371.811 ms Event stats: GcsInMemoryStore.Put - 1447 total (0 active), CPU time: mean = 35.627 us, total = 51.553 ms InternalKVGcsService.grpc_server.InternalKVPut - 1444 total (0 active), CPU time: mean = 12.616 us, total = 18.218 ms InternalKVGcsService.grpc_client.InternalKVPut - 1440 total (0 active), CPU time: mean = 6.178 us, total = 8.897 ms RayletLoadPulled - 1189 total (1 active), CPU time: mean = 7.169 us, total = 8.523 ms UNKNOWN - 399 total (1 active), CPU time: mean = 5.698 us, total = 2.273 ms GCSServer.deadline_timer.debug_state_dump - 120 total (1 active), CPU time: mean = 2.278 ms, total = 273.393 ms GCSServer.deadline_timer.debug_state_event_stats_print - 20 total (1 active, 1 running), CPU time: mean = 305.334 us, total = 6.107 ms GcsInMemoryStore.GetAll - 6 total (0 active), CPU time: mean = 14.767 us, total = 88.600 us PeriodicalRunner.RunFnPeriodically - 4 total (0 active), CPU time: mean = 678.900 us, total = 2.716 ms GcsInMemoryStore.Get - 2 total (0 active), CPU time: mean = 14.127 us, total = 28.253 us InternalKVGcsService.grpc_server.InternalKVGet - 1 total (0 active), CPU time: mean = 14.884 us, total = 14.884 us [2023-08-23 13:07:44,583 I 41356 40156] (gcs_server.exe) gcs_server.cc:873: GcsTaskManager Event stats: Global stats: 0 total (0 active) Queueing time: mean = -nan(ind) s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Execution time: mean = -nan(ind) s, total = 0.000 s Event stats: [2023-08-23 13:08:44,587 I 41356 40156] (gcs_server.exe) gcs_server.cc:255: GcsNodeManager: RegisterNode request count: 0 DrainNode request count: 0 GetAllNodeInfo request count: 0 GetInternalConfig request count: 0 GcsActorManager: RegisterActor request count: 0 CreateActor request count: 0 GetActorInfo request count: 0 GetNamedActorInfo request count: 0 GetAllActorInfo request count: 0 KillActor request count: 0 ListNamedActors request count: 0 Registered actors count: 0 Destroyed actors count: 0 Named actors count: 0 Unresolved actors count: 0 Pending actors count: 0 Created actors count: 0 owners_: 0 actor_to_register_callbacks_: 0 actor_to_create_callbacks_: 0 sorted_destroyed_actor_list_: 0 GcsResourceManager: GetResources request count: 0 GetAllAvailableResources request count0 ReportResourceUsage request count: 0 GetAllResourceUsage request count: 0 GcsPlacementGroupManager: CreatePlacementGroup request count: 0 RemovePlacementGroup request count: 0 GetPlacementGroup request count: 0 GetAllPlacementGroup request count: 0 WaitPlacementGroupUntilReady request count: 0 GetNamedPlacementGroup request count: 0 Scheduling pending placement group count: 0 Registered placement groups count: 0 Named placement group count: 0 Pending placement groups count: 0 Infeasible placement groups count: 0 GcsPublisher {} [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 0 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Total num bytes of task event stored: 0MiB -Current num of task events stored: 0 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 0 -Total num of driver tasks: 0 [2023-08-23 13:08:44,587 I 41356 40156] (gcs_server.exe) gcs_server.cc:872: Event stats: Global stats: 6374 total (4 active) Queueing time: mean = 3.181 ms, max = 947.978 ms, min = -0.001 s, total = 20.275 s Execution time: mean = 61.087 us, total = 389.370 ms Event stats: GcsInMemoryStore.Put - 1519 total (0 active), CPU time: mean = 34.233 us, total = 52.000 ms InternalKVGcsService.grpc_server.InternalKVPut - 1516 total (0 active), CPU time: mean = 12.729 us, total = 19.297 ms InternalKVGcsService.grpc_client.InternalKVPut - 1512 total (0 active), CPU time: mean = 6.244 us, total = 9.441 ms RayletLoadPulled - 1248 total (1 active), CPU time: mean = 7.367 us, total = 9.195 ms UNKNOWN - 419 total (1 active), CPU time: mean = 5.728 us, total = 2.400 ms GCSServer.deadline_timer.debug_state_dump - 126 total (1 active), CPU time: mean = 2.284 ms, total = 287.756 ms GCSServer.deadline_timer.debug_state_event_stats_print - 21 total (1 active, 1 running), CPU time: mean = 306.433 us, total = 6.435 ms GcsInMemoryStore.GetAll - 6 total (0 active), CPU time: mean = 14.767 us, total = 88.600 us PeriodicalRunner.RunFnPeriodically - 4 total (0 active), CPU time: mean = 678.900 us, total = 2.716 ms GcsInMemoryStore.Get - 2 total (0 active), CPU time: mean = 14.127 us, total = 28.253 us InternalKVGcsService.grpc_server.InternalKVGet - 1 total (0 active), CPU time: mean = 14.884 us, total = 14.884 us [2023-08-23 13:08:44,587 I 41356 40156] (gcs_server.exe) gcs_server.cc:873: GcsTaskManager Event stats: Global stats: 0 total (0 active) Queueing time: mean = -nan(ind) s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Execution time: mean = -nan(ind) s, total = 0.000 s Event stats: [2023-08-23 13:09:44,591 I 41356 40156] (gcs_server.exe) gcs_server.cc:255: GcsNodeManager: RegisterNode request count: 0 DrainNode request count: 0 GetAllNodeInfo request count: 0 GetInternalConfig request count: 0 GcsActorManager: RegisterActor request count: 0 CreateActor request count: 0 GetActorInfo request count: 0 GetNamedActorInfo request count: 0 GetAllActorInfo request count: 0 KillActor request count: 0 ListNamedActors request count: 0 Registered actors count: 0 Destroyed actors count: 0 Named actors count: 0 Unresolved actors count: 0 Pending actors count: 0 Created actors count: 0 owners_: 0 actor_to_register_callbacks_: 0 actor_to_create_callbacks_: 0 sorted_destroyed_actor_list_: 0 GcsResourceManager: GetResources request count: 0 GetAllAvailableResources request count0 ReportResourceUsage request count: 0 GetAllResourceUsage request count: 0 GcsPlacementGroupManager: CreatePlacementGroup request count: 0 RemovePlacementGroup request count: 0 GetPlacementGroup request count: 0 GetAllPlacementGroup request count: 0 WaitPlacementGroupUntilReady request count: 0 GetNamedPlacementGroup request count: 0 Scheduling pending placement group count: 0 Registered placement groups count: 0 Named placement group count: 0 Pending placement groups count: 0 Infeasible placement groups count: 0 GcsPublisher {} [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 0 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Total num bytes of task event stored: 0MiB -Current num of task events stored: 0 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 0 -Total num of driver tasks: 0 [2023-08-23 13:09:44,591 I 41356 40156] (gcs_server.exe) gcs_server.cc:872: Event stats: Global stats: 6675 total (4 active) Queueing time: mean = 3.280 ms, max = 947.978 ms, min = -0.001 s, total = 21.894 s Execution time: mean = 60.811 us, total = 405.911 ms Event stats: GcsInMemoryStore.Put - 1591 total (0 active), CPU time: mean = 32.841 us, total = 52.249 ms InternalKVGcsService.grpc_server.InternalKVPut - 1588 total (0 active), CPU time: mean = 12.728 us, total = 20.213 ms InternalKVGcsService.grpc_client.InternalKVPut - 1584 total (0 active), CPU time: mean = 6.216 us, total = 9.846 ms RayletLoadPulled - 1307 total (1 active), CPU time: mean = 7.406 us, total = 9.680 ms UNKNOWN - 438 total (1 active), CPU time: mean = 5.784 us, total = 2.533 ms GCSServer.deadline_timer.debug_state_dump - 132 total (1 active), CPU time: mean = 2.287 ms, total = 301.848 ms GCSServer.deadline_timer.debug_state_event_stats_print - 22 total (1 active, 1 running), CPU time: mean = 304.278 us, total = 6.694 ms GcsInMemoryStore.GetAll - 6 total (0 active), CPU time: mean = 14.767 us, total = 88.600 us PeriodicalRunner.RunFnPeriodically - 4 total (0 active), CPU time: mean = 678.900 us, total = 2.716 ms GcsInMemoryStore.Get - 2 total (0 active), CPU time: mean = 14.127 us, total = 28.253 us InternalKVGcsService.grpc_server.InternalKVGet - 1 total (0 active), CPU time: mean = 14.884 us, total = 14.884 us [2023-08-23 13:09:44,591 I 41356 40156] (gcs_server.exe) gcs_server.cc:873: GcsTaskManager Event stats: Global stats: 0 total (0 active) Queueing time: mean = -nan(ind) s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Execution time: mean = -nan(ind) s, total = 0.000 s Event stats: [2023-08-23 13:10:44,595 I 41356 40156] (gcs_server.exe) gcs_server.cc:255: GcsNodeManager: RegisterNode request count: 0 DrainNode request count: 0 GetAllNodeInfo request count: 0 GetInternalConfig request count: 0 GcsActorManager: RegisterActor request count: 0 CreateActor request count: 0 GetActorInfo request count: 0 GetNamedActorInfo request count: 0 GetAllActorInfo request count: 0 KillActor request count: 0 ListNamedActors request count: 0 Registered actors count: 0 Destroyed actors count: 0 Named actors count: 0 Unresolved actors count: 0 Pending actors count: 0 Created actors count: 0 owners_: 0 actor_to_register_callbacks_: 0 actor_to_create_callbacks_: 0 sorted_destroyed_actor_list_: 0 GcsResourceManager: GetResources request count: 0 GetAllAvailableResources request count0 ReportResourceUsage request count: 0 GetAllResourceUsage request count: 0 GcsPlacementGroupManager: CreatePlacementGroup request count: 0 RemovePlacementGroup request count: 0 GetPlacementGroup request count: 0 GetAllPlacementGroup request count: 0 WaitPlacementGroupUntilReady request count: 0 GetNamedPlacementGroup request count: 0 Scheduling pending placement group count: 0 Registered placement groups count: 0 Named placement group count: 0 Pending placement groups count: 0 Infeasible placement groups count: 0 GcsPublisher {} [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 0 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Total num bytes of task event stored: 0MiB -Current num of task events stored: 0 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 0 -Total num of driver tasks: 0 [2023-08-23 13:10:44,595 I 41356 40156] (gcs_server.exe) gcs_server.cc:872: Event stats: Global stats: 6977 total (4 active) Queueing time: mean = 3.243 ms, max = 947.978 ms, min = -0.001 s, total = 22.629 s Execution time: mean = 60.421 us, total = 421.558 ms Event stats: GcsInMemoryStore.Put - 1663 total (0 active), CPU time: mean = 31.577 us, total = 52.513 ms InternalKVGcsService.grpc_server.InternalKVPut - 1660 total (0 active), CPU time: mean = 12.842 us, total = 21.318 ms InternalKVGcsService.grpc_client.InternalKVPut - 1656 total (0 active), CPU time: mean = 6.292 us, total = 10.419 ms RayletLoadPulled - 1366 total (1 active), CPU time: mean = 7.385 us, total = 10.088 ms UNKNOWN - 458 total (1 active), CPU time: mean = 5.762 us, total = 2.639 ms GCSServer.deadline_timer.debug_state_dump - 138 total (1 active), CPU time: mean = 2.281 ms, total = 314.750 ms GCSServer.deadline_timer.debug_state_event_stats_print - 23 total (1 active, 1 running), CPU time: mean = 303.650 us, total = 6.984 ms GcsInMemoryStore.GetAll - 6 total (0 active), CPU time: mean = 14.767 us, total = 88.600 us PeriodicalRunner.RunFnPeriodically - 4 total (0 active), CPU time: mean = 678.900 us, total = 2.716 ms GcsInMemoryStore.Get - 2 total (0 active), CPU time: mean = 14.127 us, total = 28.253 us InternalKVGcsService.grpc_server.InternalKVGet - 1 total (0 active), CPU time: mean = 14.884 us, total = 14.884 us [2023-08-23 13:10:44,595 I 41356 40156] (gcs_server.exe) gcs_server.cc:873: GcsTaskManager Event stats: Global stats: 0 total (0 active) Queueing time: mean = -nan(ind) s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Execution time: mean = -nan(ind) s, total = 0.000 s Event stats: [2023-08-23 13:11:44,605 I 41356 40156] (gcs_server.exe) gcs_server.cc:255: GcsNodeManager: RegisterNode request count: 0 DrainNode request count: 0 GetAllNodeInfo request count: 0 GetInternalConfig request count: 0 GcsActorManager: RegisterActor request count: 0 CreateActor request count: 0 GetActorInfo request count: 0 GetNamedActorInfo request count: 0 GetAllActorInfo request count: 0 KillActor request count: 0 ListNamedActors request count: 0 Registered actors count: 0 Destroyed actors count: 0 Named actors count: 0 Unresolved actors count: 0 Pending actors count: 0 Created actors count: 0 owners_: 0 actor_to_register_callbacks_: 0 actor_to_create_callbacks_: 0 sorted_destroyed_actor_list_: 0 GcsResourceManager: GetResources request count: 0 GetAllAvailableResources request count0 ReportResourceUsage request count: 0 GetAllResourceUsage request count: 0 GcsPlacementGroupManager: CreatePlacementGroup request count: 0 RemovePlacementGroup request count: 0 GetPlacementGroup request count: 0 GetAllPlacementGroup request count: 0 WaitPlacementGroupUntilReady request count: 0 GetNamedPlacementGroup request count: 0 Scheduling pending placement group count: 0 Registered placement groups count: 0 Named placement group count: 0 Pending placement groups count: 0 Infeasible placement groups count: 0 GcsPublisher {} [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 0 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Total num bytes of task event stored: 0MiB -Current num of task events stored: 0 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 0 -Total num of driver tasks: 0 [2023-08-23 13:11:44,605 I 41356 40156] (gcs_server.exe) gcs_server.cc:872: Event stats: Global stats: 7280 total (4 active) Queueing time: mean = 3.197 ms, max = 947.978 ms, min = -0.001 s, total = 23.276 s Execution time: mean = 60.091 us, total = 437.460 ms Event stats: GcsInMemoryStore.Put - 1735 total (0 active), CPU time: mean = 30.432 us, total = 52.800 ms InternalKVGcsService.grpc_server.InternalKVPut - 1732 total (0 active), CPU time: mean = 12.831 us, total = 22.223 ms InternalKVGcsService.grpc_client.InternalKVPut - 1728 total (0 active), CPU time: mean = 6.348 us, total = 10.969 ms RayletLoadPulled - 1426 total (1 active), CPU time: mean = 7.355 us, total = 10.488 ms UNKNOWN - 478 total (1 active), CPU time: mean = 5.694 us, total = 2.722 ms GCSServer.deadline_timer.debug_state_dump - 144 total (1 active), CPU time: mean = 2.278 ms, total = 328.013 ms GCSServer.deadline_timer.debug_state_event_stats_print - 24 total (1 active, 1 running), CPU time: mean = 308.291 us, total = 7.399 ms GcsInMemoryStore.GetAll - 6 total (0 active), CPU time: mean = 14.767 us, total = 88.600 us PeriodicalRunner.RunFnPeriodically - 4 total (0 active), CPU time: mean = 678.900 us, total = 2.716 ms GcsInMemoryStore.Get - 2 total (0 active), CPU time: mean = 14.127 us, total = 28.253 us InternalKVGcsService.grpc_server.InternalKVGet - 1 total (0 active), CPU time: mean = 14.884 us, total = 14.884 us [2023-08-23 13:11:44,605 I 41356 40156] (gcs_server.exe) gcs_server.cc:873: GcsTaskManager Event stats: Global stats: 0 total (0 active) Queueing time: mean = -nan(ind) s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Execution time: mean = -nan(ind) s, total = 0.000 s Event stats: [2023-08-23 13:12:44,608 I 41356 40156] (gcs_server.exe) gcs_server.cc:255: GcsNodeManager: RegisterNode request count: 0 DrainNode request count: 0 GetAllNodeInfo request count: 0 GetInternalConfig request count: 0 GcsActorManager: RegisterActor request count: 0 CreateActor request count: 0 GetActorInfo request count: 0 GetNamedActorInfo request count: 0 GetAllActorInfo request count: 0 KillActor request count: 0 ListNamedActors request count: 0 Registered actors count: 0 Destroyed actors count: 0 Named actors count: 0 Unresolved actors count: 0 Pending actors count: 0 Created actors count: 0 owners_: 0 actor_to_register_callbacks_: 0 actor_to_create_callbacks_: 0 sorted_destroyed_actor_list_: 0 GcsResourceManager: GetResources request count: 0 GetAllAvailableResources request count0 ReportResourceUsage request count: 0 GetAllResourceUsage request count: 0 GcsPlacementGroupManager: CreatePlacementGroup request count: 0 RemovePlacementGroup request count: 0 GetPlacementGroup request count: 0 GetAllPlacementGroup request count: 0 WaitPlacementGroupUntilReady request count: 0 GetNamedPlacementGroup request count: 0 Scheduling pending placement group count: 0 Registered placement groups count: 0 Named placement group count: 0 Pending placement groups count: 0 Infeasible placement groups count: 0 GcsPublisher {} [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 0 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Total num bytes of task event stored: 0MiB -Current num of task events stored: 0 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 0 -Total num of driver tasks: 0 [2023-08-23 13:12:44,608 I 41356 40156] (gcs_server.exe) gcs_server.cc:872: Event stats: Global stats: 7583 total (4 active) Queueing time: mean = 3.150 ms, max = 947.978 ms, min = -0.001 s, total = 23.885 s Execution time: mean = 59.722 us, total = 452.870 ms Event stats: GcsInMemoryStore.Put - 1807 total (0 active), CPU time: mean = 29.322 us, total = 52.985 ms InternalKVGcsService.grpc_server.InternalKVPut - 1804 total (0 active), CPU time: mean = 12.872 us, total = 23.221 ms InternalKVGcsService.grpc_client.InternalKVPut - 1800 total (0 active), CPU time: mean = 6.415 us, total = 11.547 ms RayletLoadPulled - 1486 total (1 active), CPU time: mean = 7.394 us, total = 10.987 ms UNKNOWN - 498 total (1 active), CPU time: mean = 5.609 us, total = 2.793 ms GCSServer.deadline_timer.debug_state_dump - 150 total (1 active), CPU time: mean = 2.272 ms, total = 340.842 ms GCSServer.deadline_timer.debug_state_event_stats_print - 25 total (1 active, 1 running), CPU time: mean = 305.920 us, total = 7.648 ms GcsInMemoryStore.GetAll - 6 total (0 active), CPU time: mean = 14.767 us, total = 88.600 us PeriodicalRunner.RunFnPeriodically - 4 total (0 active), CPU time: mean = 678.900 us, total = 2.716 ms GcsInMemoryStore.Get - 2 total (0 active), CPU time: mean = 14.127 us, total = 28.253 us InternalKVGcsService.grpc_server.InternalKVGet - 1 total (0 active), CPU time: mean = 14.884 us, total = 14.884 us [2023-08-23 13:12:44,608 I 41356 40156] (gcs_server.exe) gcs_server.cc:873: GcsTaskManager Event stats: Global stats: 0 total (0 active) Queueing time: mean = -nan(ind) s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Execution time: mean = -nan(ind) s, total = 0.000 s Event stats: [2023-08-23 13:13:44,610 I 41356 40156] (gcs_server.exe) gcs_server.cc:255: GcsNodeManager: RegisterNode request count: 0 DrainNode request count: 0 GetAllNodeInfo request count: 0 GetInternalConfig request count: 0 GcsActorManager: RegisterActor request count: 0 CreateActor request count: 0 GetActorInfo request count: 0 GetNamedActorInfo request count: 0 GetAllActorInfo request count: 0 KillActor request count: 0 ListNamedActors request count: 0 Registered actors count: 0 Destroyed actors count: 0 Named actors count: 0 Unresolved actors count: 0 Pending actors count: 0 Created actors count: 0 owners_: 0 actor_to_register_callbacks_: 0 actor_to_create_callbacks_: 0 sorted_destroyed_actor_list_: 0 GcsResourceManager: GetResources request count: 0 GetAllAvailableResources request count0 ReportResourceUsage request count: 0 GetAllResourceUsage request count: 0 GcsPlacementGroupManager: CreatePlacementGroup request count: 0 RemovePlacementGroup request count: 0 GetPlacementGroup request count: 0 GetAllPlacementGroup request count: 0 WaitPlacementGroupUntilReady request count: 0 GetNamedPlacementGroup request count: 0 Scheduling pending placement group count: 0 Registered placement groups count: 0 Named placement group count: 0 Pending placement groups count: 0 Infeasible placement groups count: 0 GcsPublisher {} [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 0 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Total num bytes of task event stored: 0MiB -Current num of task events stored: 0 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 0 -Total num of driver tasks: 0 [2023-08-23 13:13:44,610 I 41356 40156] (gcs_server.exe) gcs_server.cc:872: Event stats: Global stats: 7885 total (4 active) Queueing time: mean = 3.116 ms, max = 947.978 ms, min = -0.001 s, total = 24.569 s Execution time: mean = 59.482 us, total = 469.013 ms Event stats: GcsInMemoryStore.Put - 1879 total (0 active), CPU time: mean = 28.349 us, total = 53.267 ms InternalKVGcsService.grpc_server.InternalKVPut - 1876 total (0 active), CPU time: mean = 12.981 us, total = 24.352 ms InternalKVGcsService.grpc_client.InternalKVPut - 1872 total (0 active), CPU time: mean = 6.442 us, total = 12.060 ms RayletLoadPulled - 1545 total (1 active), CPU time: mean = 7.335 us, total = 11.333 ms UNKNOWN - 518 total (1 active), CPU time: mean = 5.550 us, total = 2.875 ms GCSServer.deadline_timer.debug_state_dump - 156 total (1 active), CPU time: mean = 2.271 ms, total = 354.322 ms GCSServer.deadline_timer.debug_state_event_stats_print - 26 total (1 active, 1 running), CPU time: mean = 305.982 us, total = 7.956 ms GcsInMemoryStore.GetAll - 6 total (0 active), CPU time: mean = 14.767 us, total = 88.600 us PeriodicalRunner.RunFnPeriodically - 4 total (0 active), CPU time: mean = 678.900 us, total = 2.716 ms GcsInMemoryStore.Get - 2 total (0 active), CPU time: mean = 14.127 us, total = 28.253 us InternalKVGcsService.grpc_server.InternalKVGet - 1 total (0 active), CPU time: mean = 14.884 us, total = 14.884 us [2023-08-23 13:13:44,610 I 41356 40156] (gcs_server.exe) gcs_server.cc:873: GcsTaskManager Event stats: Global stats: 0 total (0 active) Queueing time: mean = -nan(ind) s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Execution time: mean = -nan(ind) s, total = 0.000 s Event stats: [2023-08-23 13:14:44,625 I 41356 40156] (gcs_server.exe) gcs_server.cc:255: GcsNodeManager: RegisterNode request count: 0 DrainNode request count: 0 GetAllNodeInfo request count: 0 GetInternalConfig request count: 0 GcsActorManager: RegisterActor request count: 0 CreateActor request count: 0 GetActorInfo request count: 0 GetNamedActorInfo request count: 0 GetAllActorInfo request count: 0 KillActor request count: 0 ListNamedActors request count: 0 Registered actors count: 0 Destroyed actors count: 0 Named actors count: 0 Unresolved actors count: 0 Pending actors count: 0 Created actors count: 0 owners_: 0 actor_to_register_callbacks_: 0 actor_to_create_callbacks_: 0 sorted_destroyed_actor_list_: 0 GcsResourceManager: GetResources request count: 0 GetAllAvailableResources request count0 ReportResourceUsage request count: 0 GetAllResourceUsage request count: 0 GcsPlacementGroupManager: CreatePlacementGroup request count: 0 RemovePlacementGroup request count: 0 GetPlacementGroup request count: 0 GetAllPlacementGroup request count: 0 WaitPlacementGroupUntilReady request count: 0 GetNamedPlacementGroup request count: 0 Scheduling pending placement group count: 0 Registered placement groups count: 0 Named placement group count: 0 Pending placement groups count: 0 Infeasible placement groups count: 0 GcsPublisher {} [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 0 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Total num bytes of task event stored: 0MiB -Current num of task events stored: 0 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 0 -Total num of driver tasks: 0 [2023-08-23 13:14:44,626 I 41356 40156] (gcs_server.exe) gcs_server.cc:872: Event stats: Global stats: 8188 total (4 active) Queueing time: mean = 3.074 ms, max = 947.978 ms, min = -0.001 s, total = 25.170 s Execution time: mean = 59.310 us, total = 485.634 ms Event stats: GcsInMemoryStore.Put - 1951 total (0 active), CPU time: mean = 27.466 us, total = 53.586 ms InternalKVGcsService.grpc_server.InternalKVPut - 1948 total (0 active), CPU time: mean = 12.973 us, total = 25.271 ms InternalKVGcsService.grpc_client.InternalKVPut - 1944 total (0 active), CPU time: mean = 6.502 us, total = 12.640 ms RayletLoadPulled - 1605 total (1 active), CPU time: mean = 7.301 us, total = 11.717 ms UNKNOWN - 538 total (1 active), CPU time: mean = 5.521 us, total = 2.970 ms GCSServer.deadline_timer.debug_state_dump - 162 total (1 active), CPU time: mean = 2.274 ms, total = 368.443 ms GCSServer.deadline_timer.debug_state_event_stats_print - 27 total (1 active, 1 running), CPU time: mean = 302.191 us, total = 8.159 ms GcsInMemoryStore.GetAll - 6 total (0 active), CPU time: mean = 14.767 us, total = 88.600 us PeriodicalRunner.RunFnPeriodically - 4 total (0 active), CPU time: mean = 678.900 us, total = 2.716 ms GcsInMemoryStore.Get - 2 total (0 active), CPU time: mean = 14.127 us, total = 28.253 us InternalKVGcsService.grpc_server.InternalKVGet - 1 total (0 active), CPU time: mean = 14.884 us, total = 14.884 us [2023-08-23 13:14:44,626 I 41356 40156] (gcs_server.exe) gcs_server.cc:873: GcsTaskManager Event stats: Global stats: 0 total (0 active) Queueing time: mean = -nan(ind) s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Execution time: mean = -nan(ind) s, total = 0.000 s Event stats: [2023-08-23 13:15:45,419 I 41356 40156] (gcs_server.exe) gcs_server.cc:255: GcsNodeManager: RegisterNode request count: 0 DrainNode request count: 0 GetAllNodeInfo request count: 0 GetInternalConfig request count: 0 GcsActorManager: RegisterActor request count: 0 CreateActor request count: 0 GetActorInfo request count: 0 GetNamedActorInfo request count: 0 GetAllActorInfo request count: 0 KillActor request count: 0 ListNamedActors request count: 0 Registered actors count: 0 Destroyed actors count: 0 Named actors count: 0 Unresolved actors count: 0 Pending actors count: 0 Created actors count: 0 owners_: 0 actor_to_register_callbacks_: 0 actor_to_create_callbacks_: 0 sorted_destroyed_actor_list_: 0 GcsResourceManager: GetResources request count: 0 GetAllAvailableResources request count0 ReportResourceUsage request count: 0 GetAllResourceUsage request count: 0 GcsPlacementGroupManager: CreatePlacementGroup request count: 0 RemovePlacementGroup request count: 0 GetPlacementGroup request count: 0 GetAllPlacementGroup request count: 0 WaitPlacementGroupUntilReady request count: 0 GetNamedPlacementGroup request count: 0 Scheduling pending placement group count: 0 Registered placement groups count: 0 Named placement group count: 0 Pending placement groups count: 0 Infeasible placement groups count: 0 GcsPublisher {} [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 0 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Total num bytes of task event stored: 0MiB -Current num of task events stored: 0 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 0 -Total num of driver tasks: 0 [2023-08-23 13:15:45,419 I 41356 40156] (gcs_server.exe) gcs_server.cc:872: Event stats: Global stats: 8490 total (4 active) Queueing time: mean = 3.037 ms, max = 947.978 ms, min = -0.001 s, total = 25.782 s Execution time: mean = 59.138 us, total = 502.079 ms Event stats: GcsInMemoryStore.Put - 2023 total (0 active), CPU time: mean = 26.605 us, total = 53.822 ms InternalKVGcsService.grpc_server.InternalKVPut - 2020 total (0 active), CPU time: mean = 13.001 us, total = 26.263 ms InternalKVGcsService.grpc_client.InternalKVPut - 2016 total (0 active), CPU time: mean = 6.540 us, total = 13.186 ms RayletLoadPulled - 1664 total (1 active), CPU time: mean = 7.212 us, total = 12.001 ms UNKNOWN - 558 total (1 active), CPU time: mean = 5.473 us, total = 3.054 ms GCSServer.deadline_timer.debug_state_dump - 168 total (1 active), CPU time: mean = 2.276 ms, total = 382.390 ms GCSServer.deadline_timer.debug_state_event_stats_print - 28 total (1 active, 1 running), CPU time: mean = 304.130 us, total = 8.516 ms GcsInMemoryStore.GetAll - 6 total (0 active), CPU time: mean = 14.767 us, total = 88.600 us PeriodicalRunner.RunFnPeriodically - 4 total (0 active), CPU time: mean = 678.900 us, total = 2.716 ms GcsInMemoryStore.Get - 2 total (0 active), CPU time: mean = 14.127 us, total = 28.253 us InternalKVGcsService.grpc_server.InternalKVGet - 1 total (0 active), CPU time: mean = 14.884 us, total = 14.884 us [2023-08-23 13:15:45,419 I 41356 40156] (gcs_server.exe) gcs_server.cc:873: GcsTaskManager Event stats:

Global stats: 0 total (0 active) Queueing time: mean = -nan(ind) s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Execution time: mean = -nan(ind) s, total = 0.000 s Event stats:

I also ran the step-by-step you’ve provided;

# install python3.9 from the windows app store
>python -m venv d:\temp\cpython_39
>d:\temp\cpython_39\Scripts\activate
>pip install ray[default]
>python -c "import ray; ray.init()"

The install ran without issues, but the last line returned the error OSError: [Errno 0] AssignProcessToJobObject() failed.

AchimA on Aug 24, 2023

Thanks for the info. Reopening.

mattip on Aug 23, 2022