tensorflow: Crash when running tf.keras.layers.MaxPool2D on colab

Issue type

Bug

Have you reproduced the bug with TensorFlow Nightly?

Yes

Source

binary

TensorFlow version

2.13.0

Custom code

Yes

OS platform and distribution

No response

Mobile device

No response

Python version

No response

Bazel version

No response

GCC/compiler version

No response

CUDA/cuDNN version

No response

GPU model and memory

No response

Current behavior?

Due to large elements in the input list

Standalone code to reproduce the issue

import tensorflow as tf
import os
import numpy as np
try:
  arg_0_0 = 1e+38
  arg_0_1 = 16777216
  arg_0 = [arg_0_0,arg_0_1,]
  strides_0 = 2
  strides_1 = 2
  strides = [strides_0,strides_1,]
  padding = "same"
  arg_class = tf.keras.layers.MaxPool2D(arg_0,strides=strides,padding=padding,)
  arg_input_0_tensor = tf.random.uniform([3, 74, 74, 256], dtype=tf.float32)
  arg_input_0 = tf.identity(arg_input_0_tensor)
  arg_input = [arg_input_0,]
  out = arg_class(*arg_input)
except Exception as e:
  print("Error:"+str(e))

### Relevant log output

```shell
{"pid":7,"type":"jupyter","level":40,"msg":"[D 14:24:43.837 NotebookApp] Searching ['/root/.jupyter', '/root/.local/etc/jupyter', '/usr/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files","time":"2023-08-17T14:24:43.838Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 14:24:43.838 NotebookApp] Looking for jupyter_config in /etc/jupyter","time":"2023-08-17T14:24:43.846Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 14:24:43.840 NotebookApp] Looking for jupyter_config in /usr/local/etc/jupyter","time":"2023-08-17T14:24:43.847Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 14:24:43.842 NotebookApp] Looking for jupyter_config in /usr/etc/jupyter","time":"2023-08-17T14:24:43.848Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 14:24:43.842 NotebookApp] Looking for jupyter_config in /root/.local/etc/jupyter","time":"2023-08-17T14:24:43.848Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 14:24:43.842 NotebookApp] Looking for jupyter_config in /root/.jupyter","time":"2023-08-17T14:24:43.850Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 14:24:43.843 NotebookApp] Looking for jupyter_notebook_config in /etc/jupyter","time":"2023-08-17T14:24:43.850Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 14:24:43.847 NotebookApp] Loaded config file: /etc/jupyter/jupyter_notebook_config.py","time":"2023-08-17T14:24:43.854Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 14:24:43.847 NotebookApp] Looking for jupyter_notebook_config in /usr/local/etc/jupyter","time":"2023-08-17T14:24:43.855Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 14:24:43.849 NotebookApp] Loaded config file: /usr/local/etc/jupyter/jupyter_notebook_config.json","time":"2023-08-17T14:24:43.855Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 14:24:43.849 NotebookApp] Looking for jupyter_notebook_config in /usr/etc/jupyter","time":"2023-08-17T14:24:43.862Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 14:24:43.849 NotebookApp] Looking for jupyter_notebook_config in /root/.local/etc/jupyter","time":"2023-08-17T14:24:43.863Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 14:24:43.849 NotebookApp] Looking for jupyter_notebook_config in /root/.jupyter","time":"2023-08-17T14:24:43.864Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"[D 14:24:43.857 NotebookApp] Loaded config file: /root/.jupyter/jupyter_notebook_config.py","time":"2023-08-17T14:24:43.865Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"[D 14:24:44.049 NotebookApp] Searching ['/root/.jupyter', '/root/.local/etc/jupyter', '/usr/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files","time":"2023-08-17T14:24:44.050Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"[D 14:24:44.056 NotebookApp] Looking for jupyter_config in /etc/jupyter","time":"2023-08-17T14:24:44.056Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"[D 14:24:44.056 NotebookApp] Looking for jupyter_config in /usr/local/etc/jupyter","time":"2023-08-17T14:24:44.057Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"[D 14:24:44.056 NotebookApp] Looking for jupyter_config in /usr/etc/jupyter","time":"2023-08-17T14:24:44.057Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"[D 14:24:44.056 NotebookApp] Looking for jupyter_config in /root/.local/etc/jupyter","time":"2023-08-17T14:24:44.057Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"[D 14:24:44.056 NotebookApp] Looking for jupyter_config in /root/.jupyter","time":"2023-08-17T14:24:44.057Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"[D 14:24:44.059 NotebookApp] Looking for jupyter_notebook_config in /etc/jupyter","time":"2023-08-17T14:24:44.065Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"[D 14:24:44.059 NotebookApp] Loaded config file: /etc/jupyter/jupyter_notebook_config.py","time":"2023-08-17T14:24:44.066Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"[D 14:24:44.060 NotebookApp] Looking for jupyter_notebook_config in /usr/local/etc/jupyter","time":"2023-08-17T14:24:44.066Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"[D 14:24:44.060 NotebookApp] Loaded config file: /usr/local/etc/jupyter/jupyter_notebook_config.json","time":"2023-08-17T14:24:44.066Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"[D 14:24:44.060 NotebookApp] Looking for jupyter_notebook_config in /usr/etc/jupyter","time":"2023-08-17T14:24:44.066Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"[D 14:24:44.060 NotebookApp] Looking for jupyter_notebook_config in /root/.local/etc/jupyter","time":"2023-08-17T14:24:44.067Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"[D 14:24:44.060 NotebookApp] Looking for jupyter_notebook_config in /root/.jupyter","time":"2023-08-17T14:24:44.067Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"[D 14:24:44.061 NotebookApp] Loaded config file: /root/.jupyter/jupyter_notebook_config.py","time":"2023-08-17T14:24:44.067Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"    \t/etc/jupyter/jupyter_notebook_config.json","time":"2023-08-17T14:24:44.713Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"    \t/usr/local/etc/jupyter/jupyter_notebook_config.d/panel-client-jupyter.json","time":"2023-08-17T14:24:44.715Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"    \t/usr/local/etc/jupyter/jupyter_notebook_config.json","time":"2023-08-17T14:24:44.715Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"    \t/usr/etc/jupyter/jupyter_notebook_config.json","time":"2023-08-17T14:24:44.717Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"    \t/root/.local/etc/jupyter/jupyter_notebook_config.json","time":"2023-08-17T14:24:44.719Z","v":0}
{"pid":6,"type":"jupyter","level":40,"msg":"    \t/root/.jupyter/jupyter_notebook_config.json","time":"2023-08-17T14:24:44.723Z","v":0}
{"pid":6,"type":"jupyter","level":30,"msg":"Writing notebook server cookie secret to /root/.local/share/jupyter/runtime/notebook_cookie_secret","time":"2023-08-17T14:24:44.743Z","v":0}
{"pid":6,"type":"jupyter","level":30,"msg":"Authentication of /metrics is OFF, since other authentication is disabled.","time":"2023-08-17T14:24:44.745Z","v":0}
{"pid":6,"type":"jupyter","level":30,"msg":"google.colab serverextension initialized.","time":"2023-08-17T14:24:44.800Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"    \t/etc/jupyter/jupyter_notebook_config.json","time":"2023-08-17T14:24:44.806Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"    \t/usr/local/etc/jupyter/jupyter_notebook_config.d/panel-client-jupyter.json","time":"2023-08-17T14:24:44.808Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"    \t/usr/local/etc/jupyter/jupyter_notebook_config.json","time":"2023-08-17T14:24:44.808Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"    \t/usr/etc/jupyter/jupyter_notebook_config.json","time":"2023-08-17T14:24:44.809Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"    \t/root/.local/etc/jupyter/jupyter_notebook_config.json","time":"2023-08-17T14:24:44.810Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"    \t/root/.jupyter/jupyter_notebook_config.json","time":"2023-08-17T14:24:44.811Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"Writing notebook server cookie secret to /root/.local/share/jupyter/runtime/notebook_cookie_secret","time":"2023-08-17T14:24:44.831Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"Authentication of /metrics is OFF, since other authentication is disabled.","time":"2023-08-17T14:24:44.831Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"google.colab serverextension initialized.","time":"2023-08-17T14:24:44.856Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"Serving notebooks from local directory: /","time":"2023-08-17T14:24:49.405Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"Jupyter Notebook 6.4.8 is running at:","time":"2023-08-17T14:24:49.405Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"http://172.28.0.12:9000/","time":"2023-08-17T14:24:49.406Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).","time":"2023-08-17T14:24:49.406Z","v":0}
{"pid":6,"type":"jupyter","level":30,"msg":"Serving notebooks from local directory: /","time":"2023-08-17T14:24:49.430Z","v":0}
{"pid":6,"type":"jupyter","level":30,"msg":"Jupyter Notebook 6.4.8 is running at:","time":"2023-08-17T14:24:49.431Z","v":0}
{"pid":6,"type":"jupyter","level":30,"msg":"http://172.28.0.2:9000/","time":"2023-08-17T14:24:49.432Z","v":0}
{"pid":6,"type":"jupyter","level":30,"msg":"Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).","time":"2023-08-17T14:24:49.432Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"Kernel started: 5b91d383-2a5d-4d17-8f8d-5f59277ecb11, name: python3","time":"2023-08-17T14:26:06.571Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:26:29.531813: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.","time":"2023-08-17T14:26:29.531Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.","time":"2023-08-17T14:26:29.532Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:26:32.617224: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT","time":"2023-08-17T14:26:32.617Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:26:36.113889: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:26:36.114Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:26:36.787982: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:26:36.788Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:26:36.788362: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:26:36.788Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:26:36.803571: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:26:36.803Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:26:36.803974: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:26:36.804Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:26:36.804329: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:26:36.804Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:26:40.000011: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:26:40.001Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:26:40.000338: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:26:40.001Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:26:40.000550: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:26:40.001Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:26:40.000771: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:47] Overriding orig_value setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.","time":"2023-08-17T14:26:40.001Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:26:40.000816: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13664 MB memory:  -> device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5","time":"2023-08-17T14:26:40.001Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:26:40.077428: W tensorflow/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 6093750902 exceeds 10% of free system memory.","time":"2023-08-17T14:26:40.077Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:26:42.860963: F tensorflow/core/framework/tensor_shape.cc:587] Check failed: size >= 0 (0 vs. -1248091845)","time":"2023-08-17T14:26:42.861Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"KernelRestarter: restarting kernel (1/5), keep random ports","time":"2023-08-17T14:26:45.571Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"WARNING:root:kernel 5b91d383-2a5d-4d17-8f8d-5f59277ecb11 restarted","time":"2023-08-17T14:26:45.573Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:27:36.094812: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.","time":"2023-08-17T14:27:36.100Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.","time":"2023-08-17T14:27:36.101Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:27:37.142240: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT","time":"2023-08-17T14:27:37.142Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:27:41.764145: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:27:41.764Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:27:41.816688: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:27:41.816Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:27:41.826117: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:27:41.828Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:27:41.831423: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:27:41.831Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:27:41.832476: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:27:41.832Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:27:41.833383: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:27:41.833Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:27:43.665264: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:27:43.665Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:27:43.667057: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:27:43.667Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:27:43.668052: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:27:43.668Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:27:43.668828: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:47] Overriding orig_value setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.","time":"2023-08-17T14:27:43.668Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:27:43.669432: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13664 MB memory:  -> device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5","time":"2023-08-17T14:27:43.669Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:27:43.701398: W tensorflow/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 6093750902 exceeds 10% of free system memory.","time":"2023-08-17T14:27:43.701Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:27:48.308263: F tensorflow/core/framework/tensor_shape.cc:587] Check failed: size >= 0 (0 vs. -1248091845)","time":"2023-08-17T14:27:48.308Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"KernelRestarter: restarting kernel (1/5), keep random ports","time":"2023-08-17T14:27:51.576Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"WARNING:root:kernel 5b91d383-2a5d-4d17-8f8d-5f59277ecb11 restarted","time":"2023-08-17T14:27:51.576Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:31:09.645133: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.","time":"2023-08-17T14:31:09.645Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.","time":"2023-08-17T14:31:09.646Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:31:10.714033: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT","time":"2023-08-17T14:31:10.714Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:31:12.844776: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:31:12.844Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:31:12.892144: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:31:12.892Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:31:12.894009: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:31:12.894Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:31:12.896074: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:31:12.896Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:31:12.897012: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:31:12.897Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:31:12.897897: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:31:12.897Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:31:14.331805: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:31:14.332Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:31:14.332225: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:31:14.333Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:31:14.332548: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:31:14.333Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:31:14.332751: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:47] Overriding orig_value setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.","time":"2023-08-17T14:31:14.333Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:31:14.332800: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13664 MB memory:  -> device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5","time":"2023-08-17T14:31:14.333Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:31:18.149537: I tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:424] Loaded cuDNN version 8900","time":"2023-08-17T14:31:18.149Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:31:18.151181: F tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:959] Check failed: cudnnSetPoolingNdDescriptor( handle_.get(), (pooling_descriptor.mode() == dnn::PoolingMode::kMaximum ? cudnn_max_pooling_mode : CUDNN_POOLING_AVERAGE_COUNT_EXCLUDE_PADDING), propagate_nans ? CUDNN_PROPAGATE_NAN : CUDNN_NOT_PROPAGATE_NAN, nd, shape.data(), padding.data(), strides.data()) == CUDNN_STATUS_SUCCESS (3 vs. 0)","time":"2023-08-17T14:31:18.151Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"KernelRestarter: restarting kernel (1/5), keep random ports","time":"2023-08-17T14:31:18.580Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"WARNING:root:kernel 5b91d383-2a5d-4d17-8f8d-5f59277ecb11 restarted","time":"2023-08-17T14:31:18.581Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:32:58.023248: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.","time":"2023-08-17T14:32:58.024Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.","time":"2023-08-17T14:32:58.024Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:33:00.925573: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT","time":"2023-08-17T14:33:00.925Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:34:22.817441: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:34:22.817Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:34:22.920819: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:34:22.920Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:34:22.921484: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:34:22.921Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:34:22.923040: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:34:22.923Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:34:22.923548: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:34:22.923Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:34:22.924029: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:34:22.924Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:34:24.323413: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:34:24.323Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:34:24.324064: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:34:24.324Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:34:24.324360: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:996] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:34:24.324Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:34:24.324522: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:47] Overriding orig_value setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.","time":"2023-08-17T14:34:24.325Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:34:24.324601: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1635] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13664 MB memory:  -> device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5","time":"2023-08-17T14:34:24.325Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:34:29.102979: I tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:424] Loaded cuDNN version 8900","time":"2023-08-17T14:34:29.103Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:34:29.104423: F tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:959] Check failed: cudnnSetPoolingNdDescriptor( handle_.get(), (pooling_descriptor.mode() == dnn::PoolingMode::kMaximum ? cudnn_max_pooling_mode : CUDNN_POOLING_AVERAGE_COUNT_EXCLUDE_PADDING), propagate_nans ? CUDNN_PROPAGATE_NAN : CUDNN_NOT_PROPAGATE_NAN, nd, shape.data(), padding.data(), strides.data()) == CUDNN_STATUS_SUCCESS (3 vs. 0)","time":"2023-08-17T14:34:29.104Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"KernelRestarter: restarting kernel (1/5), keep random ports","time":"2023-08-17T14:34:30.587Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"WARNING:root:kernel 5b91d383-2a5d-4d17-8f8d-5f59277ecb11 restarted","time":"2023-08-17T14:34:30.587Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:34:39.499788: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.","time":"2023-08-17T14:34:39.500Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.","time":"2023-08-17T14:34:39.500Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:34:41.797362: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT","time":"2023-08-17T14:34:41.797Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"Kernel restarted: 5b91d383-2a5d-4d17-8f8d-5f59277ecb11","time":"2023-08-17T14:34:51.616Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:34:59.388548: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.","time":"2023-08-17T14:34:59.392Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.","time":"2023-08-17T14:34:59.392Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:35:01.250311: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT","time":"2023-08-17T14:35:01.250Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:36:06.143899: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:36:06.144Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:36:06.181829: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:36:06.182Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:36:06.182572: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:36:06.182Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:36:06.184389: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:36:06.184Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:36:06.184737: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:36:06.185Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:36:06.185055: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:36:06.185Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:36:07.232416: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:36:07.232Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:36:07.232868: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:36:07.233Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:36:07.233157: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:36:07.233Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:36:07.233367: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:47] Overriding orig_value setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.","time":"2023-08-17T14:36:07.233Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:36:07.233419: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13692 MB memory:  -> device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5","time":"2023-08-17T14:36:07.233Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:36:07.261994: W tensorflow/core/framework/op_kernel.cc:1828] OP_REQUIRES failed at matrix_diag_op.cc:273 : INVALID_ARGUMENT: Encountered overflow when multiplying 12884901888 with 1610612736, result: -1","time":"2023-08-17T14:36:07.262Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:36:57.056526: W tensorflow/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 6093750902 exceeds 10% of free system memory.","time":"2023-08-17T14:36:57.056Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:37:03.543980: W tensorflow/core/framework/op_kernel.cc:1828] OP_REQUIRES failed at matrix_diag_op.cc:273 : INVALID_ARGUMENT: Encountered overflow when multiplying 3046875451 with 3046875451, result: -9163294059803098215","time":"2023-08-17T14:37:03.544Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:37:04.268280: W tensorflow/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 6093750784 exceeds 10% of free system memory.","time":"2023-08-17T14:37:04.268Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:37:08.172313: W tensorflow/core/framework/op_kernel.cc:1828] OP_REQUIRES failed at matrix_diag_op.cc:273 : INVALID_ARGUMENT: Encountered overflow when multiplying 3046875392 with 3046875392, result: -9163294419334397952","time":"2023-08-17T14:37:08.172Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:38:16.953443: I tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:432] Loaded cuDNN version 8900","time":"2023-08-17T14:38:16.953Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:38:16.953795: F tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:983] Check failed: cudnnSetPoolingNdDescriptor( handle_.get(), (pooling_descriptor.mode() == dnn::PoolingMode::kMaximum ? cudnn_max_pooling_mode : CUDNN_POOLING_AVERAGE_COUNT_EXCLUDE_PADDING), propagate_nans ? CUDNN_PROPAGATE_NAN : CUDNN_NOT_PROPAGATE_NAN, nd, shape.data(), padding.data(), strides.data()) == CUDNN_STATUS_SUCCESS (3 vs. 0)","time":"2023-08-17T14:38:16.954Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"KernelRestarter: restarting kernel (1/5), keep random ports","time":"2023-08-17T14:38:18.617Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"WARNING:root:kernel 5b91d383-2a5d-4d17-8f8d-5f59277ecb11 restarted","time":"2023-08-17T14:38:18.617Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:41:46.795637: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.","time":"2023-08-17T14:41:46.795Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.","time":"2023-08-17T14:41:46.795Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:41:47.897333: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT","time":"2023-08-17T14:41:47.897Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:41:50.174067: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:41:50.174Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:41:50.217161: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:41:50.217Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:41:50.217535: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:41:50.217Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:53:15.973272: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:53:15.973Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:53:15.977064: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:53:15.977Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:53:15.977438: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:53:15.977Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:53:17.404160: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:53:17.404Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:53:17.405355: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:53:17.405Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:53:17.406091: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T14:53:17.406Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:53:17.406821: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:47] Overriding orig_value setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.","time":"2023-08-17T14:53:17.406Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:53:17.407403: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13692 MB memory:  -> device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5","time":"2023-08-17T14:53:17.407Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:53:17.447492: W tensorflow/core/framework/op_kernel.cc:1828] OP_REQUIRES failed at matrix_diag_op.cc:273 : INVALID_ARGUMENT: Encountered overflow when multiplying 9984734776 with 1248091847, result: -5984878005326580344","time":"2023-08-17T14:53:17.447Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:56:00.832626: I tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:432] Loaded cuDNN version 8900","time":"2023-08-17T14:56:00.832Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 14:56:00.832804: F tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:983] Check failed: cudnnSetPoolingNdDescriptor( handle_.get(), (pooling_descriptor.mode() == dnn::PoolingMode::kMaximum ? cudnn_max_pooling_mode : CUDNN_POOLING_AVERAGE_COUNT_EXCLUDE_PADDING), propagate_nans ? CUDNN_PROPAGATE_NAN : CUDNN_NOT_PROPAGATE_NAN, nd, shape.data(), padding.data(), strides.data()) == CUDNN_STATUS_SUCCESS (3 vs. 0)","time":"2023-08-17T14:56:00.833Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"KernelRestarter: restarting kernel (1/5), keep random ports","time":"2023-08-17T14:56:03.623Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"WARNING:root:kernel 5b91d383-2a5d-4d17-8f8d-5f59277ecb11 restarted","time":"2023-08-17T14:56:03.623Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 15:16:21.348874: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.","time":"2023-08-17T15:16:21.349Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.","time":"2023-08-17T15:16:21.349Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 15:16:22.465628: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT","time":"2023-08-17T15:16:22.465Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 15:16:24.726483: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T15:16:24.726Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 15:16:24.772576: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T15:16:24.772Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 15:16:24.772996: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T15:16:24.773Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 15:16:24.774456: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T15:16:24.774Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 15:16:24.774875: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T15:16:24.775Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 15:16:24.775178: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T15:16:24.775Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 15:16:26.155580: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T15:16:26.155Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 15:16:26.156925: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T15:16:26.157Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 15:16:26.157283: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355","time":"2023-08-17T15:16:26.157Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 15:16:26.157467: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:47] Overriding orig_value setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.","time":"2023-08-17T15:16:26.157Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 15:16:26.157517: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13692 MB memory:  -> device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5","time":"2023-08-17T15:16:26.158Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 15:16:26.192012: W tensorflow/core/framework/op_kernel.cc:1828] OP_REQUIRES failed at matrix_diag_op.cc:273 : INVALID_ARGUMENT: Encountered overflow when multiplying 12885103520 with 1610637940, result: -1","time":"2023-08-17T15:16:26.192Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 15:18:07.005069: W tensorflow/core/framework/op_kernel.cc:1828] OP_REQUIRES failed at matrix_diag_op.cc:273 : INVALID_ARGUMENT: Encountered overflow when multiplying 12885103520 with 1610637940, result: -1","time":"2023-08-17T15:18:07.005Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 15:19:44.626474: I tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:432] Loaded cuDNN version 8900","time":"2023-08-17T15:19:44.626Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"2023-08-17 15:19:44.626614: F tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:983] Check failed: cudnnSetPoolingNdDescriptor( handle_.get(), (pooling_descriptor.mode() == dnn::PoolingMode::kMaximum ? cudnn_max_pooling_mode : CUDNN_POOLING_AVERAGE_COUNT_EXCLUDE_PADDING), propagate_nans ? CUDNN_PROPAGATE_NAN : CUDNN_NOT_PROPAGATE_NAN, nd, shape.data(), padding.data(), strides.data()) == CUDNN_STATUS_SUCCESS (3 vs. 0)","time":"2023-08-17T15:19:44.627Z","v":0}
{"pid":7,"type":"jupyter","level":30,"msg":"KernelRestarter: restarting kernel (1/5), keep random ports","time":"2023-08-17T15:19:45.628Z","v":0}
{"pid":7,"type":"jupyter","level":40,"msg":"WARNING:root:kernel 5b91d383-2a5d-4d17-8f8d-5f59277ecb11 restarted","time":"2023-08-17T15:19:45.629Z","v":0}