serving: Reshape layer throws std::bad_alloc

System information

OS:
- Windows 11, 21H2, build 22000.856
- WSL2, Ubuntu 22.04 – allocated 8Gb in .wslconfig
TF Serving installed using docker, version 2.9.1
Hardware:
- AMD Ryzen 5 3600
- RAM 16Gb

Describe the problem

I’m facing issue with Reshape layer in TF serving 2.9.1. It throws following to docker logs.

pythonproject-serving-1  | 2022-09-01 21:07:16.379819: I tensorflow_serving/model_servers/server.cc:442] Exporting HTTP/REST API at:localhost:8501 ...
pythonproject-serving-1  | [evhttp_server.cc : 245] NET_LOG: Entering the event loop ...
pythonproject-serving-1  | terminate called after throwing an instance of 'std::bad_alloc'
pythonproject-serving-1  |   what():  std::bad_alloc
pythonproject-serving-1  | /usr/bin/tf_serving_entrypoint.sh: line 3:     7 Aborted                 tensorflow_model_server --port=8500 --rest_api_port=8501 --model_name=${MODEL_NAME} --model_base_path=${MODEL_BASE_PATH}/${MODEL_NAME} "$@"

Also tried rolling back to older versions: the same mock model WORKS ON TF SERVING 2.8.2

Exact Steps to Reproduce

Mock model to demonstrate issue, TF is 2.9.1

import numpy as np
import tensorflow as tf

model = tf.keras.Sequential([
    tf.keras.layers.Input(shape=(128, 64, 3)),
    tf.keras.layers.Reshape((12, 2048))
])

model.compile()

# ensure it works both on WSL and Windows
model.predict([np.ones(shape=(128, 64, 3)).tolist()])

model.save("./.data/models/v2_96acc/5")

Then serving with docker-compose with following config:

version: "3.8"
  serving:
    image: tensorflow/serving:2.9.1
    restart: on-failure
    ports:
      - "8501:8501"
    volumes:
      - "./.data/models:/models"
    environment:
      MODEL_NAME: v2_96acc

And evaluating results

import json
import numpy as np
import requests

image = {
    "inputs": [np.ones(shape=(128, 64, 3)).tolist()]
}

res = requests.post('http://localhost:8501/v1/models/v2_96acc/versions/5:predict', data=json.dumps(image))

print(len(res.json()['outputs']))

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 7
Comments: 17 (2 by maintainers)

Most upvoted comments

I’m also receiving this error when using a predict request with the tensorflow/serving:latest (2.9.1) Docker image. Observed this error for two different NLP models on TF hub.

Fixed by switching to tensorflow/serving:2.8.2.

I’m running this using the WSL2 Docker backend - I tried adjusting the WSL2 settings to increase the memory limit, but the error still occurs with 2.9.1.

+10

williamih on Sep 5, 2022

issue persists with final 2.10.0

jc-louis on Sep 13, 2022

I am trying to deploy a BERT-based model and encountering the same issue. The problem is solved after downgrading tensorflow/serving to 2.8.2. Thank you!

liziyan1997 on Sep 8, 2022

The same issue deploying bert_multi_cased_L-12_H-768_A-12 from TF-Hub using the corresponding bert_multi_cased_preprocess, i.e.

inp = tf.keras.layers.Input(shape=(), dtype=tf.string, name='input')
preprocess = hub.KerasLayer('https://tfhub.dev/tensorflow/bert_multi_cased_preprocess/3', name='preprocessing')
encoded_inp = preprocess(inp)
bert = hub.KerasLayer('https://tfhub.dev/tensorflow/bert_multi_cased_L-12_H-768_A-12/4', trainable=True, name='bert_encoder')
output = bert(encoded_inp)
network = tf.keras.Model(inp, output, name='bert_classifier')
model = tfm.nlp.models.BertTokenClassifier(network, 5)

Before downgrading to 2.8.2 as mentioned above I have just tried the latest pre-release and the bug is still persists in 2.10.0-rc3-gpu!

rkly on Sep 8, 2022

@singhniraj08 Thanks, I don’t have chance to test it near future.

I see at least someone replied that it has been fixed, so I’m closing it.

Saoqq on Dec 6, 2022

I’m facing the same issue and solved it with @williamih 's solution! This issue should be fixed in the future release.

I’m also receiving this error when using a predict request with the tensorflow/serving:latest (2.9.1) Docker image. Observed this error for two different NLP models on TF hub. Fixed by switching to tensorflow/serving:2.8.2. I’m running this using the WSL2 Docker backend - I tried adjusting the WSL2 settings to increase the memory limit, but the error still occurs with 2.9.1.

Good to hear! Credit for the downgrading fix goes to @Saoqq - I just did that based on @Saoqq’s original post!

williamih on Sep 5, 2022