ranking: When using ELWC - OP_REQUIRES failed at example_parsing_ops.cc:91 : Invalid argument: Could not parse example input

Hi @ramakumar1729 ,

Sorry to comment on this closed issue, but I thought I would share my latest (positive & successful) findings regarding making requests to predict serving REST API. Perhaps others may find them useful:

TL;DR

I was able to successfully POST serialized & base64 encoded ELWC proto to the predict REST API and get the expected predictions. These predictions match exactly the predictions that I get if I make a gRPC request using the same ELWC proto to the TensorFlow model server over gRPC.

This gave me the confidence in behavior parity that making inference requests over HTTP vs gRPC produces consistent results for the same ELWC.

Details

Previously in https://github.com/tensorflow/ranking/issues/189#issuecomment-620256362, I was creating a tensor_proto out of the serialized ELWC proto, then serializing the tensor_proto & base64 that, which then I was POSTing to the API.

I found out that I should not create tensor_proto out of serialized ELWC proto string. What needs to be serialized & base64 encoded is the ELWC proto itself. As a result, my cURL looks like as before before, the difference is that the b64 string holds the ELWC proto:

curl -H "Content-Type: application/json" -X POST \
http://192.168.99.100:8501/v1/models/tf_ranking_v10/versions/1589680164:predict \
-d '{"instances": [{"b64": "CqABCp0BCiQK.... TRUNCATED"}]}'

We can be a little more descriptive by specifying the signature_name and the receiver_name (whatever was defined when creating serving_input_receiver_fn):

curl -H "Content-Type: application/json" -X POST \
http://192.168.99.100:8501/v1/models/tf_ranking_v10/versions/1589680164:predict \
-d '{"signature_name": "predict", "instances": [{"input_ranking_data": {"b64": "CqABCp0BCiQK.... TRUNCATED"}}]}'

The predictions from the above cURL requests match the gRPC response prediction after making the gRPC request as follows:

import grpc
from tensorflow_serving.apis import predict_pb2
from tensorflow_serving.apis import prediction_service_pb2_grpc
from tensorflow_serving.apis import input_pb2
from google.protobuf import text_format
from google.protobuf.json_format import MessageToDict


EXAMPLE_LIST_WITH_CONTEXT_PROTO = text_format.Parse(
      """
     examples {
          ....
     }
    context {
         .....
    }
    """, input_pb2.ExampleListWithContext())

example_list_with_context_proto = EXAMPLE_LIST_WITH_CONTEXT_PROTO.SerializeToString()
tensor_proto = tf.make_tensor_proto(example_list_with_context_proto, dtype=tf.string, shape=[1])

timeout_in_secs = 3
request = predict_pb2.PredictRequest()
request.inputs['input_ranking_data'].CopyFrom(tensor_proto)
request.model_spec.signature_name = 'predict' 
request.model_spec.name = 'tf_ranking_v10'

channel = grpc.insecure_channel("0.0.0.0:8500")
stub = prediction_service_pb2_grpc.PredictionServiceStub(channel)                            

grpc_response = stub.Predict(request, timeout_in_secs)
unpacked_grpc_response = MessageToDict(grpc_response, preserving_proto_field_name = True)

print(unpacked_grpc_response['outputs']['output']['float_val'])

@azagniotov Thanks for sharing your experience in successfully serving the ranking model with serialized ELWC inputs~ This will be a useful reference for others trying to do the same.

@davidmosca

TL;DR

I took your code as-is and ran it. I was able successfully (i.e.: I did not observe your aforementioned error and got back a list of predictions) to make a request to the Docker container over gRPC using TF/TFR v2.1.0 and also using v2.3.0 as another attempt, which is the versions you are on.

Question

Are you pinning down the docker container version when pulling it down or are you getting the latest? i.e.: docker pull tensorflow/serving:2.3.0 or docker pull tensorflow/serving ? The latter will pull down the latest tag, which may or may not be what you want. I remember some time ago when I was not pinning down the container version and one day things stopped working for me because I probably pulled down new functionality or changes from the upstream. So, I had to fix the Docker version.

In my Docker start command, I also explicitly start the version that I am on, e.g.:

docker run -t --rm -p 8500:8500 -p 8501:8501 \
-v "/home/azagniotov/tensorflow_ranking_model:/models/tensorflow_ranking_model" \
-e MODEL_NAME="tensorflow_ranking_model" tensorflow/serving:2.1.0 &

MORE DETAILS

tensorflow-serving-api==2.1.0 (or 2.3.0)
tensorflow==2.1.0 (or 2.3.0)
tensorflow-ranking==0.3.0
docker pull tensorflow/serving:2.1.0 (or 2.3.0)

In the following code I am using to generate ranking_serving_input_receiver_fn when saving the model. Not sure if this helps, but I thought to show how I do it:

# Dicts, e.g.: 
# spec['feature_name_a'] = tf.feature_column.numeric_column('feature_name_a', shape=[1], default_value=-1, dtype=tf.float32)
context_feature_spec = tf.feature_column.make_parse_example_spec(context_feature_columns().values())
example_feature_spec = tf.feature_column.make_parse_example_spec(example_feature_columns().values())

# I am using `input_ranking_data` name as the `receiver_name`
serving_input_receiver_fn = tfr.data.build_ranking_serving_input_receiver_fn(
        data_format=tfr.data.ELWC,
        list_size=_LIST_SIZE,
        default_batch_size=None,
        size_feature_name='example_list_size',
        receiver_name='input_ranking_data',
        context_feature_spec=context_feature_spec,
        example_feature_spec=example_feature_spec)

# ranker == tf.estimator.Estimator
ranker.export_saved_model(export_dir_base='tensorflow_ranking_model',
                          serving_input_receiver_fn=serving_input_receiver_fn,
                          assets_extra={'tf_serving_warmup_requests': 'tf_serving_warmup_requests'},
                          as_text=False,
                          checkpoint_path=None,
                          experimental_mode=tf.estimator.ModeKeys.PREDICT)

TF-Ranking model’s predict signature

MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:

signature_def['predict']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['input_ranking_data'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: input_ranking_tensor:0
  The given SavedModel SignatureDef contains the following output(s):
    outputs['output'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, -1)
        name: groupwise_dnn_v2/accumulate_scores/div_no_nan:0
  Method name is: tensorflow/serving/predict

@ramakumar1729, I got the making request to the REST API to work when hitting predict API:

import random
import base64
from random import randint

def _float_feature(value):
    return tf.train.Feature(float_list=tf.train.FloatList(value=[value]))

def _int_feature(value):
    return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))


def create_example():
    feature_names = ['f0', 'f1', 'f2']
    features = {name: _float_feature(random.uniform(300, 10)) for name in feature_names}
    features['ctx_f0'] = _int_feature(123)
    return tf.train.Example(features=tf.train.Features(feature=features))

def create_serialized_example_list(num_of_examples):
    example_protos = []
    for x in range(num_of_examples):
        example_proto = create_example()
        example_proto_serialized = example_proto.SerializeToString()

        example_protos.append(example_proto_serialized)
        
    return tf.make_tensor_proto(example_protos, dtype=tf.string)


tensor_proto = create_serialized_example_list(20)
tensor_proto_serialized = tensor_proto.SerializeToString()
tensor_proto_base64_bytes = base64.b64encode(tensor_proto_serialized)
tensor_proto_base64 = tensor_proto_base64_bytes.decode('utf-8')

print(tensor_proto_base64)

and the cURL call is:

the tensor_proto_base64 holds the CAcSBBICCBRCQwpBCg4K..... string

curl -H "Content-Type: application/json" \
-X POST http://192.168.99.100:8501/v1/models/tf_ranking_v9/versions/1588000040:predict \
-d '{"instances": [{"b64": "CAcSBBICCBRCQwpBCg4K..... TRUNCATED"}]}'

The results are as follows:

{
    "predictions": [[0.0222161338, 0.0222163089, 0.02221632, 0.0222162828, 0.0222162828, 0.0222162828, 0.0222162828, 0.0222162828, 0.0222162828, 0.0222162828, 0.0222162828, 0.0222162828, 0.0222162828, 0.0222162828, 0.0222162828, 0.0222162828, 0.0222162828, 0.0222162828, 0.0222162679, 0.0222161338]
    ]
}

What’s interesting though is that I always get this ^ same result set, no matter what the input is (i.e.: what the feature numeric values are). I can make 3 different cURL requests with a different b64 string as a payload, but the predictions will always be as the aforementioned ^

In fact, I can use completely bogus example feature names (i.e.: apples, oranges & lemons) when constructing the tensor_proto, the API still is able to return the same above predictions. Should not the model complain that the expected features f0, f1 and f2 are not there?

Any thoughts?