server: Model loaded via `model repository` api does not appear after querying it with `v2/repository/index` endpoint
Description
I’ve loaded a model via v2/repository/models/simple/load endpoint.
But when querying v2/repository/index endpoint I get a [] as a responce.
Triton Information
What version of Triton are you using?
2.42.0
Are you using the Triton container or did you build it yourself?
Triton container, version nvcr.io/nvidia/tritonserver:24.01-py3
To Reproduce
- I’ve took this model: https://github.com/triton-inference-server/server/tree/main/docs/examples/model_repository/simple
- Loaded it with with python script using
tritonclient
model_name = "simple"
config_path = models_repository[model_name]["config"]
model_path = models_repository[model_name]["model"]
with open(model_path, "rb") as f:
model_bytes = f.read()
json_obj = _pbtxt_to_json(config_path)
triton_client.load_model(
model_name=model_name,
config=json_obj,
files={
"file:1/model.graphdef": model_bytes,
},
)
- Then
triton_client.get_model_repository_index()
# returns: []
Describe the models (framework, inputs, outputs), ideally include the model configuration file (if using an ensemble include the model configuration file for that as well). Model mentioned above
Expected behavior I expect than this code:
triton_client.get_model_repository_index()
will return responce according to this specification
$repository_index_response =
[
{
"name" : $string,
"version" : $string #optional,
"state" : $string,
"reason" : $string
},
…
]
About this issue
- Original URL
- State: open
- Created 3 years ago
- Comments: 21 (10 by maintainers)
It will generally depend on the backend and how it handles things. For the python backend- model instances are in seperate processes so memory would be reclaimed. For In-Process backends like tensorflow and pytorch mileage can very on how quickly and if all memory is reclaimed. For tensorflow specifically we have seen memory being held.
@ogvalt For a temporary workaround you can find: https://github.com/triton-inference-server/core/pull/340
Need to finalize the change in behavior - but in case you’d like to see it sooner than later.
@ogvalt - we’re discussing internally and will get back on ETA.
@nnshah1 FYI: I’ve run your code and got:
Sanity test - checked
@nnshah1 logs above obtained by launching everything my way with empty repository
quick update - I believe I’m able to reproduce what you are describing - will investigate -
@nnshah1 Understood, I’m working on launching your code. Meanwhile here is a server log you asked for