server: Interaction of timeouts, ensemble scheduler and oldest sequence scheduler causes state leakage
Description We’re still in progress in diagnosing this issue but we’re seeing some strange behaviour and would like to get some feedback/ideas from others.
We’re using a setup that relies on two separate models using the oldest sequence scheduler combined with the ensemble model. Essentially
Input -> Model_1 -> Model_2 -> Output
Where model 1 and model 2 use implicit state management with the PyTorch backend, they have an initial state but then use the output from the model to update the state.
Our issue tends to arise when the triton is under load, if we use up all the candidate sequence slots - then we don’t send END flags on any of them and instead let triton time them out. We then see that Model_2 is in a bad state, in this state, if we run requests against model 2, it’s possible to leak information across requests and corrupt the outputs of the model. It tends to look something like:
Here we’re displaying the outputs of the model, we’ve corrupted one batch with NaN values and used random inputs for others - however the NaN values leak into the next request sent to the server (here the batches are run sequentially so leaking happens across executions rather than within one)
Given all this, it feels like something isn’t quite working correctly with the interaction between the ensemble, the sequence timeout and implicit state management.
Triton Information What version of Triton are you using?
We’ve reproed on NGC version 22.12, 23.10, 24.02
Are you using the Triton container or did you build it yourself?
We’ve reproed on containers based on the triton container and on versions built by us.
To Reproduce
We’re working on this, currently we only can reproduce with our own models and configuration which we can’t share. Hopefully we can find a repro with some simpler models. The basic setup is two sequential models using implicit state management with the PyTorch backend - connected together with the ensemble scheduler. The you need to timeout some/all of the candidate sequences to get Model_2 into a bad state. Once Model_2 is in this state you can fairly reliably see the state corruption reported above.
We still can’t reliably reproduce this state however so we can’t provide much more info.
Expected behavior We shouldn’t be able to affect the outputs of the model with the inputs from a different client.
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 1
- Comments: 15 (11 by maintainers)
Thanks a lot. Definitely appreciate the help here. On (1) the review and test help will be good, as I struggled in the past to get the test setup working on my Maschine.
on (2) I need to clarify that we actually haven’t noticed the wrong behaviour with the patch above yet, it’s just that the patch deals with a problem with cancellation so we have a belief that this isn’t quite fixing the problem. for us the sequence batcher code, potentially couple with ensembles is quite complex and there are a lot of edge cases that can happen in that combination. We just want to be double sure that no edge cases are missed.