DeepSpeech: Memory leakage in python client

Have I written custom code (as opposed to running examples on an unmodified clone of the repository): No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux 4.9.0-11-amd64 x86_64
TensorFlow version (use command below): 1.14.0-rc1-22-gaf24dc91b5 1.14.0
Python version: 3.6.9
Bazel version (if compiling from source):N/A
GCC/Compiler version (if compiling from source):N/A
CUDA/cuDNN version:N/A (using CPU for inference)
GPU model and memory:N/A (using CPU for inference)
Exact command to reproduce: using python client (deepspeech 0.6.1)

from deepspeech import Model
model = Model(pbmm_local_path, 512)
model.enableDecoderWithLM(
            lm_local_path,
            trie_local_path,
            alpha,
            beta,
        )
del model

Description: Repeated calls to the above command causes about 4 mb memory leak per 10 calls. I have an application where variously different models are loaded for ASR depending on the context and purpose. After about 2-3 thousand requests or so, the machine runs out of memory. Only restarting the process will free the memory.

I saw that there is this thread https://github.com/mozilla/DeepSpeech/issues/2403, which confirmed leakage on C++, python and .Net client, but only .Net is fixed?

I’m wondering if this is still a known issue on python, and whether there are plans to fix it? If not, any recommendations or workarounds?

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 48 (6 by maintainers)

Most upvoted comments

Thanks @lissyx for the detailed analysis. Looks like 0.7.3 is pretty clean, and just some allocations in python that makes it seems like there is leakage. I think we can close this issue.

For my use case, things should improve dramatically when I switch to 0.7.3.

khu834 on Jul 16, 2020

Got it, will do this week.

On Tue, Jun 16, 2020 at 3:26 AM lissyx notifications@github.com wrote:

(also on 0.7.3 please)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mozilla/DeepSpeech/issues/2967#issuecomment-644677800, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACJEOTNK4MMIFGNMZKQTQFLRW5CFDANCNFSM4MWZUONQ .

khu834 on Jun 16, 2020

I’ve been getting this warning in the console when training with DeepSpeech 0.7.3
swig/python detected a memory leak of type 'Alphabet *', no destructor found. 
It seems the valgrind check ran earlier reported an issue with TF but maybe this is also part of it?

If it’s a separate issue I can open another ticket instead.

This was taken care of in #3049

lissyx on Jun 9, 2020

Here is the info on setting it up

kdavis-mozilla on Jun 4, 2020