rasa: Including `RulePolicy` leads to infinite max_history being used to deduplicate training trackers

Sorry, it was the wrong link 🙈 Updated it with the correct one 👍🏻

As for the quick fix using caching, do you think it’s safe with regards to Rasa’s usage of RAM?

We can set a max_size for the cache 👍🏻 It’s just a couple jsons and their dict representations so I’d expect it be tiny.

So next steps:

• create PR with cache (I include details about memory usage)
• separate issue for overall performance?

didn’t really dive deeper into it tbh. I used the max_history which was specific in the config of the AugmentedMemoizationPolicy for Sara. I guess 5 might be too short for some rules?

Posting the details for the memory usage here as I haven’t created the PR yet and need to put the results somewhere 😄

• With Sara I’ve a 248 cached items with 1 424 684 hits after the training.
• The cache stores a mapping of call params and output. 248 items have a total size of 862 920 (88 992 + 773 928) bytes. With a max_size of 1000 items for the cache we have max memory usage of 3 479 516 bytes (3.32 MiB) which seems okay to me 👍🏻

Investigation results on rasa-demo:

• the contradiction checks takes the huge chunk of time (1.2 seconds for the actual training, 13 seconds for the contradiction check)
• Setting a mix history for the RulePolicy doesn’t change this
• Configuring a max_history for the rule generation leads to contradictions 🤔
• I also profiled the RulePolicy training (see results here - viewable with snakeviz). the results stress that the contradiction check takes the longest amount of time. In my opinion this clearly shows that it’s not related to max_history but rather to the amount of predictions. The biggest part seems to be caused by the json.loads calls. If we actually wanna do something then we need to revisit the mechanics of the policy itself (e.g. how data is serialized) and how we can speed up predictions. This should in my opinion be done in a separate issue.

A quick fix for this issue would to use lru_cache (with max_size > 1000) on _rule_key_to_state. By doing this I lowered the contradiction check time from 13 seconds to ~4 seconds. The issue is that it rather addresses symptoms than causes.

Yes, you did! Rule trackers are already deduplicated separately. So your 2nd approach would be completely feasible.

I’ll do some debugging later to find out where we spend so much time exactly (during deduplication or featurization) and then maybe a PR so we have something more tangible to discuss.

That makes more sense I think 🤔 I think what you’re then suggesting to set max_history here in an automated fashion then? What do you think of that @samsucik ?