crawlee-python: Memory usage average periodically printed from `AutoscaledPool` seems wrong
This issue was observed on the Apify platform - see for example https://api.apify.com/v2/logs/g7piQaNbAdteXnkFC. The problem is that the average memory usage starts at 0 and suddenly hops to 1.0. I can testify that the increase was way more gradual 🙂
2024-11-06T15:11:50.471Z [crawlee._autoscaling.autoscaled_pool] INFO current_concurrency = 1; desired_concurrency = 1; cpu = 0.581; mem = 0.0; event_loop = 0.227; client_info = 0.0
2024-11-06T15:12:50.474Z [crawlee._autoscaling.autoscaled_pool] INFO current_concurrency = 0; desired_concurrency = 1; cpu = 0.491; mem = 0.0; event_loop = 0.1; client_info = 0.0
2024-11-06T15:13:50.487Z [crawlee._autoscaling.autoscaled_pool] INFO current_concurrency = 0; desired_concurrency = 1; cpu = 0.671; mem = 0.0; event_loop = 0.104; client_info = 0.0
2024-11-06T15:14:50.496Z [crawlee._autoscaling.autoscaled_pool] INFO current_concurrency = 0; desired_concurrency = 3; cpu = 0.451; mem = 0.206; event_loop = 0.039; client_info = 0.0
2024-11-06T15:15:50.491Z [crawlee._autoscaling.autoscaled_pool] INFO current_concurrency = 0; desired_concurrency = 1; cpu = 0.242; mem = 1.0; event_loop = 0.0; client_info = 0.0
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 15 (3 by maintainers)
Next time add
Closes #629to the PR description so its linked and the issue gets closed automatically. That way the issue is also referenced in the changelog.Random thought - maybe we could just not log the number, or at least change the message from
mem = 0.42to something less… suggestive?There are bunch of issues connected to the mentioned “mem” measurement. It works ok-ish, but I found some issues that can make it’s value wrong in some cases.
Interpretation: Printed mem is not memory usage ratio. It is the time weighted ratio of samples when memory usage was higher than some defined threshold (0.9 I think). That makes it very hard to interpret by bare eye as it does not make much sense to human understanding especially with the “sampling luck”, which basically means that this measurement can skip some samples based on some timing context.
Example (for simplification lets expect all samples to have same time weight): So for example if memory usage ratio was 0.89 for couple of samples, then mem would show 0. [0.89,0.89,0.89,0.89,0.89,0.89] -> mem=0 Then memory usage ratio jumps to 91 % and depending on the “sampling luck” of the print function it can directly jump to mem = 1 or some intermediate 0<mem<1.
[0.89,0.89,0.89,0.91,0.91,0.91] -> mem=0.5 [0.91,0.91,0.91,0.91,0.91,0.91] -> mem=1
Sample time order Some code does some expectations about sample order of the sampling snapshot. But I did bunch of experiments and sample order is not guaranteed! Commonly sample snapshot is ordered from oldest to newest. But it is not enforced and sometimes this expectation is not valid. This can lead to errors in calculation when “sample duration” is negative, because previous can be newer than current
Some minor details Fake time duration for same samples When two measurement snapshots are from the same time, then they are redundant information, but there is an constant that gives them “fake duration of 0.001” and thus such samples can taint further calculation. It is small number and thus I doubt it has any significant impact, but why have it there at all? I think it is there to avoid division by zero down the line, but there are other ways how to handle that, for example:
Unnecessary reverse Even though most of the code expects time ordered smaples at some point one processing step reverses the order. This makes it inconsistent ordering within the same function as it can under different circumstances return normally ordered samples