aim: Aim is very slow since 3.12.0

🐛 Bug

I observed a huge slow down when using Aim since updating to 3.12.0. I did not have this issue with previous 3.11.2. Launching aim up takes a bit less than 1 minute. Creating a run tracker through Python with aim.Run is very long (1 to 2 minutes). Aim UI is very slow, for instance I cannot go to the Images tab, it keeps loading and then my browser tab crashes saying “Something went wrong”. I tried on both SSD and HDD.

To reproduce

Install Aim 3.12.0 or greater.
Track metrics and images in the code.
Launch Aim UI with aim up.
Launch the code.

Expected behavior

Starting Aim UI with aim up shouldn’t take that long, and Aim UI itself shouldn’t make my browser crash.

Environment

Aim Version: >=3.12.0
Python version: tested on both 3.8.13 and 3.10.7
pip version: 22.2.2
OS: Ubuntu

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 1
Comments: 22 (10 by maintainers)

Most upvoted comments

Hey @Aonnghus! We are on the way to completely re-design the locking mechanism in Aim. The fix for the performance issue related to locks will be shipped along. We target to release the new version at the end of next week.

alberttorosyan on Nov 2, 2022

Yes such warnings appeared in stdout

Got it. Thanks! We’ll include this change in the v3.14.0 as well, as a workaround for now. In the meanwhile we’ll try to improve the filesystem identification mechanism, as clearly FileLocks can be used for a case that you’ve described, but we still opt to use SoftFileLocks.

mihran113 on Sep 26, 2022

hey @Aonnghus! Sorry for the late response. Taking into consideration the fact that the performance degradation was started from v3.12.0, our guess is that it might have to do something with the new locking mechanism we introduced (we started to use SoftFileLock instead of FileLock for some filesystems). We have released a dev version of aim which forces to use FileLocks only if __AIM_FORCE_USE_FILELOCK__ environment variable is set. Could you please install it and test it out to see if everything works as expected?

pip install aim==3.14.0.dev1
export __AIM_FORCE_USE_FILELOCK__=1

Then just try out your scripts and aim up command to see if it works fine as was with earlier versions.

mihran113 on Sep 25, 2022