prefect: CLI registration inconsistently bumps flow version even when metadata hasn't changed

Opened from the Prefect Public Slack Community

dkmarsh: Hi Everyone, I’m having an issue with registering flows. Whenever I register the same flow, it bumps the version, even if the metadata is unchanged. I’ve tried using the CLI as well as flow.register(). I’ve also tried using flow.register(‘project_1’, idempotency_key=flow.serialized_hash()) and it still bumps the version. I am using a local agent, local storage and the cloud backend. Is there another configuration that needs to be set so this doesn’t happen? Or should this all be done in docker?

kevin701: Hey <@U022J2DB0UR>, I don’t think moving to Docker will do this. Will ask the team to be sure but what is your use case that you need the version pinned? We might be able to solve this some other way.

ranugoldan: I once tried with cli command prefect register flow it doesn’t bump version, but with prefect register it does

kevin701: Actually that’s right <@ULCV623AT>! I know some people on the other side have been asking why their flow is not bumping version 😅

amanda.wee: <@U01QEJ9PP53> when should the flow version be bumped? In my ECS-based setup with a bunch of flows packaged together with a local agent, I build the Docker image each time we make a change, even if it is only to tweak a single task. When the Docker container starts up, it runs a shell script that runs the Python scripts that define and register the flows (i.e., not using the CLI). Like what <@ULCV623AT> observed, with serialized_hash() the flow versions kept increasing even if it was just because the ECS task was restarted, thereby starting up the Docker container again.

My suspicion was that some Docker container metadata was changing each time, e.g., something host-related, so the serialized hash computation changed. My solution was to write my own hash function that took into account the dependencies (including Prefect version) and the code content, but I wonder if that is overkill.

kevin701: Hey <@U01DJDK6AQ7>, correct me if I’m wrong but I think what you’re referring to is that DockerStorage was not respecting the cache and rebuilding everything. Have you seen this <https://github.com/PrefectHQ/prefect/pull/4584|recent PR> that fixed it?.

amanda.wee: No, I’m using S3 storage. The Docker image is for ECS, not for Prefect.

kevin701: Oh I see what you mean. Ok I’ll ask the team and get back to you

dkmarsh: <@U01QEJ9PP53> It was my understanding that the version should only be bumped when the metadata changes. My use case is to automate the registering of flows once they are added or changed by a developer. I wrote a function that looks in a directory and collects all flows and registers them. I would like to not have the versions bumped on flows that have already been registered and have had no changes.

Interestingly enough, I tried testing again this morning and discovered that in using the CLI or python api to register, the version will get bumped every other time. For example, if I run prefect register --project Project1 -p src/register.py -n Flow1 it will register the flow, then if I run the same command a few seconds later, it will skip with the message: “Skipped (metadata unchanged)”. However, running it a third time, it will register the flow as version 2. So it seems to be skipping the registration as desired every other time.

znicholasbrown: Hi <@U022J2DB0UR> - this sounds like an issue with CLI registration; I’m going to open an issue from this thread for the Core team to look into.

znicholasbrown: <@ULVA73B9P> open “CLI registration inconsistently bumps flow version even when metadata hasn’t changed”

Original thread can be found here.