apiops: [BUG] Publisher is failing with "409. Content is '{"error":{"code":"PessimisticConcurrencyConflict","message":"Operation on the API is in progress","details":null}}'" When publishing on an already published APIM

Release version

v2

Describe the bug

When deploying the publisher for a second time, the pipeline is failing with the following error at several points:

2022-11-14T12:31:58.0180514Z       System.InvalidOperationException: HTTP request to URI https://management.azure.com/subscriptions/***/resourceGroups/rg/providers/Microsoft.ApiManagement/service/tapim/apis/api?api-version=2021-12-01-preview failed with status code 409. Content is '{"error":{"code":"PessimisticConcurrencyConflict","message":"Operation on the API is in progress","details":null}}'.
2022-11-14T12:31:58.0182453Z          at common.HttpPipelineExtensions.Validate(Response response, Uri requestUri) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/common/Http.cs:line 85
2022-11-14T12:31:58.0183502Z          at common.HttpPipelineExtensions.PutResource(HttpPipeline pipeline, Uri uri, JsonObject resource, CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/common/Http.cs:line 71
2022-11-14T12:31:58.0184544Z          at publisher.Program.<>c__DisplayClass17_0.<<GetPutRestResource>b__0>d.MoveNext() in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Program.cs:line 229
2022-11-14T12:31:58.0185500Z       --- End of stack trace from previous location ---
2022-11-14T12:31:58.0186715Z          at publisher.Api.PutApi(ApiName apiName, ApiInformationFile apiInformationFile, ApiSpecificationFile specificationFile, JsonObject configurationApiJson, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Api.cs:line 273
2022-11-14T12:31:58.0187971Z          at publisher.Api.<>c__DisplayClass23_0.<<ProcessArtifactsToPut>b__4>d.MoveNext() in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Api.cs:line 371
2022-11-14T12:31:58.0188863Z       --- End of stack trace from previous location ---
2022-11-14T12:31:58.0189433Z          at System.Threading.Tasks.Parallel.<>c__50`1.<<ForEachAsync>b__50_0>d.MoveNext()
2022-11-14T12:31:58.0190158Z       --- End of stack trace from previous location ---
2022-11-14T12:31:58.0190982Z          at common.IEnumerableExtensions.ForEachParallel[T](IEnumerable`1 enumerable, Func`2 action, CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/common/Functional.cs:line 45
2022-11-14T12:31:58.0192306Z          at publisher.Api.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Api.cs:line 365
2022-11-14T12:31:58.0193954Z          at publisher.Service.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, ListRestResources listRestResources, PutRestResource putRestResource, DeleteRestResource deleteRestResource, ILogger logger, CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Service.cs:line 51
2022-11-14T12:31:58.0195329Z          at publisher.Publisher.RunWithoutCommitId(CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Publisher.cs:line 84
2022-11-14T12:31:58.0196260Z          at publisher.Publisher.Run(CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Publisher.cs:line 69
2022-11-14T12:31:58.0197437Z          at publisher.Publisher.ExecuteAsync(CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Publisher.cs:line 47
2022-11-14T12:31:58.0202917Z info: Microsoft.Hosting.Lifetime[0]
2022-11-14T12:31:58.0203450Z       Application is shutting down...
2022-11-14T12:31:58.0221292Z fail: Microsoft.Extensions.Hosting.Internal.Host[9]
2022-11-14T12:31:58.0221867Z       BackgroundService failed
2022-11-14T12:31:58.0223869Z       System.InvalidOperationException: HTTP request to URI https://management.azure.com/subscriptions/***/resourceGroups/rg/providers/Microsoft.ApiManagement/service/apim/apis/api?api-version=2021-12-01-preview failed with status code 409. Content is '{"error":{"code":"PessimisticConcurrencyConflict","message":"Operation on the API is in progress","details":null}}'.
2022-11-14T12:31:58.0225521Z          at common.HttpPipelineExtensions.Validate(Response response, Uri requestUri) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/common/Http.cs:line 85
2022-11-14T12:31:58.0226535Z          at common.HttpPipelineExtensions.PutResource(HttpPipeline pipeline, Uri uri, JsonObject resource, CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/common/Http.cs:line 71
2022-11-14T12:31:58.0227580Z          at publisher.Program.<>c__DisplayClass17_0.<<GetPutRestResource>b__0>d.MoveNext() in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Program.cs:line 229
2022-11-14T12:31:58.0228506Z       --- End of stack trace from previous location ---
2022-11-14T12:31:58.0229689Z          at publisher.Api.PutApi(ApiName apiName, ApiInformationFile apiInformationFile, ApiSpecificationFile specificationFile, JsonObject configurationApiJson, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Api.cs:line 273
2022-11-14T12:31:58.0230946Z          at publisher.Api.<>c__DisplayClass23_0.<<ProcessArtifactsToPut>b__4>d.MoveNext() in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Api.cs:line 371
2022-11-14T12:31:58.0231842Z       --- End of stack trace from previous location ---
2022-11-14T12:31:58.0232425Z          at System.Threading.Tasks.Parallel.<>c__50`1.<<ForEachAsync>b__50_0>d.MoveNext()
2022-11-14T12:31:58.0233151Z       --- End of stack trace from previous location ---
2022-11-14T12:31:58.0233962Z          at common.IEnumerableExtensions.ForEachParallel[T](IEnumerable`1 enumerable, Func`2 action, CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/common/Functional.cs:line 45
2022-11-14T12:31:58.0235282Z          at publisher.Api.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Api.cs:line 365
2022-11-14T12:31:58.0238305Z          at publisher.Service.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, ListRestResources listRestResources, PutRestResource putRestResource, DeleteRestResource deleteRestResource, ILogger logger, CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Service.cs:line 51
2022-11-14T12:31:58.0239563Z          at publisher.Publisher.RunWithoutCommitId(CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Publisher.cs:line 84
2022-11-14T12:31:58.0240378Z          at publisher.Publisher.Run(CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Publisher.cs:line 69
2022-11-14T12:31:58.0241170Z          at publisher.Publisher.ExecuteAsync(CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Publisher.cs:line 47
2022-11-14T12:31:58.0241899Z          at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)
2022-11-14T12:31:58.0257828Z crit: Microsoft.Extensions.Hosting.Internal.Host[10]
2022-11-14T12:31:58.0258677Z       The HostOptions.BackgroundServiceExceptionBehavior is configured to StopHost. A BackgroundService has thrown an unhandled exception, and the IHost instance is stopping. To avoid this behavior, configure this to Ignore; however the BackgroundService will not be restarted.
2022-11-14T12:31:58.0261300Z       System.InvalidOperationException: HTTP request to URI https://management.azure.com/subscriptions/***/resourceGroups/rg/providers/Microsoft.ApiManagement/service/apim/apisapi?api-version=2021-12-01-preview failed with status code 409. Content is '{"error":{"code":"PessimisticConcurrencyConflict","message":"Operation on the API is in progress","details":null}}'.
2022-11-14T12:31:58.0262571Z          at common.HttpPipelineExtensions.Validate(Response response, Uri requestUri) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/common/Http.cs:line 85
2022-11-14T12:31:58.0263459Z          at common.HttpPipelineExtensions.PutResource(HttpPipeline pipeline, Uri uri, JsonObject resource, CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/common/Http.cs:line 71
2022-11-14T12:31:58.0264383Z          at publisher.Program.<>c__DisplayClass17_0.<<GetPutRestResource>b__0>d.MoveNext() in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Program.cs:line 229
2022-11-14T12:31:58.0265293Z       --- End of stack trace from previous location ---
2022-11-14T12:31:58.0266215Z          at publisher.Api.PutApi(ApiName apiName, ApiInformationFile apiInformationFile, ApiSpecificationFile specificationFile, JsonObject configurationApiJson, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Api.cs:line 273
2022-11-14T12:31:58.0267341Z          at publisher.Api.<>c__DisplayClass23_0.<<ProcessArtifactsToPut>b__4>d.MoveNext() in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Api.cs:line 371
2022-11-14T12:31:58.0268088Z       --- End of stack trace from previous location ---
2022-11-14T12:31:58.0268528Z          at System.Threading.Tasks.Parallel.<>c__50`1.<<ForEachAsync>b__50_0>d.MoveNext()
2022-11-14T12:31:58.0269128Z       --- End of stack trace from previous location ---
2022-11-14T12:31:58.0269819Z          at common.IEnumerableExtensions.ForEachParallel[T](IEnumerable`1 enumerable, Func`2 action, CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/common/Functional.cs:line 45
2022-11-14T12:31:58.0271002Z          at publisher.Api.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, PutRestResource putRestResource, ILogger logger, CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Api.cs:line 365
2022-11-14T12:31:58.0272518Z          at publisher.Service.ProcessArtifactsToPut(IReadOnlyCollection`1 files, JsonObject configurationJson, ServiceDirectory serviceDirectory, ServiceUri serviceUri, ListRestResources listRestResources, PutRestResource putRestResource, DeleteRestResource deleteRestResource, ILogger logger, CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Service.cs:line 51
2022-11-14T12:31:58.0273767Z          at publisher.Publisher.RunWithoutCommitId(CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Publisher.cs:line 84
2022-11-14T12:31:58.0274572Z          at publisher.Publisher.Run(CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Publisher.cs:line 69
2022-11-14T12:31:58.0275361Z          at publisher.Publisher.ExecuteAsync(CancellationToken cancellationToken) in /home/vsts/work/1/s/Automation/Provisioning/APIM/tools/code/publisher/Publisher.cs:line 47
2022-11-14T12:31:58.0276074Z          at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)
2022-11-14T12:31:59.3119145Z Exit code is 255

The APIM also becomes unresponsive after the failed pipeline (Management API becomes unreachable)

After deleting all APIs and attempting publish the configuration again, the pipeline completes with no issue.

Expected behavior

No 409 error are triggered when the pipeline is running against an APIM that was configured by the pipeline in the past

Actual behavior

It is not possible to publish changes to some APIs due to error HTTP error 409

Reproduction Steps

None

~~A rough summary of the APIM instance can be found here https://github.com/Azure/apiops/issues/135#issuecomment-1310804049~~