tcWebHooks: Configured webhooks break when kotlin code syncs

Expected Behavior

Configured webhooks should be configured correctly and editable.

Current Behavior

Configured webhooks produce an error when trying to edit

An unexpected error occured. Please see your browser's javascript console.
Responding with error, status code: 404 (Not Found).
Details: jetbrains.buildServer.server.rest.errors.NotFoundException: Could not find a webhook with that id
Could not find the entity requested. Check the reference is correct and the user has permissions to access the entity.

Configured webhook has lost all it’s settings

Screenshot 2024-07-18 at 11 39 26 AM

Under Versioned Settings, Loading project settings from VCS can re-create the webhooks with the correct settings. However leaves these “ghost” webhooks to persist until a service restart.

Extent of the breaking varies If a webhook is defined in code and that definition is not altered, it stays in place and can be edited, but it loses some configuration and needs to be fixed If a webhook is not defined in code (eg for subprojects, because we have no way of defining that in code), then the webhook just becomes uneditable

Steps to Reproduce (for bugs)

  1. Make a kotlin change, unrelated to webhooks
  2. Wait for Versioned Settings to synchronise and apply changes
  3. Webhooks break

Your Environment

  • tcWebHooks Version: 2.0.1
  • TeamCity Version: 2024.03.2 (build 156319)
  • TeamCity server Operating System: Amazon Linux 2 LTS
  • Are you using a WebHook Template?:

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 30 (18 by maintainers)

Most upvoted comments

No need for an apology. Thanks for the update, appreciate the progress.

I’m so sorry. I’m still working on this. I have had a conference call with jetbrains and we ruled out a few things. They have given me some guidance on what to try next.

I have managed to get it to compile offline (by default it loads libraries from your local TeamCity instance). This enables running the build in CI.

Basically there is a way to generate a very simple DSL using an XML configuration file.

However that does not allow configuration to contain a list of items (eg, headers, etc). I now have a way to build a kotlin DSL from scratch so just need to work through generating the Project features using the kotlin classes.

I am pretty sure I can do this with TDD. Previously I was hacking stuff together, building the plugin, uploading into TeamCity, restarting TeamCity, and then making GIT commits to try to trigger the configuration reload.

Tldr; some progress. Nothing to test yet.

I am loath to release anything yet because I feel like the project features format is not yet completely final, and trying to provide migrations would be impossible.

@netwolfuk thanks for the update. Happy to test out any test builds of the plugin.

A quick update…

I have not been able to reproduce this issue where the build does not resolve.

I have done the work to convert from using plugin-settings.xml to using ProjectFeatures. This will enable a path to using the kotlin DSL for configuration of webhooks.

I’ve also written a tool that converts from plugin-settings.xml to ProjectFeatures.

This is undergoing testing at the moment. The obvious shortcoming is that we need a way to convert to the new format for those that don’t allow writing to VCS Settings. I am still working through how to do this.

Hi @netwolfuk , we only have the single server TeamCity setup.

We mainly edit the project via Kotlin files and sync from VCS to TeamCity. That’s when we started seeing similar inconsistency with what was in the code vs UI (ghost webhooks and unchecked projects etc)

I’ve managed to make some really good progress on this. I have found the following:

Updates to webhooks that are driven by the plugin work well The cache is maintained correctly because the changes are applied to the settings, and then the cache is updated.

Updates driven by VCS changes without IDs cause ghost webhooks This is because the existing webhooks are not updated because they are not found due to the ID being regenerated.

Also, I was not comparing the result of the existing webhooks against the updated webhooks. The only predictable way to compare them is by ID, but newly loaded webhooks will be assigned an ID if they don’t have one defined. This will not match the existing one, so the old webhook is not updated. A new one is created.

I have added logic to compare the IDs after the cache is refreshed, and remove any webhooks from the current project if they don’t match one of the new IDs. A line like the following is printed in the log.

[2024-08-05 09:16:35,341] INFO - jetbrains.buildServer.SERVER - WebHookSettingsManagerImpl :: Found the following webhooks in the cache that no-longer map to webhooks in project 'VcsProjectTest'. WebHooks with ids '[id_377052579, id_283758663]' will be removed from the cache.

This does have the downside that IDs will churn. This causes two annoyances.

  1. The history page will link webhook events with webhooks that might no longer be relevant.
  2. If the edit webhooks page is loaded, and then the webhooks change, the edit will fail because the webhook cannot be found.

Ideally, webhooks should have unique IDs. This will work around this issue.

TeamCity’s buildTypeUnregistered event is poorly named In 2.0.1, I added a listener for the buildTypeUnregistered event. This allows me to cleanup webhooks that might be interested in a build that has been deleted.

However, I have found that this event is fired when the VCS configuration is reloaded. TeamCity appears to re-read the VCS settings, delete all builds on that project, and then recreate them with the same internal and external IDs. At which point the buildTypeUnregistered fires for each build.

When I see that event, I diligently clean up any webhooks that explicitly reference that build, and remove that build (btxxxx) id from their configuration.

This presents as webhooks having 0 builds registered. Sometimes webhooks still have builds showing. This is because a project edit (either via VCS or from the UI) fires up to 10 events in quick succession. I have a queue backed delay and de-bounce algorithm to only present one of each type of event.

I therefore only see one buildTypeUnregistered and one projectRestored, but they will fire at almost the same time. At which point a race condition exists as to whether the cache is refreshed while the build is marked as deleted or has been “undeleted”.

I have removed the buildTypeUnregistered event listener, and builds now reliably show for all webhooks. I will do some further testing to make sure that not listening for this event has no problematic side-effects.

I’ll do more testing on various configurations. I hope to have a very early beta release of 2.0.2 available for testing in 24 hours.

HI @jacobs-xero. Having an ID is required. It does not matter which format you choose. The most important thing is that they don’t clash across the whole of the teamcity instance.

If I understand correctly, your VCS is being written to by TeamCity. This means that when the webhook is loaded it is…

  1. Checked that an ID is present. If not, one is allocated.
  2. Checked that the ID does not clash. If so, a new ID is generated.
  3. In your case, it should then be written back to the VCS and you won’t have this issue again.

For ticket #248 the same is happening, but because they don’t let TeamCity update back into VCS, the issue occurs on every update.

So, for them the benefit is that using an id like FooBarPRDevOps_Webhook_1 allows them to manage uniqueness better, since the only place that can be managed is in their XML file before they commit it.

At this stage, these two tickets look very similar, but I think are actually different bugs. I am hoping to get some time on the weekend to do a deep dive on the issue you are seeing.

I suspect in both cases it is exacerbated by the nested projects that inherit from a VCS managed project, but I have yet to prove that.

Thank you for the quick feedback. I’ll add that to the array of facts I have been collecting. I am about to post an update to the other ticket (#248 ) as to what I have been able to reproduce.