backstage: 🐛 Bug Report: Github events support should cause the changed catalog entity to be immediately processed

Thank you @wanisfahmyDE for sharing your experience. All my tests and cases I’ve investigated ended similarly so far.

@freben 's comment might lead to a direction that could be the source of issues for some.

If anyone has more details, we are happy to look further into the matter. However, we need more details about your case. For now, I would consider the events to work as expected.

@pjungermann we just implemented it with GitHub and can confirm that all additions, modifications and deletions are reflected instantly 🚀

In case it helps anyone, our complete catalog.ts config looks like this:

import {CatalogBuilder} from '@backstage/plugin-catalog-backend';
import {ScaffolderEntitiesProcessor} from '@backstage/plugin-scaffolder-backend';
import {Router} from 'express';
import {PluginEnvironment} from '../types';
import {
    GithubEntityProvider,
    GithubOrgEntityProvider,
} from '@backstage/plugin-catalog-backend-module-github';

/*
 The sequence works as follows:
    1. The eventBroker kicks in with every new/modified changes on catalog-info.yaml.
    2. If the event is dropped for any reason, a fallback with builder.setProcessingIntervalSeconds(3600) kicks in for existing entities only.
    3. Twice per day our complete GitHub organization is scanned and onboards/updates the Components in Backstage (scheduler).
 */
export default async function createPlugin(
    env: PluginEnvironment,
): Promise<Router> {
    const builder = await CatalogBuilder.create(env);
    // The org URL below needs to match a configured integrations.github entry
    // specified in your app-config.
    builder.addEntityProvider(
        GithubOrgEntityProvider.fromConfig(env.config, {
            id: '<some-id>',
            orgUrl: 'https://github.com/<org-name>',
            logger: env.logger,
            schedule: env.scheduler.createScheduledTaskRunner({
                frequency: {minutes: 60},
                timeout: {minutes: 15},
            }),
        }),
    );
    const githubProvider = GithubEntityProvider.fromConfig(env.config, {
        logger: env.logger,
        schedule: env.scheduler.createScheduledTaskRunner({
            frequency: {minutes: 720}, /* twice per day checking all repos and importing/refreshing in the catalog */
            timeout: {minutes: 30},
        })
    });
    env.eventBroker.subscribe(githubProvider); /* any new add/delete/modify actions in catalog-info.yaml will be handled by the event broker */
    builder.addEntityProvider(githubProvider);
    builder.setProcessingIntervalSeconds(3600) /* only for existing entities, they would be refreshed if the eventBroker didn't catch the event - currently set every 1 hour */
    builder.addProcessor(new ScaffolderEntitiesProcessor());
    const {processingEngine, router} = await builder.build();
    await processingEngine.start();
    return router;
}

FYI: I did a bigger test using the BitbucketCloudEntityProvider which was the prototype for the event support and with regards to the update mechanism, others follow the same logic. However, there might be differences in provider-specific logic.

For my test, I used

• BitbucketCloudEntityProvider
• a single isolated project at bitbucket.org using catalogPath: catalog-test.yaml
• schedule.frequency of 5 h at the BitbucketCloudEntityProvider (full refresh)
• processing interval of 5 h configured at the CatalogBuilder
• SQS-based event ingestion with event data flow: webhook -> […] -> SQS -> AwsSqsConsumingEventPublisher -> BitbucketEventRouter -> BitbucketCloudEntityProvider.onEvent

Environment:

• entity provider
  • config under catalog.providers.bitbucketCloud
  • provider added to the CatalogBuilder (builder.addEntityProvider(provider);)
  • provider registered as a subscriber (env.eventBroker.subscribe(provider);)
• events
  • general event setup prepared
  • […]EventRouter added as publisher and subscriber (BitbucketCloudEventRouter, GithubEventRouter, etc.)
  • HttpPostIngressEventPublisher and/or AwsSqsConsumingEventPublisher to retrieve events from external
  • webhook subscription which is sent and received successfully (e.g., you see logs from the provider which is supposed to handle it)

Scenario:

• wait for initial discovery to be completed
• add catalog file + verify
• modify catalog file + verify
• delete catalog file

All changes were applied immediately.

Originally, I assumed that there might be some issue with the difference between modification and add/remove as modifications will issue a refresh for the already existing entry (similar to the refresh button at the UI) while add/remove use the delta mutation.

However, as described, all cases worked as expected and updates were applied immediately.

@benkeil re your open points:

the event support is a rather recent addition.

I think it is still good to have a full refresh to recover from any potentially missed event.

The processing interval is global and affects all Location entities. This and many other entity providers are based on these under the hood.

Basically, the providers discover catalog files and (de)register Location entities for these.

The processing interval refreshes what their targets (catalog files) contain.

There are a couple of discussions around that matter on different issues with different scopes.

Hey @vonjones and @benkeil can you share more info about the problem?

I tried a really simple reproduction and works as expected. I enabled the Github event (PR) configured webhook in the org and pushed a change in the description . I can see the result after the event is processed.

Did you can see the logs in your app? Something like that:

[1] 2023-05-17T00:18:17.676Z catalog info Processed Github push event: added 0 - removed 0 - modified 1 type=plugin target=github-provider:backtostage

Maybe we are missing something more specific about the problem.

Can you try to setup a example so we can see the issue happening and reproduce that?

CC: @angeliski

I would be interested to participate.

And just double checking - these are the instructions I referred to https://github.com/backstage/backstage/blob/master/plugins/events-backend-module-github/README.md