argo-workflows: 3.5 ListWorkflows causes server to hang when there are lots of archived workflows

Pre-requisites

  • I have double-checked my configuration
  • I can confirm the issues exists when I tested with :latest
  • I’d like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

We had >200,000 rows in the workflow archive table, and when trying to view the new combined workflow/archived workflow list page in the UI, the server times out

scanning the code, it looks like the LoadWorkflows code loads all rows from the archive table, combines them with the k8s results and then applies sorting and limiting.

as a workaround, we’ve reduced the archive ttl from 14 days to 1 day, and the endpoint now responds before timing out, but is still pretty slow.

Version

v3.5.0

— edits below by agilgur5 to add updates since this is a (very) popular issue —

Updates

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 45
  • Comments: 123 (64 by maintainers)

Commits related to this issue

Most upvoted comments

@sunyeongchoi and I have discussed a couple of potential solutions. Unfortunately, there are edge cases that we cannot get around with those approaches (due to the complexity of pagination, deduplication, sorting, etc.).

I propose that we add a dropdown to allow users to select whether to display:

  1. Only live workflows;
  2. Only archived workflows;
  3. Both live and archived workflows (with notice/warning that this is only suitable when the number of workflows is not too large);

Additional requirements:

  1. This dropdown box is only available if there are both archived workflows. UI should be smart enough to figure this out.
  2. We should also consider the ability to disable the third option as admin to avoid bringing down cluster performance.
  3. The “archived” column should only be displayed when appropriate. For example, it’s evident that a workflow is archived if the user is only viewing archived workflows.

Motivations for this proposal:

  1. Some users only care about one of these types of workflows;
  2. Since there are performance issues that we cannot get around in order to view both types of workflows, we should only provide this option with caution;
  3. Keep using the original pagination implementation for live workflows or archived workflows where the logic is much more precise while keeping the front-end codebase simple;
  4. The first two options are almost identical to previous versions, but the UI should be less buggy since they now share most of the implementation; the third option is an addition to previous versions.

Any thoughts?

I’ve had a few recent internal design discussions about how to fix this. I wanted to share the proposal, as well as a commitment from my team to fixing this in the near term (the coming weeks).

We want to fix the performance issues while preserving the feature. The crux of the issue is that K8s querying capabilities are limited to what the UI needs. So the solution to preserving the feature is to somehow make it more queryable to support the use cases of the UI. Namely:

  1. ordering & filtering by date
  2. Filtering by labels
  3. partial name matching

Earlier, I proposed using informer as an in-memory cache on the API server to perform efficient lookup without querying K8s. But the concerns about this were that:

  1. this could balloon the memory requirements of the API server.
  2. it meant that querying/filtering against the informer would also need to be a custom, handcoded implementation, thereby adding to complexity

Instead of an informer cache, we are now proposing that Argo API server maintain an in-memory SQLite database of the live workflows, which is continuously populated by a watch stream to K8s API. As workflows are created, updated, or deleted, the entries in SQLite would automatically be updated as well.

To address the memory issue, the entries in the SQLite database do not necessarily need to be the complete workflow. The UI already uses field selection to get a partial workflow and so only certain fields would need to be stored. Apart from that, the live workflow entries would also be stored in a compressed and space-efficient format as YAML/JSON is highly compressible.

The benefits we see of this approach are:

  1. we have the same powerful SQL querying capabilities for both live and archived workflows (driven by SQL)
  2. the querying implementation itself would also be consistent (the same SQL query could be used for both live/archived)

In the end, the result will be that we maintain the current UX that we currently have in v3.5, but with the performance issues resolved. It also gives us room in the future not to be hampered by whatever limitations the K8s API, and add additional querying capabilities that SQL provides.

I think the underlying reason is to try to show normal workflows and archived workflows together, but I don’t agree to show both together. Even in api, the two workflows are separated, but it feels rather confusing to show them together on the UI.

I am reverting related change https://github.com/argoproj/argo-workflows/pull/12068 for now since otherwise the UI is not usable when there are many workflows. In the meantime, we can continue discussing proposal https://github.com/argoproj/argo-workflows/issues/12025#issuecomment-1774346636 here. We can probably release a patch version next week https://github.com/argoproj/argo-workflows/issues/11997#issuecomment-1775681491.

With the unified workflow list API for both live + archived workflows we have to do the following:

  1. we need all workflows in the cluster because kubernetes API server does not support sorting by creation timestamp, but argo-server does.
  2. we should only query X workflows from the archive, where X is the requested page size. The underlying database does support filtering and sorting, so this is efficient. The fact that we query everything from archive is nonsensical.

Number 1 is a challenge because performing LIST all workflows for every request will be taxing on K8s API server as users use the UI. Listing all workflows for the purposes of only showing 20 is extremely inefficient. To get around this, I propose a technique we use in Argo CD:

Since LIST all workflows is the requirement and also the problem, we can use an informer cache on Argo API server to avoid the additional LIST calls to k8s and have all workflows ready to be returned by API server from in-memory informer cache. When API server is returning a unified list of live + archive, it would call List() against the informer cache rather than List against K8s API, and then filter/merge/sort before returning results back to the caller.

Note that this does balloon the memory requirements of argo-server because of the informer cache. But I feel this is manageable with archive policy and gc. And the benefits of a unified live+archive API, as well as reducing load on K8s API server, outweigh the extra memory requirements of argo-server.

If we are worried about expensive processing / sorting of the results of List() calls we make against the informer cache, we could consider maintaining our own ordered workflow list (by creationTimestamp) that is automatically updated with UpdateFunc, AddFunc, DeleteFunc registered to the informer. But I consider this an optimization.

I mentioned above that I think checkboxes in the left sidebar would make more sense then a select box. I also mentioned that we could hide it as “Extra Filters” in a modal as well.

Checkboxes and label being:

WORKFLOW STORAGE

  • In-cluster
  • Archived

We could have a little info icon next to the “Archived” checkbox linking to the Workflow Archive docs as well. We could also explain within info icons that “In-cluster” means as a k8s resource stored in etcd and that “Archived” means stored as a row in an Archive DB

at least until the feature “merged view” scales well (bifurcating by the labels said here or similar) and is stable.

The plan is to make them scale properly, this issue is still very much listed as a bug. We also did discuss this in the most recent Contributor Meeting and a solid use-case did pop up there for filtering out Archived Workflows: CI wherein new commits cancel existing Workflows which are then immediately archived – quick sequential commits can cause a lot of clutter. Similar fast-updating use-cases fit that too. The converse I’m not as sure of on a use-case, showing only Archived Workflows and no In-cluster Workflows

And currently PR only has a page that shows Workflows and Archived Workflows combined.

Would it be a good idea to add a filter or page that only shows Workflows and Archived Workflows separately?

Just reverted back to v3.5.0-rc1 due to this. Perhaps we can add the default started time back?

3.ii +1 to “just add back the Archived Workflows list page”

@agilgur5 yes, I can restart it.

I’m checking the logic, but regarding the archived object’s label being Persisted, is it possible to solve the duplicate issue by filtering the workflows?

Of course! If you do, I would be very grateful.

I didn’t want to intrude initially. Wanted to make sure you had time to work with it. But if we are now considering what are effectively total reverts, then I think it would be wise to have more contributors take a look before going down that direction.

This way could be nice if you take the assertion all static workflows are successfully written in Archived. Which is an assertion that (I believe) happens in robust deployments.

This is unfortunately an overly optimistic assumption that doesn’t hold up for all cases. It is mostly correct as a simplifying assumption, but not entirely. For example, even in “robust” deployments, it is very common to keep completed Workflows in k8s/etcd for some period of time before GC’ing them. This is especially the case for failed or errored Workflows, where someone may want to manually investigate the Pods etc (similarly to a dead letter queue). There’s also the race conditions when Workflows are in the process of being Archived. Both can exist simultaneously during that time. Similarly there is the case of retrying an Archived Workflow – retries are a mutable operation (compared to resubmit, which creates a new Workflow), so the Workflow is no longer “static” in that case.

There is actually specific code to deal with this edge case of having a Workflow in both the Archive and in k8s/etcd, c.f. #11336, #11371, etc.

And then of course there is also the case of not having configured a Workflow Archive at all.

Options 1 and 2 are natural and <v3.5.0 arch ones, those should definitely exist to avoid feature regression (currently happened at v3.5.0 where you lost this ability to query either etcd or your backend db).

I’m not so sure that this would be considered a “feature regression”. The API and UI were intentionally unified in #11121. As per the original issue #10781, an Argo user (not necessarily an operator) generally does not know about or care about the difference between the two.

Having optional checkboxes (not a dropdown) could make sense as those are then just “filters”. Those could also be hidden behind a modal/menu on the UI as most users won’t change the default. That doesn’t solve this issue though for when both are checked.

I am also wondering if it may be simpler to keep the APIs decoupled and have the UI do merging. This was added as a backward-compatible option to the API, so the /archived-workflows/ API is still available, and argo archive list is also still available in the CLI. EDIT: This option was previously rejected in https://github.com/argoproj/argo-workflows/issues/10781#issuecomment-1489263668

@sjhewitt thank you for filing this. I’m having the same issue. 30+ second page loads. I’m gathering data now and will post here once obtained.

@sunyeongchoi thanks for taking a look.

Can confirm this isn’t related to the “past month” default on the UI being removed from the search box like @agilgur5 states. That is only client side.

For reference, v3.5.7 has been released with a backport of #13027

Since this is for Archived Workflows specifically, I think this might be due to the select * change I mentioned in https://github.com/argoproj/argo-workflows/pull/11121#discussion_r1469101802 in my previous comment about remaining performance regressions?

@jiachengxu could you take a look at that? If fixing it resolves the performance regressions then we should get that out in a separate PR that we can backport independently (it would be a pure fix).

@agilgur5 Thank you for letting me know about this regression, I think this is related.

As you suggested, I am going to open a PR to fix this regression. Also, I will include this change on the top of the sqlite-based in memory PR so that folks can check if the memory usage decreases with the fix.

Furthermore, inspired by this regression, I think we even don’t need to store the entire live workflow in the sqlite-based informer(based on the code change of the regression, only name, namespace, uid, phase, startedat, finishedat are needed), in that case, we will be able to reduce even more memory usage so the in-memory sqlite db doesn’t balloon much memory.

I will let the community know when the above improvements are implemented and push images so that community members can test them out.

it is weird, it only happens time to time, not on every single pagination iteration

also, when filtering by a label, sometimes, it throws No workflows existing, for workflows that exist on archived workflows (and in a future query moment, it gives a list again - not sure if exhaustive)

@Guillermogsjc Thank you so much for testing it. I found the issue, and it is related to how pagination is implemented currently.

The current pagination (on main branch) is implemented as follows, given the limit X and offset Y as the cursor pagination params:

  1. Get X live workflows starting from Y from k8s API server.
  2. Get X archived workflows starting from Y from the archived workflow database descendingly ordered by startedAt time.
  3. Merge them and return the first X.

This pagination mechanism never worked due to Kubernetes’ limited capability of pagination.

Once we switched to use the in-memory sqlite-based store for live workflows(in PR https://github.com/argoproj/argo-workflows/pull/12736), I tried to use the same mechanism:

  1. Get X live workflows starting from Y from sqlite store descendingly ordered by startedAt time.
  2. Get X archived workflows starting from Y from the archived workflow database ordered by startedAt time.
  3. Merge them and return the first X.

This definitely improved the situation, but still doesn’t work and it could lead to having duplications in the results(example here: https://github.com/argoproj/argo-workflows/issues/12025#issuecomment-1793482528). Also, the continue for the next page could not be computed precisely, that’s why during the testing, sometimes it throws No workflows existing or related issues.

So I came up with the following approach, given the limit X and offset Y as the cursor pagination params::

  1. If X + Y < num(live workflows), we return workflows from live workflow in-memory database directly, and no need to get archved workflows and merge them, since even if we merge them, the live workflow will take high priority and will be returned.
  2. If Y < num(live workflows) && X + Y > num(live workflows), this means the return will contain both live and archived workflows, in this case:
    1. Get the rest of the live workflow starting from Y in the sqlite inmemory db descendingly ordered by startedAt time, mark count of the returned live workflow Z, and the last element in the return which has the oldest startedAt time T,
    2. Get the rest of the (X - Z) in archived workflows, older than T, and descendingly ordered by startedAt time.
    3. Append them and return as the final result.
  3. If Y > num(live workflows), this means that all workflows will be fetched from archived workflow:
    1. Get the live workflow which has the oldest startedAt time T,
    2. Get the X in archived workflow with starting from Y-num(live workflows), older than T, and descendingly ordered by startedAt time.
    3. Return as the final result.

This approach provides the following benefits:

  • No duplications in the return and fixes the issue with the current merging approach.
  • Reduce the unnecessary call to archived workflow db if the results can be returned from live workflow in memory db directly.
  • continue and showRemainingItemCount can be computed precisely and correctly so that UI and client can work better.

I have pushed this implementation to quay.io/jiachengxu/argocli:fix-v3, note the tag is fix-v3, feel free to test and see if there are any issues.

In general, it works (UI) as a charm, filtering labels, state filtering, etc. Pretty fast over tons of ectd and archived workflows 😃

When going through pagination, sometimes (did not not observe any particular pattern) it is thrown:

Internal Server Error: no such table: argo_workflows

image

but it self heals after few seconds and the internal error dissapears without doing nothing

It is a deployment with 3 replicas, maybe there is something to do with one replica not having the in-memory SQLite created in the another replica, and load balancer derivating pagination requests to replicas “not initialized”. Maybe deterministic initializations of those sqlite could be settled somehow to allow balancing between them in a replicaset.

This could be one of the shortcomings of having this in-memory approach, do not know if it could be adjusted.

Another path to allow the distributed way could be to put those auxiliary tables on the dedicated database (in case of existing in the deploy), to avoid the in-memory on the case of multiple replicas + dedicated database at least.

1.i That’s right. this is also why I implemented this feature https://github.com/argoproj/argo-workflows/pull/12237. In my opinion, if this problem cannot be resolved immediately, I think it would be okay to temporarily provide a separate page first.

2.i If it still have problems even after revert to before this function, could it be the problem below?

When I tested it, there was a problem that it freeze when I quickly clicked on the previous and next pages. when I looked at it in the F12 developer tool, I saw that it freeze when the pending status of eventSource type data accumulated a lot.

image

3.ii I agree with this opinion. Of course, we have already implemented to show both data(k8s & db) on one page through k8s labels selector https://github.com/argoproj/argo-workflows/pull/12237/. However, I also had many concerns while implementing the function. Because combining the two data with labels is not completely cover everything, as you have said many times. Also, Considering the filter function requested by many users, filtering is expected to become more complicated when db resources and k8s resources are combined into one. I think that managing different resources for k8s and db on different pages would be more stable and show more accurate data.

CI wherein new commits cancel existing Workflows which are then immediately archived – quick sequential commits can cause a lot of clutter. Similar fast-updating use-cases fit that too. The converse I’m not as sure of on a use-case, showing only Archived Workflows and no In-cluster Workflows

All of these scenarios are important to us. We are using argo-workflows for CI. The last view that @agilgur5 mentions is important if I need to reference test failures from a previous workflow run. I may filter by label and then by archived.

A checkbox on the left panel that is checked by default is the ideal UI change IMO. I have started a PR locally for it but don’t want to waste the time if the work is already done as part of ~#12397~ (update: wrong one) #12237 (update: correct one)/cc @sunyeongchoi

I think this is the best way.

I mentioned above that I feel a drop-down seems like clunkier UX than a separate page (the previous behavior) to me.

If this is the primary option we are considering, I would ask if I could take some time to take a look through in-depth myself (I was not previously involved with this change). I’ve built lists with complex sorting, filtering, and pagination with efficient implementations in the past. If needed, usually the worst-case scenario is that you have to send to the API a list of all IDs to exclude (e.g. duplicates).

However, instead of paginate Archived and Workflows separately in step 3, fetch all Archived and Workflows just once at first and combine them and remove duplicates.

Fetching all is pretty much never an option. There is always an amount at which it can overload a server and is also a big load spike.

Even with an Informer cache (which only handles in-cluster Workflows), it can cause spikiness when the Informer rebuilds the cache periodically as well. Spikiness is generally a behavior to avoid from an operations perspective (my day-to-day work these days is primarily SRE-focused), if possible. Informers are a bit more transparent than generic caches, but even then, it adds a layer of complexity (i.e. more potential for bugs, which we have quite a bit of, some due to Informer misconfigurations) to the Server that we should avoid if possible.

Note that this does balloon the memory requirements of argo-server because of the informer cache.

I think it’s worth noting that this balloons the memory requirements of each Argo Server Pod. If you have a standard, 3 replica HA set-up, your memory usage is going to go way up.

@sunyeongchoi and I have discussed a couple of potential solutions. Unfortunately, there are edge cases that we cannot get around with those approaches (due to the complexity of pagination, deduplication, sorting, etc.).

What are some of those edge cases? We can still over/under fetch so long as it does not overload the server. For instance, in the worst-case, if a user has 20 Workflows per page set, we can retrieve 20 from k8s and 20 from the Archive DB, which is not horrendous (but for sure could be optimized).

I’m similarly curious… I wonder if it would be possible to use a cursor that encodes 2 offsets - one for the k8s api and one for the db, then fetches limit rows from both sources with the given offset, merges the results together and applies the limit to that combined list.

something like:

orderBy = ...
filters = ...
limit = 20
cursor = 0, 0
k8sResults = fetchK8s(cursor[0], limit, filters, orderBy)
dbResults = fetchDB(cursor[1], limit, filters, orderBy)
results = mergeResults(k8sResults, dbResults).slice(0, limit)

newK8sOffset = getLastK8sResult(results)
newDBOffset = getLastDbResult(results)
newCursor = (newK8sOffset, newDBOffset)

@agilgur5 , I switched to 3.5.8 just now. I still see lag when switching namespaces in argo workflows UI.

We are running 3.5.7 with a few 1000 archived workflows. The UI is still loading awfully slow, compared to the 3.4 branch. The page loads in 20-30 seconds. How can I verify that this fix is working as intended? What other things can I tinker with to get faster loading times.

Thank you both for helping me gain a clearer understanding of the current review process.

I was the main analyzer here in the issue, but I haven’t had time to give #12736 an in-depth review unfortunately. Getting the approach right here was always my top priority though and I think we ironed the heck out of it. I didn’t see any glaring issues either after the CGO removal, otherwise I would’ve left a blocking comment. If other Approver+ folks gave it a thorough look-over, I think it’s had sufficient due diligence. If there are follow-up issues or follow-up code review comments, we can follow up on them (see also https://github.com/argoproj/argo-workflows/pull/12912#discussion_r1572901283 as an example of that). I wouldn’t anticipate any substantial changes though; that’s why I was trying to iron out the approach after all.

We otherwise don’t have firm rules on the process outside of the branch protection requirements (we don’t have many Approvers either, see also the Sustainability Effort).

So I think Terry made the right call there (and also gave some more time than strictly necessary for others to take a look too, which I’m pretty sure was intentional). Would be great if I could have taken a deeper look too, but unfortunately we have bottlenecks and this is the top P1, so we all have to use our best available judgment often.

Please correct me if I missed anything:

  1. All comments from @agilgur5 in https://github.com/argoproj/argo-workflows/pull/12736 were addressed and marked as resolved by @agilgur5.
  2. There were no unresolved comments. No requested changes from reviewers.
  3. The PR was approved by 3 approvers. We actually only need one approval to merge a PR.

If you see any additional issues, feel free to follow up by commenting here or submitting a new issue.

change I made https://github.com/argoproj/argo-workflows/issues/12025#issuecomment-2044952613(sqlite db only stores the live workflows that don’t have workflows.argoproj.io/workflow-archiving-status set to Archived), we get rid of the workflows which are archived but still in the k8s cluster on the sqlite-based informer level, we don’t have the duplications anymore.

this partitioning looks optimal

on v6 there are all archived workflows and looks fine

it uses plenty of memory when pagination is done

image

but seems bounded somehow, so it would be tweakeable by deployment (the important point is to have bounded memory usage on operations depending on workflow column mean size of tables)

just to remember that v6 behaviour shows first all non archived, then archived, so this can feel weird if you have very old etcd workflows (mostly because TTL worker errored deleting them I guess), but this is not a main concern probably.

@agilgur5 I can confirm that it is not seen this behaviour at least on our deployment, in v3.5.5

v3.5.5 shows all the workflows from both sources. Sqlite versions look like missing a big amount of workflows from archived (etcd workflows appear normally)

@jiachengxu it is an issue that has been around all the time in all fixes 😦

@jiachengxu v4 feedback:

on several pagination iterations it is found: Internal Server Error: sql: Scan error on column index 7, name "annotations": converting NULL to string is unsupported -> this looks as a small bug related to some incomplete filtering.

I find this really weird gap on pagination from few hours to tons of days (there are obviously workflows between those two timestamps):

image

from memory point of view looks stable (but it was already quite stable on v3), deployed so far v3 with really few incidents of memory and restart of the pods.

Hi @agilgur5, I have created a PR to fix the regression: https://github.com/argoproj/argo-workflows/pull/12912 Also, I have included the change into the sqlite-based in memory PR so that folks can check if the memory usage decreases with the fix. This can be tested with quay.io/jiachengxu/argocli:fix-v4. cc: @Guillermogsjc @jmeridth

Furthermore, inspired by this regression, I think we even don’t need to store the entire live workflow in the sqlite-based informer(based on the code change of the regression, only name, namespace, uid, phase, startedat, finishedat are needed), in that case, we will be able to reduce even more memory usage so the in-memory sqlite db doesn’t balloon much memory.

I tried to implement this but it looks like this doesn’t work for live workflows since there are more fields (e.g., status.nodes) needed by the UI. I think this can be considered as a future improvement for the sqlite-based in-memory solution.

@Guillermogsjc Thank you for testing and giving the feedback!

As my current knowledge (to be corrected if wrong), a workflow that reaches a status in the static set, gets archived into the optional database when finishes.

This works if the archived database is enabled, but if the archived database is not enabled, I guess we still need to get them from the in-memory database?

Ignore duplicated (when there are some), or deduplicate just bringing one field (UID) on SELECT from both tables, before doing a second batch query with more information.

The pagination of fix-v3 is implemented(for more details, please check above) in a way that doesn’t require query from both tables and perform merge, depends on the cursor pagination params, either query everything from only in memory, or in memory with archived workflow appended, or archived workflow only.

deployed quay.io/jiachengxu/argocli:fix-v2, fast and great when filtering by labels etc, but…

when trying to paginate by 50 workflows (NEXT PAGE button) with Succeeded included (those 98 k in archived), it explodes

image

it is weird, it only happens time to time, not on every single pagination iteration

also, when filtering by a label, sometimes, it throws No workflows existing, for workflows that exist on archived workflows (and in a future query moment, it gives a list again - not sure if exhaustive)

Hi @Guillermogsjc, I did reproduce the issue on my local setup, and I think that was because sqlite multi-thread mode was disabled and that error could occur due to concurrency issues or race conditions when multiple threads or processes are accessing the same in-memory database simultaneously.

As you might know, I refactored the PR recently to make it cgo-free, and the lib(https://github.com/zombiezen/go-sqlite) which I am using will always enable the multi-thread mode by default.

I have created a new image and tested it, and I couldn’t see the error anymore. Could you pull the image and test it again in your setup? You can still pull the image with latest, or you can use fix-v2 as the image tag.

there is something that is a dead end and makes single pod deployment to reach an unhealthy unusable status.

It was working Ok, then at a moment, it locked on Internal Server Error: no such table: argo_workflows forever.

this log was collected from a replicaset with only 1 element:

logs -> logs-from-argo-server-in-argo-server-5967db98b8-wbvtr.log

more logs -> logs-from-argo-server-in-argo-server-85c66ffd6d-8dtsb.log

In general, it works (UI) as a charm, filtering labels, state filtering, etc. Pretty fast over tons of ectd and archived workflows 😃

Thank you for the quick test @Guillermogsjc ! Would you mind sharing your scale, e.g., how many live workflows and archived workflows you have?

Maybe deterministic initializations of those sqlite could be settled somehow to allow balancing between them in a replicaset.

Yeah, this is also my thought, once the sqlite is initialized for all replicas, this should be ok. However, I am more concerned that this approach balloons the memory usage too much, which could cause OOM for the argo server pods, in this case, the pods might be OOM killed all the time and this error will happen from time to time. Would you mind sharing how much memory that per argo server pod is consuming and how much it was before? Have you observed any argo server pods got OOM killed?

@agilgur5 I like the combined view but we may need to split again for a bit until we address some of the items you mention. Thank you for deep diving on this stuff and also taking over the handoff.

@agilgur5 yes, I can restart it.

So I took over the client-side workaround this week as @jmeridth has been busy on the day job etc, although I managed to get pretty sick with the flu the past 2 weeks too 😅 (EDIT: and then started passing a kidney stone… 😖)

Was working on this today and I did notice a handful of things that may warrant further discussion:

  1. Apparently there is currently no API field to request live workflows only in 3.5 – archived workflows are retrieved unconditionally. I thought there was one based on https://github.com/argoproj/argo-workflows/issues/10781#issuecomment-1544253935, but I guess it wasn’t implemented?
    1. Due to this, I’d have to implement some server-side logic to handle this, which is going to at least partly conflict with #12237.
    2. That’s potentially fine, just involves more re-work, but that also means it wasn’t exactly agreed upon before?
    3. That also means that a CLI only downgrade as mentioned directly above should have no effect if the Server is v3.5.1+ (or v3.5.0 either, which had most of the performance issue, see below)
  2. Most of the performance regression part of this issue should have been solved in the revert of #11761 in #12068 (which did re-instate a different bug: #11715), which was released in v3.5.1
    1. Specifically, it reverted to the behavior of retrieving the same amount from both archived and live. So this should be relatively performant if pagination is enabled in v3.5.1+. But it appears like people are still having performance issues in v3.5.1+?
    2. If that is the case, I might have found a different performance regression that would only happen with large Archived Workflows: https://github.com/argoproj/argo-workflows/pull/11121#discussion_r1469101802 seems to select all fields
  3. I was also thinking about adding back the name filters per #12161, but there’s a reason that filter didn’t exist for non-Archived Workflows: it’s not k8s native and therefore won’t necessarily be performant.
    1. I noticed this in a related feature PR that added a “name prefix” filter to a different list, the WorkflowTemplate list. That PR ended up also needing to request all WorkflowTemplates from the k8s API to implement the filter: https://github.com/argoproj/argo-workflows/pull/11684#discussion_r1469090044
    2. Given that, re-implementing it would be complex for a combined view, so now I’m reconsidering whether we should just add back the Archived Workflows list page…
      1. Basically only because there are certain filters that are only performant / only have native implementations in a (SQL) DB.
      2. But we do already have some non-native filters like the date filter mentioned above as well as requests for more, so maybe we should bite the bullet on implementing these non-natively instead of just having Archive-specific filters. It may be good to add warnings and potentially flags for non-native search fields though, as I mentioned in https://github.com/argoproj/argo-workflows/pull/11684#discussion_r1469111527
        1. Notably, the UI date filter has a performance regression for Archived Workflows as well, as they’re no longer server-side / done natively in SQL as the Archived API (which supports native date filtering) is not used in the combined view
      3. I do agree with the UX conceptually of the combined view, since archiving is more of an implementation detail for a user. But, from a k8s native perspective, archiving is the opposite of k8s native – so maybe it makes sense to have them split, especially if all users are k8s aware (which is somewhat likely with the way Argo inherits k8s fields etc).
      4. If we do re-split the Archived Workflows list page, well, then there’s no point in a client-side only workaround either… it’s not a ton of work though, but as it requires a new API field, removing it after adding it would be a breaking change (which is maybe fine in this case given the priority / popularity and that Argo does not strictly follow SemVer as-is).

Answers, thoughts, feedback, concerns, discussion, etc welcome. Feel free to quote a specific part of that big list above that you’re responding to. Can also use the list numbers, e.g. 3.ii.a

Would it be a good idea to add a filter or page that only shows Workflows and Archived Workflows separately?

I’m thinking about UI design to show Workflows and Archived Workflows separately. I’m thinking about using a select box like the picture below. Do you have any other good ideas?

image

I propose that we add a dropdown to allow users to select whether to display:

Only live workflows; Only archived workflows; Both live and archived workflows (with notice/warning that this is only suitable when the number of workflows is not too large);

for the 3rd case query, where the problems arise,

Both live and archived workflows (with notice/warning that this is only suitable when the number of workflows is not too large);

You could break the query this way

  • ongoing workflows -> Workflows / k8s API / etcd
  • static workflows -> Archived / backend database

And paginate exhausting first ongoing workflows generator/list, then static workflows generator ( at least for sure against postgresql database you could get a cursor/generator sorted by creation timestamp with skip N elements with K elements window).

This way could be nice if you take the assertion all static workflows are successfully written in Archived. Which is an assertion that (I believe) happens in robust deployments.

we can set the cursor to the last value in the list, i.e. 2023-01-07 then when querying for the next page, we can query the database with startedat > 2023-01-07 limit n and the Workflows can be sorted by startedat, filtered to values > 2023-01-07 and then n items taken from the top of the list.

Thanks for your idea and good example. However, I think that this way also creates a situation where duplicate values ​​cannot be removed as follows, unless sorting is performed after the entire search.

First Page as (Archived):

((a, '2023-01-04'), (b, '2023-01-05'), (c, '2023-01-06'), (d, '2023-01-07'), (e, '2023-01-08'))

Second Page as (Archived):

((f, '2023-01-01'), (g, '2023-01-02'), (h, '2023-01-03'))

Second Page as (Workflows): values > 2023-01-03:

((a, '2023-01-04'), (b, '2023-01-05'))

Suggestion: If 20 pieces of data are needed on one page, fetch 20 Archived Workflows, fetch 20 Workflows, and then merge them. Use the cursorPaginationByResourceVersion function when searching Workflows data for the next page.

@sunyeongchoi That might be a good optimization for the third option I listed in https://github.com/argoproj/argo-workflows/issues/12025#issuecomment-1774346636. It could help a lot when there are a lot more archived workflows vs live workflows. Although we still need to fetch the entire list of live workflows on all pages.

Also probably on this pagination, it would be useful to change the defaults on time ranges to show. Currently, it is one month, but probably it would be better to have a default on 1 or 2 days, to free that argo-server list workflows. This, together with the flags “show archived” that you are commenting, would help a lot.

@Guillermogsjc Thanks for the suggestion. However, we still need to make the same list query in the backend and then filter in the front-end.

it is not only breaking and making unusable UI on v3.5.0, it is also crashing badly with OOM on 3200mb guaranteed deployed pod, with a lot of archived workflows (postgresql) and few at etcd (6 hours TTL on worfkflows with working GC).

The issue is at the main view where all workflows are listed.

Also probably on this pagination, it would be useful to change the defaults on time ranges to show. Currently, it is one month, but probably it would be better to have a default on 1 or 2 days, to free that argo-server list workflows. This, together with the flags “show archived” that you are commenting, would help a lot.

There are some optimizations we can do, e.g. https://github.com/argoproj/argo-workflows/issues/12030.

However, this issue is more specific to argo-server/backend. The root cause is that the pagination method we implemented for ListWorkflows() requires retrieval of the entire list of workflows at once.

Let’s follow-up on any slow queries in #13295.

In general, for any lingering issues in 3.5.8+, please open a new issue with specifics

I’m experiencing the same.

In the logs I see 2 queries being executed that are considered slow. The first count is scanning thousands of rows.

2024/06/28 14:01:18     Session ID:     00001                                                                                                                                                                                                                                         
    Query:          SELECT count(*) as total FROM `argo_archived_workflows` WHERE ((`clustername` = ? AND `instanceid` = ?) AND `namespace` = ? AND not exists (select 1 from argo_archived_workflows_labels where clustername = argo_archived_workflows.clustername and uid = argo_ar
chived_workflows.uid and name = 'workflows.argoproj.io/controller-instanceid') AND exists (select 1 from argo_archived_workflows_labels where clustername = argo_archived_workflows.clustername and uid = argo_archived_workflows.uid and name = 'workflows.argoproj.io/phase' and val
ue in ('Error', 'Failed', 'Pending', 'Running')))                                                                                                                                                                                                                                     
    Arguments:      []interface {}{"default", "", "workflows"}                                                                                                                                                                                                                             
    Stack:                                                                                                                                                                                                                                                                            
        fmt.(*pp).handleMethods@/usr/local/go/src/fmt/print.go:673                                                                                                                                                                                                                    
        fmt.(*pp).printArg@/usr/local/go/src/fmt/print.go:756                                                                                                                                                                                                                         
        fmt.(*pp).doPrint@/usr/local/go/src/fmt/print.go:1211                                                                                                                                                                                                                         
        fmt.Append@/usr/local/go/src/fmt/print.go:289                                                                                                                                                                                                                                 
        log.(*Logger).Print.func1@/usr/local/go/src/log/log.go:261                                                                                                                                                                                                                    
        log.(*Logger).output@/usr/local/go/src/log/log.go:238                                                                                                                                                                                                                         
        log.(*Logger).Print@/usr/local/go/src/log/log.go:260                                                                                                                                                                                                                          
        github.com/argoproj/argo-workflows/v3/persist/sqldb.(*workflowArchive).CountWorkflows@/go/src/github.com/argoproj/argo-workflows/persist/sqldb/workflow_archive.go:241                                                                                                        
        github.com/argoproj/argo-workflows/v3/server/workflow.(*workflowServer).ListWorkflows@/go/src/github.com/argoproj/argo-workflows/server/workflow/workflow_server.go:193                                                                                                       
        github.com/argoproj/argo-workflows/v3/pkg/apiclient/workflow._WorkflowService_ListWorkflows_Handler.func1@/go/src/github.com/argoproj/argo-workflows/pkg/apiclient/workflow/workflow.pb.go:1826                                                                               
        github.com/argoproj/argo-workflows/v3/server/apiserver.(*argoServer).newGRPCServer.RatelimitUnaryServerInterceptor.func5@/go/src/github.com/argoproj/argo-workflows/util/grpc/interceptor.go:65                                                                               
        github.com/argoproj/argo-workflows/v3/server/apiserver.(*argoServer).newGRPCServer.ChainUnaryServer.func6.1.1@/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25                                                                                     
        github.com/argoproj/argo-workflows/v3/server/auth.(*gatekeeper).UnaryServerInterceptor.func1@/go/src/github.com/argoproj/argo-workflows/server/auth/gatekeeper.go:98                                                                                                          
        github.com/argoproj/argo-workflows/v3/server/apiserver.(*argoServer).newGRPCServer.ChainUnaryServer.func6.1.1@/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25                                                                                     
        github.com/argoproj/argo-workflows/v3/util/grpc.glob..func1@/go/src/github.com/argoproj/argo-workflows/util/grpc/interceptor.go:45                                                                                                                                            
        github.com/argoproj/argo-workflows/v3/server/apiserver.(*argoServer).newGRPCServer.ChainUnaryServer.func6.1.1@/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25                                                                                     
        github.com/argoproj/argo-workflows/v3/server/apiserver.(*argoServer).newGRPCServer.PanicLoggerUnaryServerInterceptor.func4@/go/src/github.com/argoproj/argo-workflows/util/grpc/interceptor.go:26                                                                             
        github.com/argoproj/argo-workflows/v3/server/apiserver.(*argoServer).newGRPCServer.ChainUnaryServer.func6.1.1@/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25                                                                                     
        github.com/grpc-ecosystem/go-grpc-middleware/logging/logrus.UnaryServerInterceptor.func1@/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/logging/logrus/server_interceptors.go:31                                                                             
        github.com/argoproj/argo-workflows/v3/server/apiserver.(*argoServer).newGRPCServer.ChainUnaryServer.func6.1.1@/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25                                                                                     
        github.com/grpc-ecosystem/go-grpc-prometheus.init.(*ServerMetrics).UnaryServerInterceptor.func3@/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:107                                                                                         
    Error:          upper: slow query                                                                                                                                                                                                                                                 
    Time taken:     16.07711s                                                                                                                                                                                                                                                         
    Context:        context.Background                                                                     

Should this be a new issue? The server does not hang, but it’s continuously timing out and almost unusable.

I’m not sure it is the same. haven’t seen the same crash behaviour as 3.5, it’s just slow to respond if I have workflows in multiple namespaces, when using the UI, it is slow to respond when changing the namespace.

@ryancurrah What bulk review? I don’t see any blocking review comments from @agilgur5 in the PR.

@terrytangyuan, @agilgur5 was doing the bulk of the review here why did it get merged before he gave his final review?

@Guillermogsjc Are you sure about this behaviour? In my case the workflows are sorted by the started time per page, and the behaviour is defined by UI.

sure, look this deployment of v6 on pro env, and those not archived workflows with so many days. Nevermind @jiachengxu this does not happen in the other environment, so maybe those are corrupt manifests somehow, do not bother with it

image

How does it compare to the previous version? I am expecting that the memory usage should be lower since I did the optimization mentioned https://github.com/argoproj/argo-workflows/issues/12025#issuecomment-2044952613 and also https://github.com/argoproj/argo-workflows/pull/12912.

Loads of memory were breaking 2700mib on older versions and now in our case stay bounded around 1.6 gb by pod. I would say it is significantly lower, but more strict checks are welcome

Sqlite versions look like missing a big amount of workflows from archived (etcd workflows appear normally)

@agilgur5 @Guillermogsjc I finally reproduced the issue, and it is indeed related to the cursor querying. As I mentioned the pagination method above, I was trying to query the archived workflows which are older than the oldest live workflow startedAt, and this caused the missing of the archived workflows which started earlier before this oldest live workflow, but were completed and archived already.

The original reason that I used this oldest startedAt to query archived workflows was to avoid duplications because the workflows that have newer startedAt can exist both in live workflow and archived workflow. However, because the change I made here(sqlite db only stores the live workflows that don’t have workflows.argoproj.io/workflow-archiving-status set to Archived), we get rid of the workflows which are archived but still in the k8s cluster on the sqlite-based informer level, we don’t have the duplications anymore.

I pushed a new image that contains the above fix/improvement to quay.io/jiachengxu/argocli:fix-v6, please give it a try.

@agilgur5 I’ve passed this request over to some folks that can test this out. May not be immediate.

Also would be great if we could get other users here to test quay.io/jiachengxu/argocli:fix-v6 and report any feedback as well as their k8s version, number of archived and live workflows, and resource usage, especially memory usage and the difference in memory usage from 3.4.x to the in-memory SQLite DB! See this comment above as an example of some performance numbers That would greatly help detect any possible issues with this change that may only appear in certain scenarios or certain scales (such as those that were previously missed in the 3.5.x RCs, leading to this issue).

cc @jmeridth I know you previously mentioned that you could test this out

Addressing some questions and assumptions from above comments:

  • Static: Succeeded, Failed and Error

@Guillermogsjc the existing terminology in Argo for this is “Completed” (as opposed to “Static”).

As my current knowledge (to be corrected if wrong), a workflow that reaches a status in the static set, gets archived into the optional database when finishes.

Correct. More specifically, Workflows are labeled for archiving when they are completed. This label is then picked up by the Informer which kicks off archiving.

As I’ve written a few times before though, there are race conditions when a Workflow is in both.

( maybe only when a workflow is resumed it goes from archived into etcd and in this specific case you could have a ongoing workflow on the two sources?)

*retried. Yes that is one of the scenarios I have mentioned before, and as I wrote there, there are more scenarios where a Workflow will be in both.

Probably it is an over-simplification, but, could the UI just bifurcate the queries depending on status between the two sources?

As such, yes, this is indeed an over-simplification.

As @jiachengxu wrote above, the cursor pagination approach (with equivalent filtering & sorting now implemented for both in-cluster and archived with the new in-memory SQLite DB) avoids all those races and does the “bifurcation” within the API itself (matching the intent of a unified API in https://github.com/argoproj/argo-workflows/issues/10781#issuecomment-1489263668). That approach should also work the same whether or not an Archive DB is enabled.

  • To scale properly, at argo documentation, there is recommended to keep workflow TTLs as short as possible.

For reference, this is already mentioned in the Cost Optimization docs, as it is a best practice at large enough scale.

I haven’t read your updated code, so maybe the pseudo-code here is imprecise, but do you mean this with cursor pagination?

Yes, the limit and offset are the cursor pagination, and the offset here is meant to be the query params in the requests, I have updated the previous comment to make it more clear.

As, depending on the sorting / ordering, there could be archived workflows in between X and X + Y

Yes, this is the second case that I describe above

  • Get X live workflows starting from Y from sqlite store descendingly ordered by startedAt time.

👍 For reference, note that this was previously not possible due to lack of etcd ordering / sorting. (See also https://github.com/argoproj/argo-workflows/issues/4546#issuecomment-728989425, https://github.com/argoproj/argo-workflows/pull/3565#issuecomment-663157888, https://github.com/argoproj/argo-workflows/issues/3157#issuecomment-1977940399, #2926, https://github.com/kubernetes/kubernetes/issues/80602)

Also, the continue for the next page could not be computed precisely

This should also fix #11715 (which was re-opened due to #12068 from this issue)

  • If X + Y < num(live workflows), we return workflows from live workflow in-memory database directly

I haven’t read your updated code, so maybe the pseudo-code here is imprecise, but do you mean this with cursor pagination? As, depending on the sorting / ordering, there could be archived workflows in between X and X + Y. In other words, num() does not necessarily account for sorting. If you’re using cursors though, I think this should work, just that + and num() are a bit imprecise to describe cursors.

This is also still better than the current approach either way.


Also, thanks again @Guillermogsjc for testing and @jiachengxu for iterating and describing in detail the approach here 🙂

hi, sorry @agilgur5 for not being able to answer before

Could you provide a 3.4 comparison @Guillermogsjc ?

For anyone else testing, this number in particular is very relevant to the memory usage, so if you have a lot of etcd workflows normally, it would be useful to see your memory usage and diff

This is more hard than rolling just the image between v3.5.5 (currently deployed), I would try to do as soon as possible the test to give feedback, do not know if anyone else having problems with memory could also collaborate here.

In any case let’s keep discussion to the MVP that fixes the current issues at hand and not stray too far. There are certainly future features that could use this, but this thread is long and winding as is and fixing the bugs is the top priority. If there are memory issues or other bugs, we can discuss those then.

yes sorry for the noise introduced

Have you put an eye on those logs or could reproduce this incidence? https://github.com/argoproj/argo-workflows/issues/12025#issuecomment-1984909833 I think this bug on a deployment with one replica is quite relevant because was an absolute crash into unhealthy state @jiachengxu @agilgur5 . Have you tested both cases @jiachengxu when the deployment has persistence configured and when not? Because the logs I provided align with the later one, it is a deployment that has persistence configured like:

  persistence: |
    connectionPool:
      maxIdleConns: 100
      maxOpenConns: 0
      connMaxLifetime: 10000s
    nodeStatusOffLoad: true
    archive: true
    archiveTTL: 90d
    skipMigration: false
    postgresql:
      host: argo-server-postgresql-backend.argo
      port: 5432
      database: postgres
      tableName: argo_workflows
      # the database secrets must be in the same namespace of the controller
      userNameSecret:
        name: argo-postgres-config
        key: username
      passwordSecret:
        name: argo-postgres-config
        key: password
      ssl: false
      # sslMode must be one of: disable, require, verify-ca, verify-full
      # you can find more information about those ssl options here: https://godoc.org/github.com/lib/pq
      sslMode: disable

This could be one of the shortcomings of having this in-memory approach, do not know if it could be adjusted.

Yea the in-memory approach with replicas was something I was concerned about more generally with Informers too. For replicas, we could potentially use one of the newer distributed SQLite DBs, but that may actually hamper performance and cause more races due to requiring consensus between replicas. We could also shard the Workflow space between the replicas and appropriately direct traffic, although that is substantially more complex. For in-memory, could potentially have a flag to use the disk, that would allow operators to trade-off between higher memory + higher performance to lower memory + lower performance (+ higher disk space). (similar can be done with a tmpfs too, although that’s likely less efficient than using a built-in in-memory mode).

To address the memory issue, the entries in the SQLite database do not necessarily need to be the complete workflow.

Interestingly enough, I proposed something similar for Controller sharding in https://github.com/argoproj/argo-workflows/issues/9990#issuecomment-1732337352. If this partial storage approach works well for the Server, regardless of sharding or not, this would be a way to enable greedy, lock-less Controller sharding as well (with a pure in-memory map, no need for a DB there).

which is continuously populated by a watch stream to K8s API

I was also concerned about watcher issues in k8s 1.27+, i.e. #11657. For anyone testing this, please list your k8s version as well, would definitely like to test on 1.27+

argo server pods are stable in memory usage around 160 MB, each replica is provisioned with 3200Mi in EC2 m7g.medium.

k8s 1.28

  • over 91585 archived workflows
  • around 200 concurrent etcd workflows

I can not really compare with v3.5.5 because it was not breaking on memory, probably because I dropped selecting SUCCEEDED workflows sistematically (to avoid the break), so normal navigation to compare here is hard.

once the sqlite is initialized for all replicas, this should be ok

mmm observing this comment, the message Internal Server Error: no such table: argo_workflows, is happening much more than 3 times (indeed it is happening constantly). Are there maybe workers and SQLites are initialized per worker in the replica? In this case it would surely balloon memory, together with an almost eternal Internal Server Error: no such table: argo_workflows message. If this is not the case, on one of the replicas there is a permanent lack of initialization and every time the endpoint derivates to it just fails (it looks more like this thing).

it throws

exec /bin/argo: exec format error

on an amd64 node

(previously running v3.5.5 with the same manifest, just changed image)

@Guillermogsjc Sorry that I forgot to push linux/amd64 image, could you give it another try?

Hi community, I have created a PR to implement the proposal (sqlite-based memory store for live workflows), and the PR is ready for review https://github.com/argoproj/argo-workflows/pull/12736.

For testing the PR, I have created an image and pushed it to quay.io: https://quay.io/repository/jiachengxu/argocli, and you can replace the image of the argo-server container in the argo-server Deployment with quay.io/jiachengxu/argocli:latest for testing.

Feedback is welcome!

I think if @jiachengxu made a comment here then GitHub would allow us to assign him.

2.i If it still have problems even after revert to before this function, could it be the problem below?

When I tested it, there was a problem that it freeze when I quickly clicked on the previous and next pages. when I looked at it in the F12 developer tool, I saw that it freeze when the pending status of eventSource type data accumulated a lot.

Oh this just got reported in https://github.com/argoproj/argo-workflows/issues/12626#issuecomment-1931197829 with a lot of details, so I’m going to investigate that soon. But from a quick glance, I wouldn’t think that would impact the Server, just the UI client.

Also, Considering the filter function requested by many users, filtering is expected to become more complicated when db resources and k8s resources are combined into one. I think that managing different resources for k8s and db on different pages would be more stable and show more accurate data.

Yea that’s my thought process as well. Long-term, we’ll only get more filters and they are easier and less error prone when separated than when combined. I mentioned this on Slack as well: with the unified view, any new filter has to immediately have two implementations (k8s and DB) with very different characteristics, and handle merging / deduplication. That’s a lot of complexity for a filter which makes it both harder to contribute and harder to review, making it more error prone especially to such performance regressions.

It is crystal clear that the options (UI):

A) show only etcd workflows B) show only archived workflows C) show both

are really missed from v3.4 and necessary 😃

Is there any way to disable the feature through configuration? I see discussions about the UI, but can we get a configuration to disable loading archived workflows in that view?

We don’t need the archived workflows in that view. We have already doubled the amount of resources for the server and we keep getting timeouts when loading the workflow view.

Spoke offline with @sunyeongchoi who will be busy for a few weeks. In the meantime, I think we should get a client-side workaround (the checkboxes) out ASAP since this is the most upvoted P1 bug and blocks some 3.5 upgrades.

@jmeridth do you want to restart the work on the checkboxes? If not, I can tackle that probably this week

@jmeridth Thank you for your explanation! Yes I’ll apply archived checkbox UI in this PR. So you can stop the work.

@sunyeongchoi yes. Sorry. If you will do the archived checkbox work as part of this PR I can stop the work I started locally for it. I do not have a GitHub PR for it. Does this help?

I think this problem can be solved by filtering through k8s’ label selector.

This is a tad incorrect as well unfortunately. It is relatively close though, as the exceptions are somewhat edge cases / race conditions:

  • A Pending Workflow can already be in the Archive DB before it is marked as Archived (that’s a race condition that exists).
  • An Archived Workflow is not necessarily the same in the Archive DB and in the cluster.
    • For example, if you retry an Archived Workflow, the state in the cluster and in the DB will differ, at least temporarily. That’s why there is some front-end logic (see also #11906 and #11336) to prefer the in-cluster version if available.

I mentioned these in my previous comment as well

Would it be a good idea to add a filter or page that only shows Workflows and Archived Workflows separately?

Yes please, if you are ok with it, recovering this option is very necesary, at least until the feature “merged view” scales well (bifurcating by the labels said here or similar) and is stable.

If really need to show two resources(Archived, Workflows) on one page, I think this problem can be solved by filtering through k8s’ label selector.

	// LabelKeyWorkflowArchivingStatus indicates if a workflow needs archiving or not:
	// * `` - does not need archiving ... yet
	// * `Pending` - pending archiving
	// * `Archived` - has been archived and has live manifest
	// * `Persisted` - has been archived and retrieved from db
	// See also `LabelKeyCompleted`.
	LabelKeyWorkflowArchivingStatus = workflow.WorkflowFullName + "/workflow-archiving-status"

I think it would be a good idea to discuss again whether to separate these resources or show them together.

And personally, I think archived workflow is unmanaged item. It stored in database(3th party object) and from that point on, argo workflows is not responsible for the data stored like this. (I actually can’t understand why argo workflow provides api for archived workflow. It feels like argo workflows offers an artificial storage(s3) lookup api.)🤔

ongoing workflows -> Workflows / k8s API / etcd static workflows -> Archived / backend database

I think this way might be good. I will implement and then test it. thank you 😃

If this is the primary option we are considering, I would ask if I could take some time to take a look through in-depth myself

Of course! If you do, I would be very grateful.

I heard that Archived and Workflows was separated page before and merged them recently. because user should not need to know the difference between two types of workflows.

So I’m also considering third option that showing Archived and Workflows together as much as possible.

Options 1 and 2 are natural and <v3.5.0 arch ones, those should definitely exist to avoid feature regression (currently happened at v3.5.0 where you lost this ability to query either etcd or your backend db).

Best way for 3rd option would skip deduplication and also the forbidden fetch all.

There are two kind of workflow status:

  • ongoing: pending and running
  • static: successful, failed, error

In a deployment where archiving process is robust, there are no reasons to query static state workflows against etcd. Similarly, there are no ongoing workflows on backend database.

Based on this statement, you can on option 3, just break the query in two disjoint sets:

  • ongoing workflows -> etcd
  • static workflows -> backend database

When there is needed to show a set on the UI that contains both kinds, exhaust first ongoing list, then go for static list.

The only lose of ergonomics for the user here are the case where static workflows are younger than ongoing ones, as those will be placed without honoring creation timestamp order. But this situation is understandable and the overall balance positive.

I propose that we add a dropdown to allow users to select whether to display:

  1. Only live workflows;
  2. Only archived workflows;
  3. Both live and archived workflows (with notice/warning that this is only suitable when the number of workflows is not too large);

I think this is the best way.

However, instead of paginate Archived and Workflows separately in step 3, fetch all Archived and Workflows just once at first and combine them and remove duplicates.

And store that data in the cache. And when go to next page, we use data in the cache.

It is not yet clear whether it is possible to store it in the cache, and should also investigate the informer cache suggested above.

But if possible, I think this is the best way.

Are there any other good opinions?

@sunyeongchoi Great! Let me know if you need any help.

Thank you so much for so many people suggest good ideas.

First, I will start with optimizing Archived Workflows first.

we should only query X workflows from the archive, where X is the requested page size. The underlying database does support filtering and sorting, so this is efficient. The fact that we query everything from archive is nonsensical.

After that I will investigate the informer cache 😃

When API server is returning a unified list of live + archive, it would call List() against the informer cache rather than List against K8s API, and then filter/merge/sort before returning results back to the caller.

ahh, I see - I didn’t have much knowledge of the k8s api, so didn’t realize it doesn’t really support filtering/ordering/pagination.

The 3 options I see are:

  • separate the k8s backed api/ui from the db backed api/ui (reverting to previous behaviour)
  • fetch the whole k8s data and merge it with a subset of the db data
  • persist workflows in the db from the moment they are submitted, updating their state as they are scheduled/change state. then (if the db archive is enabled) make the UI solely reliant on querying from the db. In this case, the data could be augmented with data from the k8s API if the workflows are still running…

@sunyeongchoi and I have discussed a couple of potential solutions. Unfortunately, there are edge cases that we cannot get around with those approaches (due to the complexity of pagination, deduplication, sorting, etc.).

What are some of those edge cases? We can still over/under fetch so long as it does not overload the server. For instance, in the worst-case, if a user has 20 Workflows per page set, we can retrieve 20 from k8s and 20 from the Archive DB, which is not horrendous (but for sure could be optimized).

Did the previous Archived Workflows page not have pagination? If so, I would think it would have been similarly susceptible to this, just not as frequently hit since it was a separate page.

4. The first two options are almost identical to previous versions, but the UI should be less buggy since they now share most of the implementation; the third option is an addition to previous versions.

I feel like separate pages is a better UX than a drop-down. If the APIs are identical, some careful refactoring could make them share a UI implementation.

For posterity, this was actually discussed yesterday at the Contributors Meeting. @jmeridth had been looking into it as it is blocking his team from upgrading as well and had eventually traced it to this PR discussion: https://github.com/argoproj/argo-workflows/pull/11761#discussion_r1317888160.

(I was involved as I made substantial refactors to the UI for 3.5 – #11891 in particular – in case those were the cause, but the UI is actually unrelated in this case, and the refactor actually decreased the number of network requests. Also #11840 removed a default date filter, but that was entirely client-side anyway, so did not impact any networking.)

Hello. I will test the issue as soon as possible and think about a solution. thank you.

Thanks for this issue. This is a known issue if you have a lot of archived workflows. It’s caused by the pagination method that first loads all live workflows and archived workflows and then performs pagination. cc @sunyeongchoi who worked on this in https://github.com/argoproj/argo-workflows/pull/11761.