argo-workflows: 3.5 ListWorkflows causes server to hang when there are lots of archived workflows
Pre-requisites
- I have double-checked my configuration
- I can confirm the issues exists when I tested with
:latest - I’d like to contribute the fix myself (see contributing guide)
What happened/what you expected to happen?
We had >200,000 rows in the workflow archive table, and when trying to view the new combined workflow/archived workflow list page in the UI, the server times out
scanning the code, it looks like the LoadWorkflows code loads all rows from the archive table, combines them with the k8s results and then applies sorting and limiting.
as a workaround, we’ve reduced the archive ttl from 14 days to 1 day, and the endpoint now responds before timing out, but is still pretty slow.
Version
v3.5.0
— edits below by agilgur5 to add updates since this is a (very) popular issue —
Updates
- Broadly caused by #11121, although some specific regressions were introduced later, see below
- Most of the performance regression part of this issue should have been solved by https://github.com/argoproj/argo-workflows/pull/12068, which was released in v3.5.1
- That did re-instate a different bug, however: https://github.com/argoproj/argo-workflows/issues/11715
- Another performance regression was fixed in #12912, which was released in v3.5.6
- Performance was also improved by #13021 / #12736, which was released in v3.5.7
- There are still some remaining regressions due to the Archived + Live merge in 3.5, which should be fixed in later patches now that Live Workflows can be filtered via the SQLite DB:
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 45
- Comments: 123 (64 by maintainers)
Commits related to this issue
- feat: add sqlite-based memory store for live workflows. Fixes #12025 (#12736) Signed-off-by: Jiacheng Xu <xjcmaxwellcjx@gmail.com> — committed to argoproj/argo-workflows by jiachengxu 3 years ago
- chore: Revert "feat: add sqlite-based memory store for live workflows. Fixes #12025 (#12736)" This reverts commit f1ab5aa32f766090998bcaf5a2706c7e3a6cc608. Signed-off-by: Alan Clucas <alan@clucas.or... — committed to Joibel/argo-workflows by Joibel 3 years ago
- feat: add sqlite-based memory store for live workflows. Fixes #12025 (#13021) — committed to argoproj/argo-workflows by jiachengxu 3 years ago
@sunyeongchoi and I have discussed a couple of potential solutions. Unfortunately, there are edge cases that we cannot get around with those approaches (due to the complexity of pagination, deduplication, sorting, etc.).
I propose that we add a dropdown to allow users to select whether to display:
Additional requirements:
Motivations for this proposal:
Any thoughts?
I’ve had a few recent internal design discussions about how to fix this. I wanted to share the proposal, as well as a commitment from my team to fixing this in the near term (the coming weeks).
We want to fix the performance issues while preserving the feature. The crux of the issue is that K8s querying capabilities are limited to what the UI needs. So the solution to preserving the feature is to somehow make it more queryable to support the use cases of the UI. Namely:
Earlier, I proposed using informer as an in-memory cache on the API server to perform efficient lookup without querying K8s. But the concerns about this were that:
Instead of an informer cache, we are now proposing that Argo API server maintain an in-memory SQLite database of the live workflows, which is continuously populated by a watch stream to K8s API. As workflows are created, updated, or deleted, the entries in SQLite would automatically be updated as well.
To address the memory issue, the entries in the SQLite database do not necessarily need to be the complete workflow. The UI already uses field selection to get a partial workflow and so only certain fields would need to be stored. Apart from that, the live workflow entries would also be stored in a compressed and space-efficient format as YAML/JSON is highly compressible.
The benefits we see of this approach are:
In the end, the result will be that we maintain the current UX that we currently have in v3.5, but with the performance issues resolved. It also gives us room in the future not to be hampered by whatever limitations the K8s API, and add additional querying capabilities that SQL provides.
I think the underlying reason is to try to show normal workflows and archived workflows together, but I don’t agree to show both together. Even in api, the two workflows are separated, but it feels rather confusing to show them together on the UI.
I am reverting related change https://github.com/argoproj/argo-workflows/pull/12068 for now since otherwise the UI is not usable when there are many workflows. In the meantime, we can continue discussing proposal https://github.com/argoproj/argo-workflows/issues/12025#issuecomment-1774346636 here. We can probably release a patch version next week https://github.com/argoproj/argo-workflows/issues/11997#issuecomment-1775681491.
With the unified workflow list API for both live + archived workflows we have to do the following:
Number 1 is a challenge because performing LIST all workflows for every request will be taxing on K8s API server as users use the UI. Listing all workflows for the purposes of only showing 20 is extremely inefficient. To get around this, I propose a technique we use in Argo CD:
Since LIST all workflows is the requirement and also the problem, we can use an informer cache on Argo API server to avoid the additional LIST calls to k8s and have all workflows ready to be returned by API server from in-memory informer cache. When API server is returning a unified list of live + archive, it would call List() against the informer cache rather than List against K8s API, and then filter/merge/sort before returning results back to the caller.
Note that this does balloon the memory requirements of argo-server because of the informer cache. But I feel this is manageable with archive policy and gc. And the benefits of a unified live+archive API, as well as reducing load on K8s API server, outweigh the extra memory requirements of argo-server.
If we are worried about expensive processing / sorting of the results of List() calls we make against the informer cache, we could consider maintaining our own ordered workflow list (by creationTimestamp) that is automatically updated with UpdateFunc, AddFunc, DeleteFunc registered to the informer. But I consider this an optimization.
I mentioned above that I think checkboxes in the left sidebar would make more sense then a select box. I also mentioned that we could hide it as “Extra Filters” in a modal as well.
Checkboxes and label being:
WORKFLOW STORAGE
We could have a little info icon next to the “Archived” checkbox linking to the Workflow Archive docs as well. We could also explain within info icons that “In-cluster” means as a k8s resource stored in etcd and that “Archived” means stored as a row in an Archive DB
The plan is to make them scale properly, this issue is still very much listed as a bug. We also did discuss this in the most recent Contributor Meeting and a solid use-case did pop up there for filtering out Archived Workflows: CI wherein new commits cancel existing Workflows which are then immediately archived – quick sequential commits can cause a lot of clutter. Similar fast-updating use-cases fit that too. The converse I’m not as sure of on a use-case, showing only Archived Workflows and no In-cluster Workflows
And currently PR only has a page that shows
WorkflowsandArchived Workflowscombined.Would it be a good idea to add a filter or page that only shows
WorkflowsandArchived Workflowsseparately?Just reverted back to
v3.5.0-rc1due to this. Perhaps we can add the default started time back?3.ii +1 to “just add back the Archived Workflows list page”
@agilgur5 yes, I can restart it.
I’m checking the logic, but regarding the archived object’s label being
Persisted, is it possible to solve the duplicate issue by filtering the workflows?I didn’t want to intrude initially. Wanted to make sure you had time to work with it. But if we are now considering what are effectively total reverts, then I think it would be wise to have more contributors take a look before going down that direction.
This is unfortunately an overly optimistic assumption that doesn’t hold up for all cases. It is mostly correct as a simplifying assumption, but not entirely. For example, even in “robust” deployments, it is very common to keep completed Workflows in k8s/etcd for some period of time before GC’ing them. This is especially the case for failed or errored Workflows, where someone may want to manually investigate the Pods etc (similarly to a dead letter queue). There’s also the race conditions when Workflows are in the process of being Archived. Both can exist simultaneously during that time. Similarly there is the case of retrying an Archived Workflow – retries are a mutable operation (compared to resubmit, which creates a new Workflow), so the Workflow is no longer “static” in that case.
There is actually specific code to deal with this edge case of having a Workflow in both the Archive and in k8s/etcd, c.f. #11336, #11371, etc.
And then of course there is also the case of not having configured a Workflow Archive at all.
I’m not so sure that this would be considered a “feature regression”. The API and UI were intentionally unified in #11121. As per the original issue #10781, an Argo user (not necessarily an operator) generally does not know about or care about the difference between the two.
Having optional checkboxes (not a dropdown) could make sense as those are then just “filters”. Those could also be hidden behind a modal/menu on the UI as most users won’t change the default. That doesn’t solve this issue though for when both are checked.
I am also wondering if it may be simpler to keep the APIs decoupled and have the UI do merging. This was added as a backward-compatible option to the API, so the
/archived-workflows/API is still available, andargo archive listis also still available in the CLI. EDIT: This option was previously rejected in https://github.com/argoproj/argo-workflows/issues/10781#issuecomment-1489263668@sjhewitt thank you for filing this. I’m having the same issue. 30+ second page loads. I’m gathering data now and will post here once obtained.
@sunyeongchoi thanks for taking a look.
Can confirm this isn’t related to the “past month” default on the UI being removed from the search box like @agilgur5 states. That is only client side.
For reference, v3.5.7 has been released with a backport of #13027
@agilgur5 Thank you for letting me know about this regression, I think this is related.
As you suggested, I am going to open a PR to fix this regression. Also, I will include this change on the top of the sqlite-based in memory PR so that folks can check if the memory usage decreases with the fix.
Furthermore, inspired by this regression, I think we even don’t need to store the entire live workflow in the sqlite-based informer(based on the code change of the regression, only
name,namespace,uid,phase,startedat,finishedatare needed), in that case, we will be able to reduce even more memory usage so the in-memory sqlite db doesn’t balloon much memory.I will let the community know when the above improvements are implemented and push images so that community members can test them out.
@Guillermogsjc Thank you so much for testing it. I found the issue, and it is related to how pagination is implemented currently.
The current pagination (on main branch) is implemented as follows, given the
limit Xandoffset Yas the cursor pagination params:startedAttime.This pagination mechanism never worked due to Kubernetes’ limited capability of pagination.
Once we switched to use the in-memory sqlite-based store for live workflows(in PR https://github.com/argoproj/argo-workflows/pull/12736), I tried to use the same mechanism:
startedAttime.startedAttime.This definitely improved the situation, but still doesn’t work and it could lead to having duplications in the results(example here: https://github.com/argoproj/argo-workflows/issues/12025#issuecomment-1793482528). Also, the
continuefor the next page could not be computed precisely, that’s why during the testing, sometimes it throwsNo workflows existingor related issues.So I came up with the following approach, given the
limit Xandoffset Yas the cursor pagination params::X + Y < num(live workflows), we return workflows from live workflow in-memory database directly, and no need to get archved workflows and merge them, since even if we merge them, the live workflow will take high priority and will be returned.Y < num(live workflows) && X + Y > num(live workflows), this means the return will contain both live and archived workflows, in this case:Yin the sqlite inmemory db descendingly ordered bystartedAttime, mark count of the returned live workflowZ, and the last element in the return which has the oldeststartedAttimeT,(X - Z)in archived workflows, older thanT, and descendingly ordered bystartedAttime.Y > num(live workflows), this means that all workflows will be fetched from archived workflow:startedAttimeT,Xin archived workflow with starting fromY-num(live workflows), older thanT, and descendingly ordered bystartedAttime.This approach provides the following benefits:
continueandshowRemainingItemCountcan be computed precisely and correctly so that UI and client can work better.I have pushed this implementation to
quay.io/jiachengxu/argocli:fix-v3, note the tag isfix-v3, feel free to test and see if there are any issues.In general, it works (UI) as a charm, filtering labels, state filtering, etc. Pretty fast over tons of ectd and archived workflows 😃
When going through pagination, sometimes (did not not observe any particular pattern) it is thrown:
Internal Server Error: no such table: argo_workflowsbut it self heals after few seconds and the internal error dissapears without doing nothing
It is a deployment with 3 replicas, maybe there is something to do with one replica not having the in-memory SQLite created in the another replica, and load balancer derivating pagination requests to replicas “not initialized”. Maybe deterministic initializations of those sqlite could be settled somehow to allow balancing between them in a replicaset.
This could be one of the shortcomings of having this in-memory approach, do not know if it could be adjusted.
Another path to allow the distributed way could be to put those auxiliary tables on the dedicated database (in case of existing in the deploy), to avoid the in-memory on the case of multiple replicas + dedicated database at least.
1.i That’s right. this is also why I implemented this feature https://github.com/argoproj/argo-workflows/pull/12237. In my opinion, if this problem cannot be resolved immediately, I think it would be okay to temporarily provide a separate page first.
2.i If it still have problems even after revert to before this function, could it be the problem below?
When I tested it, there was a problem that it freeze when I quickly clicked on the previous and next pages. when I looked at it in the F12 developer tool, I saw that it freeze when the
pendingstatus ofeventSourcetype data accumulated a lot.3.ii I agree with this opinion. Of course, we have already implemented to show both data(k8s & db) on one page through k8s labels selector https://github.com/argoproj/argo-workflows/pull/12237/. However, I also had many concerns while implementing the function. Because combining the two data with labels is not completely cover everything, as you have said many times. Also, Considering the filter function requested by many users, filtering is expected to become more complicated when
dbresources andk8sresources are combined into one. I think that managing different resources for k8s and db on different pages would be more stable and show more accurate data.All of these scenarios are important to us. We are using argo-workflows for CI. The last view that @agilgur5 mentions is important if I need to reference test failures from a previous workflow run. I may filter by label and then by archived.
A checkbox on the left panel that is checked by default is the ideal UI change IMO. I have started a PR locally for it but don’t want to waste the time if the work is already done as part of ~#12397~ (update: wrong one) #12237 (update: correct one)/cc @sunyeongchoi
I mentioned above that I feel a drop-down seems like clunkier UX than a separate page (the previous behavior) to me.
If this is the primary option we are considering, I would ask if I could take some time to take a look through in-depth myself (I was not previously involved with this change). I’ve built lists with complex sorting, filtering, and pagination with efficient implementations in the past. If needed, usually the worst-case scenario is that you have to send to the API a list of all IDs to exclude (e.g. duplicates).
Fetching all is pretty much never an option. There is always an amount at which it can overload a server and is also a big load spike.
Even with an Informer cache (which only handles in-cluster Workflows), it can cause spikiness when the Informer rebuilds the cache periodically as well. Spikiness is generally a behavior to avoid from an operations perspective (my day-to-day work these days is primarily SRE-focused), if possible. Informers are a bit more transparent than generic caches, but even then, it adds a layer of complexity (i.e. more potential for bugs, which we have quite a bit of, some due to Informer misconfigurations) to the Server that we should avoid if possible.
I think it’s worth noting that this balloons the memory requirements of each Argo Server Pod. If you have a standard, 3 replica HA set-up, your memory usage is going to go way up.
I’m similarly curious… I wonder if it would be possible to use a cursor that encodes 2 offsets - one for the k8s api and one for the db, then fetches
limitrows from both sources with the given offset, merges the results together and applies the limit to that combined list.something like:
@agilgur5 , I switched to 3.5.8 just now. I still see lag when switching namespaces in argo workflows UI.
We are running 3.5.7 with a few 1000 archived workflows. The UI is still loading awfully slow, compared to the 3.4 branch. The page loads in 20-30 seconds. How can I verify that this fix is working as intended? What other things can I tinker with to get faster loading times.
Thank you both for helping me gain a clearer understanding of the current review process.
I was the main analyzer here in the issue, but I haven’t had time to give #12736 an in-depth review unfortunately. Getting the approach right here was always my top priority though and I think we ironed the heck out of it. I didn’t see any glaring issues either after the CGO removal, otherwise I would’ve left a blocking comment. If other Approver+ folks gave it a thorough look-over, I think it’s had sufficient due diligence. If there are follow-up issues or follow-up code review comments, we can follow up on them (see also https://github.com/argoproj/argo-workflows/pull/12912#discussion_r1572901283 as an example of that). I wouldn’t anticipate any substantial changes though; that’s why I was trying to iron out the approach after all.
We otherwise don’t have firm rules on the process outside of the branch protection requirements (we don’t have many Approvers either, see also the Sustainability Effort).
So I think Terry made the right call there (and also gave some more time than strictly necessary for others to take a look too, which I’m pretty sure was intentional). Would be great if I could have taken a deeper look too, but unfortunately we have bottlenecks and this is the top P1, so we all have to use our best available judgment often.
Please correct me if I missed anything:
If you see any additional issues, feel free to follow up by commenting here or submitting a new issue.
this partitioning looks optimal
on v6 there are all archived workflows and looks fine
it uses plenty of memory when pagination is done
but seems bounded somehow, so it would be tweakeable by deployment (the important point is to have bounded memory usage on operations depending on
workflowcolumn mean size of tables)just to remember that v6 behaviour shows first all non archived, then archived, so this can feel weird if you have very old etcd workflows (mostly because TTL worker errored deleting them I guess), but this is not a main concern probably.
@agilgur5 I can confirm that it is not seen this behaviour at least on our deployment, in v3.5.5
v3.5.5 shows all the workflows from both sources. Sqlite versions look like missing a big amount of workflows from archived (etcd workflows appear normally)
@jiachengxu it is an issue that has been around all the time in all fixes 😦
@jiachengxu v4 feedback:
on several pagination iterations it is found:
Internal Server Error: sql: Scan error on column index 7, name "annotations": converting NULL to string is unsupported-> this looks as a small bug related to some incomplete filtering.I find this really weird gap on pagination from few hours to tons of days (there are obviously workflows between those two timestamps):
from memory point of view looks stable (but it was already quite stable on v3), deployed so far v3 with really few incidents of memory and restart of the pods.
Hi @agilgur5, I have created a PR to fix the regression: https://github.com/argoproj/argo-workflows/pull/12912 Also, I have included the change into the sqlite-based in memory PR so that folks can check if the memory usage decreases with the fix. This can be tested with
quay.io/jiachengxu/argocli:fix-v4. cc: @Guillermogsjc @jmeridthI tried to implement this but it looks like this doesn’t work for live workflows since there are more fields (e.g.,
status.nodes) needed by the UI. I think this can be considered as a future improvement for the sqlite-based in-memory solution.@Guillermogsjc Thank you for testing and giving the feedback!
This works if the archived database is enabled, but if the archived database is not enabled, I guess we still need to get them from the in-memory database?
The pagination of
fix-v3is implemented(for more details, please check above) in a way that doesn’t require query from both tables and perform merge, depends on the cursor pagination params, either query everything from only in memory, or in memory with archived workflow appended, or archived workflow only.deployed
quay.io/jiachengxu/argocli:fix-v2, fast and great when filtering by labels etc, but…when trying to paginate by 50 workflows (NEXT PAGE button) with Succeeded included (those 98 k in archived), it explodes
it is weird, it only happens time to time, not on every single pagination iteration
also, when filtering by a label, sometimes, it throws
No workflows existing, for workflows that exist on archived workflows (and in a future query moment, it gives a list again - not sure if exhaustive)Hi @Guillermogsjc, I did reproduce the issue on my local setup, and I think that was because sqlite multi-thread mode was disabled and that error could occur due to concurrency issues or race conditions when multiple threads or processes are accessing the same in-memory database simultaneously.
As you might know, I refactored the PR recently to make it cgo-free, and the lib(https://github.com/zombiezen/go-sqlite) which I am using will always enable the multi-thread mode by default.
I have created a new image and tested it, and I couldn’t see the error anymore. Could you pull the image and test it again in your setup? You can still pull the image with
latest, or you can usefix-v2as the image tag.there is something that is a dead end and makes single pod deployment to reach an unhealthy unusable status.
It was working Ok, then at a moment, it locked on
Internal Server Error: no such table: argo_workflowsforever.this log was collected from a replicaset with only 1 element:
logs -> logs-from-argo-server-in-argo-server-5967db98b8-wbvtr.log
more logs -> logs-from-argo-server-in-argo-server-85c66ffd6d-8dtsb.log
Thank you for the quick test @Guillermogsjc ! Would you mind sharing your scale, e.g., how many live workflows and archived workflows you have?
Yeah, this is also my thought, once the sqlite is initialized for all replicas, this should be ok. However, I am more concerned that this approach balloons the memory usage too much, which could cause OOM for the argo server pods, in this case, the pods might be OOM killed all the time and this error will happen from time to time. Would you mind sharing how much memory that per argo server pod is consuming and how much it was before? Have you observed any argo server pods got OOM killed?
@agilgur5 I like the combined view but we may need to split again for a bit until we address some of the items you mention. Thank you for deep diving on this stuff and also taking over the handoff.
So I took over the client-side workaround this week as @jmeridth has been busy on the day job etc, although I managed to get pretty sick with the flu the past 2 weeks too 😅 (EDIT: and then started passing a kidney stone… 😖)
Was working on this today and I did notice a handful of things that may warrant further discussion:
selectall fieldsnamefilters per #12161, but there’s a reason that filter didn’t exist for non-Archived Workflows: it’s not k8s native and therefore won’t necessarily be performant.WorkflowTemplatelist. That PR ended up also needing to request allWorkflowTemplatesfrom the k8s API to implement the filter: https://github.com/argoproj/argo-workflows/pull/11684#discussion_r1469090044Answers, thoughts, feedback, concerns, discussion, etc welcome. Feel free to quote a specific part of that big list above that you’re responding to. Can also use the list numbers, e.g. 3.ii.a
I’m thinking about UI design to show Workflows and Archived Workflows separately. I’m thinking about using a select box like the picture below. Do you have any other good ideas?
for the 3rd case query, where the problems arise,
You could break the query this way
Workflows/ k8s API / etcdArchived/ backend databaseAnd paginate exhausting first ongoing workflows generator/list, then static workflows generator ( at least for sure against postgresql database you could get a cursor/generator sorted by creation timestamp with skip N elements with K elements window).
This way could be nice if you take the assertion all static workflows are successfully written in
Archived. Which is an assertion that (I believe) happens in robust deployments.Thanks for your idea and good example. However, I think that this way also creates a situation where duplicate values cannot be removed as follows, unless sorting is performed after the entire search.
First Page as (
Archived):Second Page as (
Archived):Second Page as (
Workflows):values > 2023-01-03:@sunyeongchoi That might be a good optimization for the third option I listed in https://github.com/argoproj/argo-workflows/issues/12025#issuecomment-1774346636. It could help a lot when there are a lot more archived workflows vs live workflows. Although we still need to fetch the entire list of live workflows on all pages.
@Guillermogsjc Thanks for the suggestion. However, we still need to make the same list query in the backend and then filter in the front-end.
it is not only breaking and making unusable UI on v3.5.0, it is also crashing badly with OOM on 3200mb guaranteed deployed pod, with a lot of archived workflows (postgresql) and few at etcd (6 hours TTL on worfkflows with working GC).
The issue is at the main view where all workflows are listed.
Also probably on this pagination, it would be useful to change the defaults on time ranges to show. Currently, it is one month, but probably it would be better to have a default on 1 or 2 days, to free that argo-server list workflows. This, together with the flags “show archived” that you are commenting, would help a lot.
There are some optimizations we can do, e.g. https://github.com/argoproj/argo-workflows/issues/12030.
However, this issue is more specific to argo-server/backend. The root cause is that the pagination method we implemented for
ListWorkflows()requires retrieval of the entire list of workflows at once.Let’s follow-up on any slow queries in #13295.
In general, for any lingering issues in 3.5.8+, please open a new issue with specifics
I’m experiencing the same.
In the logs I see 2 queries being executed that are considered slow. The first count is scanning thousands of rows.
Should this be a new issue? The server does not hang, but it’s continuously timing out and almost unusable.
I’m not sure it is the same. haven’t seen the same crash behaviour as 3.5, it’s just slow to respond if I have workflows in multiple namespaces, when using the UI, it is slow to respond when changing the namespace.
@ryancurrah What bulk review? I don’t see any blocking review comments from @agilgur5 in the PR.
@terrytangyuan, @agilgur5 was doing the bulk of the review here why did it get merged before he gave his final review?
sure, look this deployment of v6 on pro env, and those not archived workflows with so many days. Nevermind @jiachengxu this does not happen in the other environment, so maybe those are corrupt manifests somehow, do not bother with it
Loads of memory were breaking 2700mib on older versions and now in our case stay bounded around 1.6 gb by pod. I would say it is significantly lower, but more strict checks are welcome
@agilgur5 @Guillermogsjc I finally reproduced the issue, and it is indeed related to the cursor querying. As I mentioned the pagination method above, I was trying to query the archived workflows which are older than the oldest live workflow
startedAt, and this caused the missing of the archived workflows which started earlier before this oldest live workflow, but were completed and archived already.The original reason that I used this oldest
startedAtto query archived workflows was to avoid duplications because the workflows that have newerstartedAtcan exist both in live workflow and archived workflow. However, because the change I made here(sqlite db only stores the live workflows that don’t haveworkflows.argoproj.io/workflow-archiving-statusset toArchived), we get rid of the workflows which are archived but still in the k8s cluster on the sqlite-based informer level, we don’t have the duplications anymore.I pushed a new image that contains the above fix/improvement to
quay.io/jiachengxu/argocli:fix-v6, please give it a try.@agilgur5 I’ve passed this request over to some folks that can test this out. May not be immediate.
Also would be great if we could get other users here to test
quay.io/jiachengxu/argocli:fix-v6and report any feedback as well as their k8s version, number of archived and live workflows, and resource usage, especially memory usage and the difference in memory usage from 3.4.x to the in-memory SQLite DB! See this comment above as an example of some performance numbers That would greatly help detect any possible issues with this change that may only appear in certain scenarios or certain scales (such as those that were previously missed in the 3.5.x RCs, leading to this issue).cc @jmeridth I know you previously mentioned that you could test this out
Addressing some questions and assumptions from above comments:
@Guillermogsjc the existing terminology in Argo for this is “Completed” (as opposed to “Static”).
Correct. More specifically, Workflows are labeled for archiving when they are completed. This label is then picked up by the Informer which kicks off archiving.
As I’ve written a few times before though, there are race conditions when a Workflow is in both.
*retried. Yes that is one of the scenarios I have mentioned before, and as I wrote there, there are more scenarios where a Workflow will be in both.
As such, yes, this is indeed an over-simplification.
As @jiachengxu wrote above, the cursor pagination approach (with equivalent filtering & sorting now implemented for both in-cluster and archived with the new in-memory SQLite DB) avoids all those races and does the “bifurcation” within the API itself (matching the intent of a unified API in https://github.com/argoproj/argo-workflows/issues/10781#issuecomment-1489263668). That approach should also work the same whether or not an Archive DB is enabled.
For reference, this is already mentioned in the Cost Optimization docs, as it is a best practice at large enough scale.
Yes, the
limitandoffsetare the cursor pagination, and theoffsethere is meant to be the query params in the requests, I have updated the previous comment to make it more clear.Yes, this is the second case that I describe above
👍 For reference, note that this was previously not possible due to lack of etcd ordering / sorting. (See also https://github.com/argoproj/argo-workflows/issues/4546#issuecomment-728989425, https://github.com/argoproj/argo-workflows/pull/3565#issuecomment-663157888, https://github.com/argoproj/argo-workflows/issues/3157#issuecomment-1977940399, #2926, https://github.com/kubernetes/kubernetes/issues/80602)
This should also fix #11715 (which was re-opened due to #12068 from this issue)
I haven’t read your updated code, so maybe the pseudo-code here is imprecise, but do you mean this with cursor pagination? As, depending on the sorting / ordering, there could be archived workflows in between
XandX + Y. In other words,num()does not necessarily account for sorting. If you’re using cursors though, I think this should work, just that+andnum()are a bit imprecise to describe cursors.This is also still better than the current approach either way.
Also, thanks again @Guillermogsjc for testing and @jiachengxu for iterating and describing in detail the approach here 🙂
hi, sorry @agilgur5 for not being able to answer before
This is more hard than rolling just the image between v3.5.5 (currently deployed), I would try to do as soon as possible the test to give feedback, do not know if anyone else having problems with memory could also collaborate here.
yes sorry for the noise introduced
Have you put an eye on those logs or could reproduce this incidence? https://github.com/argoproj/argo-workflows/issues/12025#issuecomment-1984909833 I think this bug on a deployment with one replica is quite relevant because was an absolute crash into unhealthy state @jiachengxu @agilgur5 . Have you tested both cases @jiachengxu when the deployment has persistence configured and when not? Because the logs I provided align with the later one, it is a deployment that has persistence configured like:
Yea the in-memory approach with replicas was something I was concerned about more generally with Informers too. For replicas, we could potentially use one of the newer distributed SQLite DBs, but that may actually hamper performance and cause more races due to requiring consensus between replicas. We could also shard the Workflow space between the replicas and appropriately direct traffic, although that is substantially more complex. For in-memory, could potentially have a flag to use the disk, that would allow operators to trade-off between higher memory + higher performance to lower memory + lower performance (+ higher disk space). (similar can be done with a tmpfs too, although that’s likely less efficient than using a built-in in-memory mode).
Interestingly enough, I proposed something similar for Controller sharding in https://github.com/argoproj/argo-workflows/issues/9990#issuecomment-1732337352. If this partial storage approach works well for the Server, regardless of sharding or not, this would be a way to enable greedy, lock-less Controller sharding as well (with a pure in-memory map, no need for a DB there).
I was also concerned about watcher issues in k8s 1.27+, i.e. #11657. For anyone testing this, please list your k8s version as well, would definitely like to test on 1.27+
argo server pods are stable in memory usage around 160 MB, each replica is provisioned with 3200Mi in EC2 m7g.medium.
k8s 1.28
I can not really compare with v3.5.5 because it was not breaking on memory, probably because I dropped selecting SUCCEEDED workflows sistematically (to avoid the break), so normal navigation to compare here is hard.
mmm observing this comment, the message
Internal Server Error: no such table: argo_workflows, is happening much more than 3 times (indeed it is happening constantly). Are there maybe workers and SQLites are initialized per worker in the replica? In this case it would surely balloon memory, together with an almost eternal Internal Server Error: no such table: argo_workflows message. If this is not the case, on one of the replicas there is a permanent lack of initialization and every time the endpoint derivates to it just fails (it looks more like this thing).@Guillermogsjc Sorry that I forgot to push
linux/amd64image, could you give it another try?Hi community, I have created a PR to implement the proposal (sqlite-based memory store for live workflows), and the PR is ready for review https://github.com/argoproj/argo-workflows/pull/12736.
For testing the PR, I have created an image and pushed it to quay.io: https://quay.io/repository/jiachengxu/argocli, and you can replace the image of the
argo-servercontainer in theargo-serverDeployment withquay.io/jiachengxu/argocli:latestfor testing.Feedback is welcome!
I think if @jiachengxu made a comment here then GitHub would allow us to assign him.
Oh this just got reported in https://github.com/argoproj/argo-workflows/issues/12626#issuecomment-1931197829 with a lot of details, so I’m going to investigate that soon. But from a quick glance, I wouldn’t think that would impact the Server, just the UI client.
Yea that’s my thought process as well. Long-term, we’ll only get more filters and they are easier and less error prone when separated than when combined. I mentioned this on Slack as well: with the unified view, any new filter has to immediately have two implementations (k8s and DB) with very different characteristics, and handle merging / deduplication. That’s a lot of complexity for a filter which makes it both harder to contribute and harder to review, making it more error prone especially to such performance regressions.
It is crystal clear that the options (UI):
A) show only etcd workflows B) show only archived workflows C) show both
are really missed from v3.4 and necessary 😃
Is there any way to disable the feature through configuration? I see discussions about the UI, but can we get a configuration to disable loading archived workflows in that view?
We don’t need the archived workflows in that view. We have already doubled the amount of resources for the server and we keep getting timeouts when loading the workflow view.
Spoke offline with @sunyeongchoi who will be busy for a few weeks. In the meantime, I think we should get a client-side workaround (the checkboxes) out ASAP since this is the most upvoted P1 bug and blocks some 3.5 upgrades.
@jmeridth do you want to restart the work on the checkboxes? If not, I can tackle that probably this week
@jmeridth Thank you for your explanation! Yes I’ll apply archived checkbox UI in this PR. So you can stop the work.
@sunyeongchoi yes. Sorry. If you will do the archived checkbox work as part of this PR I can stop the work I started locally for it. I do not have a GitHub PR for it. Does this help?
This is a tad incorrect as well unfortunately. It is relatively close though, as the exceptions are somewhat edge cases / race conditions:
PendingWorkflow can already be in the Archive DB before it is marked asArchived(that’s a race condition that exists).ArchivedWorkflow is not necessarily the same in the Archive DB and in the cluster.ArchivedWorkflow, the state in the cluster and in the DB will differ, at least temporarily. That’s why there is some front-end logic (see also #11906 and #11336) to prefer the in-cluster version if available.I mentioned these in my previous comment as well
Yes please, if you are ok with it, recovering this option is very necesary, at least until the feature “merged view” scales well (bifurcating by the labels said here or similar) and is stable.
If really need to show two resources(
Archived,Workflows) on one page, I think this problem can be solved by filtering through k8s’ label selector.I think it would be a good idea to discuss again whether to separate these resources or show them together.
And personally, I think archived workflow is unmanaged item. It stored in database(3th party object) and from that point on, argo workflows is not responsible for the data stored like this. (I actually can’t understand why argo workflow provides api for archived workflow. It feels like argo workflows offers an artificial storage(s3) lookup api.)🤔
I think this way might be good. I will implement and then test it. thank you 😃
Of course! If you do, I would be very grateful.
I heard that
ArchivedandWorkflowswas separated page before and merged them recently. because user should not need to know the difference between two types of workflows.So I’m also considering third option that showing
ArchivedandWorkflowstogether as much as possible.Options 1 and 2 are natural and <v3.5.0 arch ones, those should definitely exist to avoid feature regression (currently happened at v3.5.0 where you lost this ability to query either etcd or your backend db).
Best way for 3rd option would skip deduplication and also the forbidden fetch all.
There are two kind of workflow status:
In a deployment where archiving process is robust, there are no reasons to query static state workflows against etcd. Similarly, there are no ongoing workflows on backend database.
Based on this statement, you can on option 3, just break the query in two disjoint sets:
When there is needed to show a set on the UI that contains both kinds, exhaust first ongoing list, then go for static list.
The only lose of ergonomics for the user here are the case where static workflows are younger than ongoing ones, as those will be placed without honoring creation timestamp order. But this situation is understandable and the overall balance positive.
I think this is the best way.
However, instead of paginate
ArchivedandWorkflowsseparately in step 3, fetch allArchivedandWorkflowsjust once at first and combine them and remove duplicates.And store that data in the cache. And when go to next page, we use data in the cache.
It is not yet clear whether it is possible to store it in the cache, and should also investigate the informer cache suggested above.
But if possible, I think this is the best way.
Are there any other good opinions?
@sunyeongchoi Great! Let me know if you need any help.
Thank you so much for so many people suggest good ideas.
First, I will start with optimizing Archived Workflows first.
After that I will investigate the informer cache 😃
ahh, I see - I didn’t have much knowledge of the k8s api, so didn’t realize it doesn’t really support filtering/ordering/pagination.
The 3 options I see are:
What are some of those edge cases? We can still over/under fetch so long as it does not overload the server. For instance, in the worst-case, if a user has 20 Workflows per page set, we can retrieve 20 from k8s and 20 from the Archive DB, which is not horrendous (but for sure could be optimized).
Did the previous Archived Workflows page not have pagination? If so, I would think it would have been similarly susceptible to this, just not as frequently hit since it was a separate page.
I feel like separate pages is a better UX than a drop-down. If the APIs are identical, some careful refactoring could make them share a UI implementation.
For posterity, this was actually discussed yesterday at the Contributors Meeting. @jmeridth had been looking into it as it is blocking his team from upgrading as well and had eventually traced it to this PR discussion: https://github.com/argoproj/argo-workflows/pull/11761#discussion_r1317888160.
(I was involved as I made substantial refactors to the UI for 3.5 – #11891 in particular – in case those were the cause, but the UI is actually unrelated in this case, and the refactor actually decreased the number of network requests. Also #11840 removed a default date filter, but that was entirely client-side anyway, so did not impact any networking.)
Hello. I will test the issue as soon as possible and think about a solution. thank you.
Thanks for this issue. This is a known issue if you have a lot of archived workflows. It’s caused by the pagination method that first loads all live workflows and archived workflows and then performs pagination. cc @sunyeongchoi who worked on this in https://github.com/argoproj/argo-workflows/pull/11761.