earthly: `failed to get edge: inconsistent graph state`

A user is reporting the following error, when running CI on a local, but re-used shared buildkit instance:

Error: build target: build main: failed to solve: failed to get edge: inconsistent graph state

The error is intermittent and usually works after a re-run. They are not using shared cache flags, due to previous suggestions we gave them.

See here for context: https://earthlycommunity.slack.com/archives/C01DL2928RM/p1669293808712079

Note I remember running into this error frequently when we first started developing satellites several months ago, but a combination of fixes and removal of shared cache flags resolved it (IIRC).

~Possibly related issue: https://github.com/earthly/earthly/issues/2957~

~EDIT: Since this issue is a top priority for us we will be providing weekly updates on our progress in #2957.~

EDIT 2: this issue is not related; we will continue to try to debug/reproduce it

About this issue

Original URL
State: open
Created 2 years ago
Reactions: 6
Comments: 22 (4 by maintainers)

Most upvoted comments

There doesn’t seem to be a correlation between renaming and the failures on our CI 😕 . In the last 30 days, we’ve seen inconsistent graph state 14 times. Of those:

10 builds only had modified files in the commit
1 build had deleted files in the commit
3 builds had added files in the commit
0 builds had file renames.

Note that these numbers only look at the commit for which the build was run. I don’t know the cache state before, so if some of these were on old branches with an outdated cache, there might’ve been renamed files in the meantime.

FelixDQ on Dec 18, 2023

We have also encountered it… I trier rebuilding failed job and I got same error 3 times in a row (I run it 4th time with --no-cache and it started)

Remote builkit

00:00:27.005             buildkitd | Connecting to tcp://buildkitd-earthly-buildkit-chart.jenkins.svc.cluster.local:8372...
00:00:28.374             buildkitd | Version github.com/earthly/buildkit v0.7.21 0b107b045b571b4f3184d6d975e567bd46d213bd
00:00:28.374             buildkitd | Platforms: linux/amd64 (native) linux/amd64/v2 linux/amd64/v3 linux/amd64/v4 linux/386
00:00:28.374             buildkitd | Utilization: 0 other builds, 0/20 op load
00:00:28.374             buildkitd | GC stats: 419 GB cache, avg GC duration 245ms, all-time GC duration 21m55.763s, last GC duration 0s, last cleared 0 B

Client version

 earthly version v0.7.20 9600f376026c11d23eb43fc68ad716d6ae4cca2e linux/amd64; Debian GNU/Linux 11 (bullseye)

Error

build target: build main: failed to solve: async force execution for +kbt-prepare: unlazy force execution: failed to get edge: inconsistent graph state

ingwarsw on Oct 31, 2023

Thanks for confirming the remote version.

We will soon migrate to satellites though, so maybe that will help?

It could help with greater log visibility, but unfortunately the problem exists there too (as it’s the same codebase)

We’ll continue to keep our eyes open for any reproduction-cases.

alexcb on Oct 30, 2023

#2957 has been fixed as of of v0.7.21; that issue was a race condition inside the buildkit scheduler, in particular when merging (deduplicating) multiple edges within the build graph. I think this issue is related, but triggered a different error message. I’m going to close this as fixed, but if it happens again please reopen this.

alexcb on Oct 24, 2023

Any updates or roadmap on this issue? @alexcb

We’ve been working through some data race warnings, which are documented under https://github.com/earthly/earthly/issues/2957#issuecomment-1625808777

We’ll be posting more updates under #2957 as we make progress.

alexcb on Jul 7, 2023

Also happens to us (using earthly version v0.7.5 0857e35cefb1a0155638f47b3ac0558844b48c3f linux/amd64; Arch Linux).

I’m actually running a single earthly command, but the projects includes a quite big graph (around 500-1000 targets with all nested permutations). Thinking about splitting that up in individual invocations for the time being.

Any updates or roadmap on this issue? @alexcb

sdaschner on May 19, 2023