taskwarrior: 3.0.0 sync with GCloud suddenly stopped working - "Failed to synchronize with server"

Nice work! I set up a service account to replicate this, without the storage.objects.delete permission, and sure enough it works for a few syncs but then stops when it comes time to overwrite latest.

However, I don’t reproduce exactly the error you’ve seen – a newer version in sync_meta than in latest. Instead, I see

⸩ task sync
Failed to synchronize with server: taskwarrior-testing@taskwarrior-testing.iam.gserviceaccount.com does not have storage.objects.delete access to the Google Cloud Storage object.

which before #3411 would have just said “Failed to synchronize with server”. My bucket has

latest
salt
v-00000000000000000000000000000000-faeb7fdd9677421f8adbfae4f2565142 
v-faeb7fdd9677421f8adbfae4f2565142-fb1e1171d5f347d9a4ee05dd6860ba77 

so two versions: faeb7f followed by fb1e11. The latest file contains faeb7f.., so it wasn’t updated. And select * from sync_meta gives the same, so it also wasn’t updated. So, all of this failed in an expected way that wouldn’t lead to OutOfSync. However, if I run task sync again:

⸩ task sync
Syncing with GCP bucket djmitche-taskwarrior-devel

It works?! And even more bizarre, sync_meta gets updated to fb1e11...

Ah, I see what’s happening here! On the last (successful) run of task sync, the replica begins by trying to update to the latest version from the server. So, it looks for the child version of faeb7f and finds fb1e11, and applies that locally. But it shouldn’t - that version is not on the chain from latest!

So, there are two bugs here now.

• https://github.com/GothenburgBitFactory/taskchampion/issues/387 - the child-version search must check against latest
• #3441 - updating the manpage to include the required permission.

Any chance you want to make a PR for the second one?

Thanks so much for your patience tracking this down. That’s three bugs found and soon to be fixed in one issue!

Damn, got the same error again:

$ ./task add Become a sponsor of Taskwarrior because djmitche has been an absolute legend
Created task 21.

$ ./task sync
[taskchampion/taskchampion/src/replica.rs:233:26] e = OutOfSync
Failed to synchronize with server

gcloud bucket appears unchanged since the last screenshot (made sure to refresh too). I DID NOT end up trying with my other devices like I said I was going to at the end of my last comment, by the way.

It looks like it successfully uploaded a new version, and subsequent runs had nothing new to upload so didn’t fail. Maybe try modifying a task and running ./task sync (with the local build) again?

Huh, so the table is actually

Version	Parent	Notes
a32e4b	000000	In bucket, latest
ec3c99	a32e4b	In bucket

which means that the replica uploaded ec3c99, but somehow latest didn’t get updated, yet it still stored its version in the sync_meta table.

Here’s the code that handles updating latest: https://github.com/GothenburgBitFactory/taskwarrior/blob/ef9613e2d610d81bf3ea88eaefc51ad188707773/taskchampion/taskchampion/src/server/cloud/server.rs#L371-L386 So, if that compare_and_swap operation fails, it deletes the object from the bucket and returns AddVersionResult::ExpectedParentVersion with the value from latest. Here’s the code that calls add_version: https://github.com/GothenburgBitFactory/taskwarrior/blob/ef9613e2d610d81bf3ea88eaefc51ad188707773/taskchampion/taskchampion/src/taskdb/sync.rs#L82-L113 That updates the base_version in sync_meta if it gets AddVersionResult::Ok, or tries again if it gets AddVersionResult::ExpectedParentVersion.

So presumably there was an attempt to upload new version ec3c99 which

• did upload the version object to the bucket
• did not successfully change the value of latest
• did not delete the version from the bucket
• did update base_version in sync_meta

I think what that means is that compare_and_swap failed to update latest but didn’t return false. That’s implemented here: https://github.com/GothenburgBitFactory/taskwarrior/blob/ef9613e2d610d81bf3ea88eaefc51ad188707773/taskchampion/taskchampion/src/server/cloud/gcp.rs#L105-L167 and looking at all the places Ok(..) occurs there (the return values), those are all false before the call to upload_object, and then false if that failed with a 412 error. The upload_res? statement there should handle any other error as an actual error, propagated back and causing task sync to fail. Looking at the implementation of is_http_error suggests that even the 412 is represented as Result::Err so I think that’s a valid assumption.

So, I’m stumped – what have I missed?

Thanks for your patience, @jcoffa. The good news is, the fix here is pretty easy. Just put ec3c99a195cd4ab09f950c61cf249807 (with no newline!) in the latest file in your bucket and things should start working again.

Are you able to recompile from source? If so, I can make a nice patch to hopefully get better debugging info. I filed #3411 to track the error message problem.