restic: Cannot create/use repository in S3 with S3 Object Lock enabled
Output of restic version
restic 0.9.4 compiled with go1.11.4 on linux/amd64
How did you run restic exactly?
RESTIC_REPOSITORY=s3:s3.amazonaws.com/[myBucketName]/[myPrefixInBucket] \
RESTIC_PASSWORD_FILE=/home/me/.resticpassword \
./restic init --json
What backend/server/service did you use to store the repository?
S3, with S3 Object Lock enabled and configured to apply to all objects.
Expected behavior
Restic should have initialized the repository.
Actual behavior
Fatal: create key in repository at s3:s3.amazonaws.com/[myBucketName]/[myPrefixInBucket] failed: client.PutObject: Content-MD5 HTTP header is required for Put Object requests with Object Lock parameters
Steps to reproduce the behavior
Create a bucket with s3 object lock enabled and configured to apply to all objects. Here’s some (partial) terraform code:
resource "aws_s3_bucket" "bucket" {
bucket = "${var.name}"
versioning {
enabled = true
}
lifecycle_rule {
enabled = true
prefix = ""
abort_incomplete_multipart_upload_days = "7"
noncurrent_version_expiration {
days = "${var.noncurrent_version_expiration_days}"
}
}
object_lock_configuration {
object_lock_enabled = "Enabled"
rule {
default_retention {
mode = "COMPLIANCE"
days = "${var.object_lock_days}"
}
}
}
}
variable "name" {
type = "string"
default = "myBucketName"
}
variable "noncurrent_version_expiration_days" {
type = "string"
description = "In reality, this would be large, like 30-90."
default = "1"
}
variable "object_lock_days" {
type = "string"
description = "In reality this would be large, like 180. Don't make this big when testing!"
default = "3"
}
Do you have any idea what may have caused this?
Restic is attempting to PUT to S3 without computing Content-MD5 and including that in the request.
Do you have an idea how to solve the issue?
Compute Content-MD5 and set the header when making requests to s3.
Did restic help you or made you happy in any way?
😐
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 27 (10 by maintainers)
I agree that Object Lock should be a top priority as Veeam is really the only player in this space that supports it well. We are unable to use restic on the majority of our clients only because it lacks this feature.
PS. I love restic and use it in a hobbyist/small business environment.
The use case for Object Lock is to protect against destruction by an attack that gains as much access to the storage infrastructure as the authorized user has.
If Object Lock is supported, it is extremely practical to have a backup solution that is robust under that threat model.
It is challenging to achieve anything that is actually comparable to Object Lock—nearly impossible if you assume compromise of (your) administrator credentials.
Something that would work, but necessitates additional cost/points of failure:
The cost and fragility of that workaround is upsetting when consideration is given to what the roadblock actually is for Restic to directly use an S3 bucket that has Object Lock enabled: Restic needs to assemble cloud storage objects fully before issuing PutObject and include the MD5 when making the request. That’s it. Everything else is non-blocking UX improvements that apply to all WORM storage.
Aside: The requirement to build objects fully can actually be cheated a bit! Unlike Google Cloud Storage’s equivalent, S3’s Object Lock doesn’t preclude doing multi-part uploads. You can do the assemble-and-compute-MD5 at the granularity of each UploadPart, but there are minimum part size requirements and a limit on the number of parts that make this less flexible than one might imagine.
Think of it as a giant write-once CDROM: indeed, I think it’s safe to say that a remote attack is exceedingly unlikely to succeed against back-end AWS infrastructure, and an attacker targeting AWS is probably not going to target you as an individual user, so your alternative backups would probably survive such an event. Bottom line is, we agree that supporting Object Lock would address the existing known ransomware attack vectors.
Is this on the roadmap? Ransomware has been all over the news for months. The only reliable mechanism supplied by AWS and other S3-compatible cloud providers for defending against an attacker who has obtained API credentials to the targeted account’s data is Object Lock. This is a critical feature.
@instantlinux There are at least three ways to get working append only backups with restic, even with S3 - it is just not working out of the box as easily as it could. I am using different methods successfully for over a year now.
If you need help, send me a DM and I will try to help.
Why would no other solution work to prevent ransomware attacks? Are you suggesting that the only viable solution to ransomware attacks are object locks?
Regarding AWS, what if the attacker manages to gain access to Amazon’s systems and kills the object lock? To completely avoid that, you’d need to air-cap those systems too, just like anything else.
I get you, you want this feature. I’m not disputing that, just pointing out that there are other ways to reach similar protection (all of them have their pros and cons of course).
BTW, there’s a similar discussion (and some related references) in #3195.
My point is that the “other ways to solve the problem” do not work in the face of today’s ransomware attackers. A leaked AWS API credential can be used by an attacker to defeat any AWS-based alternative. Object Lock is the gold standard for protection against ransomware: please consider prioritizing this above “a lot of features”. As for non-AWS solutions, I don’t believe there is better protection against ransomware on any other cloud storage service; you’d have to air-gap your backups using Iron Mountain’s pickup/drop-off vault services (or the equivalent using your own vehicle).