dapr: Proposal: Building block API for cryptography

I think we need a way to also make envelope encryption exposed through Dapr. Some cloud providers (AWS, IBM) expose envelope encryption as a first class concept. Additionally, it is a common pattern for enabling secure high-throughput client side encryption. It might be a worth concept to have as a building block for distributed apps exposed through Dapr.

Context

So what is envelop encryption?[1], [2], [3]

Envelope encryption is the practice of encrypting plaintext data with a data key, and then encrypting the data key under another key.

Why even doing this? From the KMS Tutorial (GCP docs for envelope encryption is also pretty good):

AWS KMS can encrypt or decrypt data up to 4 KB in length. This can make it difficult to work with larger pieces of data. Additionally, when you encrypt data, it must be transferred over the network, processed by KMS, and returned; often, the associated network load and latency and complexity makes this undesirable.

A popular alternative to encrypting data directly with KMS is the use of envelope encryption . Rather than sending the data to KMS to be encrypted, we use KMS to generate a data key, which is then encrypted with a KMS-managed master key, or CMK. We then use this key - not the CMK itself - to encrypt and decrypt data, locally, where we are processing the data.

So the idea of envelope encryption is that

1. You generate a random key per encryption request to encrypt your data (your data encryption key or DEK). Data of any size can be encrypted locally, fast and unbounded by your cloud provided limits/quotas.
2. You encrypt your data with this DEK.
3. You encrypt this DEK with a key stored in your key management service, in a process called key wrapping. That will be your Key Encryption Key (or CMK, depending on the terminology 😉 ). Multiple DEK can be encrypted with the same KEK.
4. The DEK along with the data it wrapped (encrypted) are stored/sent together.

Why envelope encryption?

1. Reduced costs when working for cloud services - although some providers+SDKs will charge you for envelop encryption, the costs of using envelop encryption are lower than requesting keys and encryption of all data directly to the cloud provider.
2. Resiliency and increased TPS - cloud providers have limits on the rate of encryption operations one can requests and amount of data requests. Envelop encryption reduces those.
3. Decreased latency - encrypted data doesn’t have to leave premises and are encrypted locally, saving transfer and latency fees
4. Consistency - cloud providers already offer this. By not offering an equivalent to this on Dapr we alienate those customers and decrease Dapr usefulness on this regard.

Support by cloud provider and encruption

• AWS exposes envelope encryption as a first class concept in their SDK.

• Azure Key Vault doesn’t expose this directly in its APIs. It is handled transparently with its managed services and performed by its client libraries for storage

• GCP doesn’t explicitly offer envelop encryption as a first class concept in their SDK but its Tink Cryptographic library exposes it. That said, envelope encryption is the under-the-hood solution for encryption data with asymmetric cryptography (but that is expected).

• IBM cloud exposes envelop encryption as an API

• Hashicorp Vault transit backend also offers primitives for generating DEK through its generate-data-key operation. src

Why we need Dapr support? Can’t users implement this themselves?

One thing to keep in mind regarding envelop encryption is that when one performs the key wrapping step, metadata about the KEK used (version, name etc) has to be persisted together with the (now encrypted) DEK. This is required so so KEK rotation can be performed without impacting already issued DEK and data they encrypt. KEK metadata is not something we are exposing to clients in this proposal. So in order to offer client-side encryption using envelop encryption we would need to do it on the component side.

Again, some SDKs expose this as a first class concept. If we don’t offer the right abstraction for this customers will be barred from this.

EDITS

• Section "Why we need Dapr support
• Note about vault transit backend having support for envelope encryption

We discussed this issue in the community call today. Here goes the general feedback on it (including some of my recommendations too):

1. Encrypt API over HTTP

POST http://localhost:3500/v1.0/crypto/{component}/encrypt
Payload goes directly into the body.

Response: RAW encrypted payload in the HTTP response body, passing response metadata (encryptedKey) via Headers Alternative: Return the Dapr’s encryption envelope as JSON

{
  "ciphertext": "AJGIGBKJGH",
  "encryptedKey": "BJKHGIUXCBKLDLK"
}

1. Decrypt API over HTTP

POST http://localhost:3500/v1.0/crypto/{component}/decrypt
Response body from `encrypt` goes directly into this HTTP request body and propagate Headers too since `encryptedKey` is required to decrypt.

Response: RAW plaintext in the HTTP response body.

1. Built-in encryption component

• Cipher information is part of the component
• Fetches encryption key from secret store

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: mycipher
spec:
  type: crypto.builtin
  version: v1
  metadata:
  - name: cipher
    value: AES256
  - name: secretStore
    value: mysecretstore
  - name: encryptionKeyName
    value: mysymmkey

1. Encryption components

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: vault
spec:
  type: crypto.hashicorp.vault
  version: v1
  metadata:
  - name: vaultAddr
    value: "https://127.0.0.1:8200"
  - name: encryptionPath
    value: /transit/encrypt/mykey

1. Dapr encryption envelope Dapr runtime will have an opinion on how to encrypt messages and minimize data sent/received from encryption services. It means that the Dapr runtime will generate on-time encryption keys and encrypt those with services instead of encrypting the whole payload. The payload is then encrypted with the on-time encryption key instead. The Dapr’s encryption envelop will contain the ciphertext and the encrypted key (at least). The envelope can be split into body and headers in the encrypt response or serialized as JSON.

My understanding is that the high-level encrypt/decrypt is implementing client-side/envelope encryption, where the encryption of payloads happens in the dapr runtime, and NOT in the KeyVault / KSM? Have I got that right?

Correct.

There was previous discussion around allowing key names to be dynamic

In the proposal I wrote above, the key name is always dynamic, meaning that components have access to all keys they are authorized to, and apps need to specify what key they want to use.

I think doing the opposite (hardcoding a key in the component’s YAML) would also encourage bad practices, such as using the same key for data encipherment and signing.

So I’m reading this as you’re trying to protect developers from themselves, and channeling them to use gRPC just incase they ever need to encrypt large files? Alternatively, could you allow HTTP but set a max request size soft-limit to some upper bound that you feel comfortable allowing? Rationale : HTTP is incredibly accessible for developers, and many do not use gRPC, do we want to alienate those developers who are not comfortable with gRPC?

Yes, this is a situation where we’re essentially trying to protect the developers from themselves. The idea is that access to the high level APIs should happen using the Dapr SDKs (also because the code to perform the “back and forth” over the gRPC streams is not going to be very simple).

If there’s strong demand for the API to be exposed over HTTP, I think it can be considered. But people should be aware that it does require all data to be loaded into memory first, ballooning the memory requirements of the Dapr sidecar.

I don’t think Dapr would add value by providing low level primitives - there are plenty of libraries doing that today.

The value Dapr provides is that it abstracts the various cloud key vaults. All libraries that offer the primitives expect the key material to be passed directly.