age: Windows PowerShell 5.1 output file redirection corrupts files

Environment

OS: Windows 10 Pro
age version: v1.0.0-rc.3

What were you trying to do

I was trying to encrypt a g1.txt file containing text passwords with help of a passphrase. It got successfully encrypted with filename g1.txt.age. But when I tried to decrypt it, it is showing this error (see screenshot). age_error

What happened

Actually, the g1.txt file, which I am trying to encrypt, is already an encrypted file of passwords. What I did is: First, I encrypted my original txt file of passwords with help of AES pbkdf2 python program, which generated an encrypted file g1.txt. Then for further security, I thought that I should implement Age Encryption on the encrypted file itself (for double security). But after encryption, during decryption, it shows this error in windows terminal:

age: error: failed to read header: parsing age header: unexpected intro: "\xff\xfea\xoog\x00e\x00-\x00e\x00n\x00c\x00r\x
oOy\x00p\x00t\x00i\x000\x00n\x00.\x000\x00r\xO0g\x00/\x00v\x001\x00\r\x00\n"
age: report unexpected or unhelpful errors at https://filippo.io/age/report

So, I think the Age algorithm is unable to encrypt data which is already encrypted.

About this issue

Original URL
State: open
Created 3 years ago
Comments: 23 (7 by maintainers)

Links to this issue

Encrypting a file full of TOTP keys - method vs madness

Commits related to this issue

cmd/age: offer a hint when the file was corrupted by PowerShell I would still like to find a way to offer a warning while doing the encryption, rather than at decryption time, but better than nothing... — committed to FiloSottile/age by FiloSottile 2 years ago

Most upvoted comments

I simply downloaded the “age-v1.0.0-rc.3-windows-amd64.zip” file from pre-built binaries page on your Github repo. Nothing else I did.

namitgpta on Jun 28, 2021

The short version (having recently battled against PowerShell and Windows terminal subsystem) is that its best if the program (age) can write the file directly.

It (really does) help to understand that whereas Unix systems treat content going through a pipe as bytestreams (or at least as character streams of usually 8-bit bytes with no regard to encoding), Windows (in its own also very long established lineage) treats them as ‘text’ streams, which for PowerShell and its modern terminal-subsystem means UTF-16 LittleEndian text streams with all the painful cross-platforming phun that can imply. (Arguably though, that’s not correct since what flows across a ‘|’ in PowerShell is really an object, but …)

So any documentation (for Windows at least) would be provide by using the -o flag. Probably deserves a section on the main page.

Windows 10 (I’m on the latest; 20H1) with PowerShell will not provide a happy path when using redirections; the easiest way of using redirections is to redirect a textual output; then open it in an editor to change the encoding. That’s about as close to a happy-path as you’re likely to get at least until you can do ... | Out-File -Encoding UTF8NoBOM -Filename blah.txt

Presumably should shouldn’t be a problem if using WSL-2 or similar, but is sure is frustrating when trying to do DevOps kind of things in Windows (where I feel this kind of pain the most).

cameronkerrnz on Jul 2, 2021

This came up before, and as far as I know, this is a fundamental issue of PowerShell pipes and file redirection, and the way to fix it is to use | Out-File -Encoding UTF8NoBOM (which is a mouthful). Is rage handling this better? Is there a way we can tell PowerShell we are putting binary on stdout and ask pretty please not to UTF-16 it? Failing that, can we detect it to print a warning?

FiloSottile on Jun 28, 2021