rclone: Add --no-obscure flag so we don't try to reveal passwords
What is the problem you are having with rclone?
There is a requirement that the password for SFTP be obscured. If you are using an ENV VAR to set the password along with other attributes for an SFTP connection the system will error:
Failed to create file system for "sftp:": input too short when revealing password - is it obscured?
This requirement creates challenges in automation as the process to use ENV VAR to run rclone with break with SFTP. Oddly, the requirement to obscure is not present for other sensitive information like an AWS Access and Secret Key.
The only method that worked was running the rclone config process manually for SFTP with the same info present in the variables. The ideal would be that this is not a forced setting when using ENV VARs.
What is your rclone version (eg output from rclone -V)
rclone -V
rclone v1.40
- os/arch: linux/amd64
- go version: go1.10
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Alpine Linux 3.7.x
Which cloud storage system are you using? (eg Google Drive)
SFTP
The command you were trying to run (eg rclone copy /tmp remote:tmp)
rclone lsd sftp:
Here is a sample of the ENV used:
RCLONE_CONFIG_MYSFTP_TYPE=sftp
RCLONE_CONFIG_MYSFTP_HOST=sftp.host.io
RCLONE_CONFIG_MYSFTP_USER=test-user
RCLONE_CONFIG_MYSFTP_PORT=443
RCLONE_CONFIG_MYSFTP_PASS=mypassword
RCLONE_CONFIG_MYSFTP_KEY_FILE=
RCLONE_CONFIG_MYSFTP_USE_INSECURE_CIPHER=false
RCLONE_CONFIG_MYSFTP_DISABLE_HASHCHECK=
or
RCLONE_CONFIG_MYSFTP_TYPE=sftp
RCLONE_CONFIG_MYSFTP_HOST=sftp.host.io
RCLONE_CONFIG_MYSFTP_USER=test-user
RCLONE_CONFIG_MYSFTP_PORT=443
RCLONE_CONFIG_MYSFTP_PASS=mypassword
About this issue
- Original URL
- State: open
- Created 6 years ago
- Reactions: 9
- Comments: 36 (18 by maintainers)
Commits related to this issue
- Make obscure deterministic Just use base64 encoding and decoding instead of encrypting with a random IV. This should fix https://forum.rclone.org/t/different-cache-dir-suffixes-for-same-password/4111... — committed to jleeuwes/rclone by jleeuwes 8 months ago
- Make obscure deterministic Just use base64 encoding and decoding instead of encrypting with a random IV. This should fix https://forum.rclone.org/t/different-cache-dir-suffixes-for-same-password/4111... — committed to jleeuwes/rclone by jleeuwes 8 months ago
- Make obscure deterministic Just use base64 encoding and decoding instead of encrypting with a random IV. This should fix https://forum.rclone.org/t/different-cache-dir-suffixes-for-same-password/4111... — committed to jleeuwes/rclone by jleeuwes 8 months ago
Having this feature would be nice, we store our
RCLONE_SFTP_PASSin vault, and inject it into a kubernetes pod env variable at runtime, so its already secure. In order to get this to work I had to change the name of the variable in vault, and add this line in a bash script before my rclone process runs in the kubernetes pod. Not that its impossible to get around, but it would be nice to disable this behavior with a runtime flag.I think it will be easier to replace the obscurity algorithm with a simple base64 of password. This will be simplier to automate, because many programs already have base64 conversion out of the box (ansible, k8s, etc.), and this approach will still be protected from the “eye behind the shoulder” scenario.
OMG just found
revealin source!!! https://github.com/rclone/rclone/blob/master/cmd/reveal/reveal.go Thisrclone revealshould defenetly land in docsP.S. Now rclone become two times awsome! 😃
Thanks, I’ll have a look.
--no-obscurewould be a really nice addition for automation via scripts. I’m calling rclone via Python’s subprocess module, and those arguments are never revealed anywhere, there is no chance of ending up in .bash_history, etc. It’s just an internal communication via processes, and calling that separate subprocess first to obscure the password is a bit un-elegant. Nothing too bad, but I found rclone in general to be more “elegant” than this 😃I take your point. However an
--unobscuredor--no-obscureflag would have to work for all methods of configuration as by the time the config is used we don’t know whether it came from the config file, a flag or an environment variable.I guess we could mark passwords as
:unobscured:mysecretpassword:unobscured:- would that work for you?Well, using
rclone revealis kind of a hack for achieving idempotence - dump config, reveal obscured value from dump, obscure value from variable, in rclone.conf set from dump if passwords match or obscured - if doesnt match… So my +1 for--no-obscure😦Do you fancy working on this? It would be reasonably straight forward I think. Adding a new flag
--no-obscuresay then using this to disable the Reveal and Obscure functions. Though we’d need to keep a version of Reveal for the cloud provider secrets.I understand the idea here for the obscure SFTP password, but if someone is glancing at your SFTP password you have bigger problems 😀
A config file should be viewed as something that needs to be secured in general. AWS does the same thing with the credentials files you download. The keys are in plain text on your system. You need to make sure the info is secure.
I think the issue is right now the obscure password command is required for SFTP unless using
rclone config. This creates an odd separation of responsibilities on who and where securing secrets should occur. In this case, SFTP has a unique implementation different from other secrets. This will break any ENV use case that would leverage SFTP, something that is not true for other secrets.For reference, I authored this project: https://github.com/openbridge/ob_bulkstash/
The sample config is set in an ENV config https://github.com/openbridge/ob_bulkstash/tree/develop/env
The ENV file itself is often encrypted, or if not using a file, can be done via other means with Docker secrets, Amazon KMS or other systems.