sftp: "File does not exist" when copying remote file - SFTP client
I really wanted this to be a last resort, but I’m at my wits end trying to figure this out. My code is at the bottom. The main error I keep getting when using the typical configuration is “file does not exist.” WinSCP has no issues.
Things I’ve tried:
-
Attempted the same operation using RClone to make sure my code is not the issue (see https://forum.rclone.org/t/sftp-remote-to-local-copy-failed-to-copy-file-does-not-exist/14414/2). RClone fails with the “File does not exist” error.
-
sftp.NewClient(conn) with sftpClient.Open(srcInvPath) - Received the error: “File does not exist”
-
sftp.NewClient(conn, sftp.MaxPacket(20480)) with sftpClient.OpenFile(srcInvPath, os.O_WRONLY|os.O_CREATE|os.O_TRUNC) - Received the error: “read from 13 for 20480 from 20497 not supported. (SSH_FX_FAILURE)”
-
sftp.NewClient(conn) with sftpClient.OpenFile(srcInvPath, os.O_WRONLY|os.O_CREATE|os.O_TRUNC) - Received the error: “read from 13 for 32768 from 32785 not supported. (SSH_FX_FAILURE)”
-
sftp.NewClient(conn) with sftpClient.OpenFile(srcInvPath, 0700) - Received the error : “EOF”. Have no idea why I tried this.
package main
import (
"bufio"
"fmt"
"io"
"log"
"os"
"path"
"path/filepath"
"regexp"
"strings"
"github.com/pkg/sftp"
"golang.org/x/crypto/ssh"
)
var invImpRootPath = "C:\\Invoices\\"
var invImpErrPath = filepath.Join(invImpRootPath, "Errors")
var invImpProcPath = filepath.Join(invImpRootPath, "Processed")
var invRegExp = regexp.MustCompile(`(?m)^(Invoice)\.([0-9]*)\.([0-9]*)\.([0-9]{1,5})\.xml$`)
var sftpInvPath = "/Invoices/outbound"
func main() {
err := os.MkdirAll(invImpRootPath, os.ModeDir)
if err != nil {
fmt.Printf("Error creating Invoices directory: %s", err.Error())
panic(err)
}
err = os.MkdirAll(invImpErrPath, os.ModeDir)
if err != nil {
fmt.Printf("Error creating Errors directory: %s", err.Error())
panic(err)
}
err = os.MkdirAll(invImpProcPath, os.ModeDir)
if err != nil {
fmt.Printf("Error creating Processed directory: %s", err.Error())
panic(err)
}
user := {removed}
pass := {removed}
host := {removed}
port := "22"
hostKey := getHostKey(host)
config := &ssh.ClientConfig{
User: user,
Auth: []ssh.AuthMethod{
ssh.Password(pass),
},
HostKeyCallback: ssh.FixedHostKey(hostKey),
}
conn, err := ssh.Dial("tcp", host+":"+port, config)
if err != nil {
log.Fatal(err)
}
defer conn.Close()
sftpClient, err := sftp.NewClient(conn, sftp.MaxPacket(20480))
if err != nil {
log.Fatal(err)
}
defer sftpClient.Close()
files, err := sftpClient.ReadDir(sftpInvPath)
if err != nil {
log.Fatal(err)
}
for _, file := range files {
srcInvPath := path.Join(sftpInvPath, file.Name())
dstInvPath := filepath.Join(invImpRootPath, file.Name())
dstFile, err := os.Create(dstInvPath)
if err != nil {
log.Fatal(err)
}
defer dstFile.Close()
srcFile, err := sftpClient.OpenFile(srcInvPath, os.O_WRONLY|os.O_CREATE|os.O_TRUNC)
if err != nil {
log.Fatal(err)
}
bytes, err := io.Copy(dstFile, srcFile)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%d bytes copied\n", bytes)
err = dstFile.Sync()
if err != nil {
log.Fatal(err)
}
}
}
func getHostKey(host string) ssh.PublicKey {
// parse OpenSSH known_hosts file
// ssh or use ssh-keyscan to get initial key
file, err := os.Open("known_hosts")
if err != nil {
log.Fatal(err)
}
defer file.Close()
scanner := bufio.NewScanner(file)
var hostKey ssh.PublicKey
for scanner.Scan() {
fields := strings.Split(scanner.Text(), " ")
if len(fields) != 3 {
continue
}
if strings.Contains(fields[0], host) {
var err error
hostKey, _, _, _, err = ssh.ParseAuthorizedKey(scanner.Bytes())
if err != nil {
log.Fatalf("error parsing %q: %v", fields[2], err)
}
break
}
}
if hostKey == nil {
log.Fatalf("no hostkey found for %s", host)
}
return hostKey
}
Server information from WinSCP:
Thank you!
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 26 (6 by maintainers)
Hello @puellanivis and @eikenb, thank you for your help, I was not able to post more details, something was blocking me from posting.
Anyways, after some attempts, I decided to rewrite my code and finally solved my issue! I based my new code in this example here the differences from this code to my new code is that I am copying a specific file,
In order to this to work, at line 61 I am using my role path with the file name
r, err := c.Open("/dev/zero/myfile.txt")
Before line 67 I added
os.Create(destinationPath)
And I put my role path in the destination as well
w, err := os.OpenFile(destinationPath, syscall.O_WRONLY, 0600)
Hey @newdayrising,
I found a couple issues with your code that fixing should get you going. The main thing is how you open the source file…
Those flags
os.O_WRONLY|os.O_CREATE|os.O_TRUNC
are what you pass when you want to truncate and create a new file. But you are calling that for your source file. What you want is more along the lines…The client.Open() method is designed for read-only operations like this one. It uses the flag
os.O_RDONLY
in the open call.The other thing was that you don’t close the files properly. You close the dstFile using defer, which could queue up closing those files until they were all processed and the function exited. And you don’t close the source file at all. Instead you should close both the dstFile and srcFile at the bottom of the loop block, after the Sync() call.
Note that you’ll want to check your source files. The file, the one triggering the error, gets truncated (all the contents deleted) by that bad OpenFile call.