Skip to content

use the copy-data extension in File.WriteTo between two Files - #709

Open
prudhvi57 wants to merge 3 commits into
pkg:masterfrom
prudhvi57:copy-data
Open

prudhvi57 wants to merge 3 commits into
pkg:masterfrom
prudhvi57:copy-data

Conversation

@prudhvi57

Copy link
Copy Markdown

Closes #707.

When File.WriteTo is given another *File from the same Client and the server advertises the OpenSSH copy-data extension (OpenSSH 9.0+), this asks the server to copy the data directly instead of reading it to the client and writing it back. So io.Copy(dst, src) between two remote files on the same server no longer moves the data over the network twice.

  • The copy starts at the source's current offset and writes at the destination's current offset, so callers control ranges with Seek. Both offsets advance by the bytes copied.
  • The exact remaining length is sent (from fstat/stat) rather than 0, so WriteTo can return an accurate byte count.
  • Falls back to the existing read/write path when the server doesn't advertise the extension, the source isn't a regular file, or the files are the same or from different clients.
  • No new public API. I left WriteToLimited out to keep this small and can follow up with it.

Tested with go test -race -integration and -integration -testserver (OpenSSH 9.6 sftp-server for the copy-data path, the Go server for the fallback).

This is groundwork for server-side copy in rclone's sftp backend (rclone/rclone#6624).
I've reviewed and tested the change and I'm happy to make any changes you need.

When File.WriteTo is given another *File from the same Client, and the
server advertises the "copy-data" extension (OpenSSH 9.0 and later),
ask the server to copy the data directly from the source's current
offset to the destination's current offset, instead of reading it to
the client and writing it back. Both offsets advance by the bytes
copied, so io.Copy(dst, src) and Seek work as before.

If the server lacks the extension, the source is not a regular file,
or the files are the same or from different clients, WriteTo falls
back to the existing read and write path.

@puellanivis puellanivis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😟 hm… not actually looking good now that I see the PR…

This extension is just a bad idea for fitting into Golang in general. Too many assumptions being made that we cannot rely upon.

Comment thread client.go Outdated
Comment on lines +1539 to +1544
var fileStat *FileStat
if f.c.useFstat {
fileStat, err = f.c.fstat(f.handle)
} else {
fileStat, err = f.c.stat(f.path)
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fun fact: we usually don’t want to be doing this anymore. There are a lot of issues with trying to find the file length without certain servers deleting the file as completely read.

Of course, these servers are unlikely to support the copy-data extension in the first place, right?

Comment thread client.go Outdated
Comment on lines +1532 to +1533
dst.mu.Lock()
defer dst.mu.Unlock()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We’re atempting here to grab a new lock while holding another lock. This is always an great opportunity for creating a deadlock. Worse, as designed, there’s no way to keep clients from trying to lock the files in the opposite order.

As such, we should probably record this file’s handle to a local variable, unlock, grab the dst lock, record its handle to a local variable, then unlock. Now, we’re not holding either lock. So, we can now lock them in the sorted order of their handles. (Of course, if either is empty, then we would return os.ErrClosed rather than attemptiung to relock at all.)

Once we’ve released both locks and then relocked in a deterministic order, then need to again recheck to make sure each is not closed handle == "" while holding both locks.

Comment thread client.go Outdated
return 0, false, nil
}

// copy-data treats a length of zero as "until EOF", which would leave us not knowing how much was copied.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fun fact: even if we know the file length, and request copying that much data, we still won’t know how many bytes were actually copied.

As an example, another process could trunc the file in between when we did our stat and when the copy-data request was sent/finishes.

Reading the openssh code, if it hits an EOF before the full amount of requested length has been read, then it should still return an SSH_FX_OK. But we would then erroneously report that we copied the whole set of data.

😐 This is actually a failure of the RFC design. There’s simply no way to get the information of how many bytes were actually copied back to the client. … this might make this a non-starter for this API, unless we implement a WriteToLimited(dst *File, n int64) error receiver method, that one will note: does not report back how many bytes were read/written. It should have been defined an SSH_FXP_EXTENDED_REPLY packet that reports back a success with how many bytes were copied.

Per review: do not guess the length with a stat, since some servers
remove a file once it has been read, and copy-data cannot report how
many bytes were actually copied.

Instead add WriteToLimited(dst *File, n int64) error. The caller
chooses the length and offsets (via Seek), zero is never sent (it would
mean "until EOF"), and the two file locks are taken in a fixed order
by handle, rechecking for a close, so opposite-direction calls cannot
deadlock. WriteTo is unchanged.

@puellanivis puellanivis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Go ahead and push any updates from this round you would like to propose. But I’m going to put further reviews on ice for a bit, while I consider how we might want to shove this square peg into a round hole.

The design of this is just… so incompatible with what we already have.

Comment thread client.go
Comment on lines +1538 to +1541
if n == 0 {
// A length of zero means "until EOF" to copy-data, so never send it.
return nil
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sending a ReadDataLength == 0 means copy until EOF, but that’s not the semantics we have for n here.

Here, it means to copy zero bytes, and as such, is immediately successful as a no-op.

Comment thread client.go Outdated

first, second := f, dst
if dstHandle < srcHandle {
first, second = dst, f

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

prefer explicitly swapping the values:

Suggested change
first, second = dst, f
first, second = second, first

Comment thread client.go Outdated
second.mu.Lock()
defer second.mu.Unlock()

// A file may have been closed while we were not holding its lock.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We’re not testing if it’s closed (that is f.handle == ""), we’re testing a wider sanity check: "making sure that the handle hasn’t changed (for example, when closed)".

Either we should test for being closed (against ""), or we should provide a proper error message of the actual problem: the file handler changed underneath us.

Comment thread client.go
Comment on lines +1599 to +1600
f.offset += n
dst.offset += n

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 I’m not sure we should be doing this. If we knew the length read/written, then this would be justifiable. But we don’t actually know how much was written. So, we’re just making a guess here. This is the whole reason we cannot return written int64!

Comment thread client_integration_test.go Outdated
require.NoError(t, err)

var wg sync.WaitGroup
for i := 0; i < 50; i++ {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
for i := 0; i < 50; i++ {
for i := range 50 {

require.NoError(t, err)
defer dst.Close()

err = src.WriteToLimited(dst, 1)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to be making the assumption that our server implementation will never implement the copy-data extension.

Weirdly, because of the bad design of the extension, it would actually be kind of trivial for us to implement it, the same as the OpenSSH sftp-server…

And you can bet contributors to this package will be interested in providing an implementation soon-ish. We have plenty of not just users of our client in this package, but users of our servers as well… (no matter how ill-advised I keep insisting that is).

Comment thread client.go Outdated
// the server may stop at EOF and still report success.
// Callers that need to be sure should check the size of dst afterwards.
//
// f and dst must be different files opened by the same Client.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

f and dst do not have to be separate files, they have to be separate handles (in our use case).

As mentioned in the standard:

The server SHOULD allow 'read-from-handle' and 'write-to-handle' to be the same handle as long as the range of data is not overlapping.

In our case, we need separate offsets, even if it’s the same file… although, it seems, that sftp-server does not allow this: https://github.com/openssh/openssh-portable/blob/fa4b2a82be7df9768272cfabc2d78e4a12c2b3c0/sftp-server.c#L1651-L1655

However, we shouldn’t be making that decision. The standard says servers should allow even the two handles to be the same… On our side, the only reason we don’t want f == dst is because the two files will be sharing the same offset… oh, and they also share the same mutex… so, that would deadlock.

It should say something like f and dst must be different [File] structs, opened by the same [Client].

Comment thread client.go Outdated
Comment on lines +1526 to +1527
// If the server does not advertise "copy-data", WriteToLimited sends nothing
// and returns a *StatusError with the code SSH_FX_OP_UNSUPPORTED.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’ve had things misunderstand this statement.

  1. This should be a separate paragraph. It doesn’t relate to the previous “must be different Files”.
  2. “It returns an error if the server did not advertise "copy-data" support, and no request will be sent.”

Comment thread client.go Outdated
// On success, the offsets of both f and dst are advanced by n.
//
// The server only reports success or failure, not how many bytes it copied.
// If f is shorter than expected, for example because it is truncated while the copy runs,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Parentheticals should be parenthesized.

Suggested change
// If f is shorter than expected, for example because it is truncated while the copy runs,
// If f is shorter than expected (for example: because it is truncated while the copy runs),

Comment thread client.go
// f and dst must be different files opened by the same Client.
// If the server does not advertise "copy-data", WriteToLimited sends nothing
// and returns a *StatusError with the code SSH_FX_OP_UNSUPPORTED.
func (f *File) WriteToLimited(dst *File, n int64) error {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 with all the feedback from this round, I’m starting to think we just implement this as func (f *File) CopyData(dst *File, at, n int64) error…

Then we can specially consider where src == dst and avoid the deadlock from that, while keeping potential support to copy from the same handle to the same handle—but elsewhere. And leave it up to the server to either accept or refuse the operation.

…and docs

- handle f == dst by locking once and leaving the decision to the server
- report a changed handle separately from a closed one after locking
- docs: File structs rather than files, unsupported case in its own paragraph
- tests: same-file case, range-over-int loop, note on the unsupported test
@prudhvi57

Copy link
Copy Markdown
Author

Thanks for taking another look, and no problem putting this on ice. I pushed the updates from this round:

  • n == 0 is now a no-op and never sends a request (copy-data reads 0 as "until EOF")
  • the lock swap is explicit now (first, second = second, first)
  • after locking, a closed handle returns os.ErrClosed and a changed handle returns its own error
  • same File as src and dst is handled by locking once; whether that is allowed is left to the server
  • docs now say File structs instead of files, the unsupported case is its own paragraph, and the truncation note is in parentheses
  • tests use for range 50, there is a same-file test, and the unsupported test notes why it skips when the server has copy-data

Two things I was not sure about:

  1. Offsets. Since the reply has no byte count, advancing f.offset by n can be wrong after a short copy. Would you prefer I leave both offsets alone and let the caller Seek?
  2. Your CopyData(dst *File, at, n int64) idea. Is at the read offset in f, with the write going to dst's current offset? If you like that shape better I'm happy to switch to it.

If it still doesn't fit the package, that's completely fine and I can close this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Proposal: client support for the OpenSSH "copy-data" extension

2 participants