Any chance this work can be upstreamed into mainline SSH? I'd love to have better performance for SSH, but I'm probably not going to install and remember to use this just for the few times it would be relevant.
The only downside I have found using lftp is that given there is no corresponding daemon for rsync on the destination then directory enumeration can be slow if there are a lot of nested sub-directories. Oh and the syntax is a little odd for me anyway. I always have to look at my existing scripts when setting up new automation.
Demo to play with, download only. Try different values. This will be faster on your servers, especially anything within the data-center.
ssh mirror@mirror.newsdump.org # do this once to accept key as ssh-keyscan will choke on my big banner
mkdir -p /dev/shm/test && cd /dev/shm/test
lftp -u mirror, -e "mirror --parallel=4 --use-pget=8 --no-perms --verbose /pub/big_file_test/ /dev/shm/test;bye" sftp://mirror.newsdump.org
I often find myself needing to move a single large file rather than many smaller ones but TCP overhead and latency will always keep speeds down.
The design of sftp is such that it cannot exploit "TCP sliding windows" to maximize bandwidth on high-latency connections. Thus, the migration from scp to sftp has involved a performance loss, which is well-known.
https://daniel.haxx.se/blog/2010/12/08/making-sftp-transfers...
The rsync question is not a workable answer, as OpenBSD has reimplemented the rsync protocol in a new codebase:
An attempt to combine the BSD-licensed rsync with OpenSSH would likely see it stripped out of GPL-focused implementations, where the original GPL release has long standing.
It would be more straightforward to design a new SFTP implementation that implements sliding windows.
I understand (but have not measured) that forcibly reverting to the original scp protocol will also raise performance in high-latency conditions. This does introduce an attack surface, should not be the default transfer tool, and demands thoughtful care.
Rsync is great when moving files between trusted systems that one has a shell on but the downside is that rsync can not split up files into multiple streams so there is still a limit based on source+dest buffer+rtt and one has to provide people a shell or add some clunky way to prevent a shell by using wrappers unless using native rsync port 873 which is not encrypted. Some people break up jobs on the client side and spawn multiple rsync jobs in the background. It appears that openrsync is still very much work in progress.
SCP is being or has been deprecated but the binaries still exist for now. People will have to hold onto old binaries and should probably static compile them as the linked libraries will likely go away at some point.
If you want to use the historic scp server instead, a command line option is provided to allow this:
"In case of incompatibility, the scp(1) client may be instructed to use the legacy scp/rcp using the -O flag."
https://www.openssh.org/releasenotes.html
The old scp behavior hasn't been removed, but you need to specifically request it. It is not the default.
It would seem to me that an alternate invocation for file transfer could be tested against sftp in high latency situations:
ssh yourhost 'cat somefile' > somefile
ssh yourhost 'tar cf - yourdir' | tar xpf -
If your BDP is less than 2MB you still might get some benefit if you are CPU limited and use the parallel ciphers. However, the fastest cipher is AES-GCM and we haven't parallelized that as of yet (that's next on the list).
Moving a terabyte database in an upgrade, I have connected three ports direct (no switch), then used xargs to keep all three connections busy with transferring the 2gb data files. I can get the transfer done in under an hour this way.
I don't currently have a performance need for an encrypted transfer, but one may arise.
I'm assuming that would have different limits than you outlined above, although I don't think they do multi core.
I thought openrsync existed solely because of rpki. Even OpenBSD devs recommend using the real version from ports.
Better question is 'does it have any actual improvements in day-to-day operations'? Because it seems like it mostly changes up some ciphering which is already very fast.
- IF you don't trust it
- AND you want to use it
=> run it on a private network
You don't have to trust it for security to use it. Putting services on secure networks when the public doesn't need access is standard practice.
But I don’t miss having those limitations.
On what basis are making that claim? Because AFAICT, concern about it being less secure is entirely reasonable and is one of the big caveats to it.
You may as well just use tailscale ssh in that case. It already disables ssh encryption because your connection is encrypted with WireGuard anyway.
I've been developing it for 20+ years and if you have any specific questions I'd be happy to answer them.
OpenBSD's filesystem operations are slow not because of UFS2, but because they simply haven't been optimized up-and-down the stack the way Ext4 has been Linux or UFS2 on FreeBSD. And of course, OpenBSD's implementation doesn't have a journal (both UFS and Ext had journaling bolted late in life) so filesystem checks (triggered on an unclean shutdown or after N boots) can take a long time, which often cause people to think their system has frozen or didn't come up. That user interface problem notwithstanding, UFS2 is extremely robust. OpenBSD is very conservative about optimizations, especially when they increase code complexity, and particularly for subsystems where the project doesn't have time available to give it the necessary attention.
I believe that these were recently removed. Perhaps they don't play well with SMP.
[1] https://github.com/freebsd/freebsd-ports/blob/main/security/...
[2] https://github.com/NixOS/nixpkgs/blob/d85ef06512a3afbd6f9082...