Hacker News Reader

Comment by rocky1138 | Hacker News Reader

An active FOSS alternative to this is Syncthing. https://github.com/syncthing

I really like the philosophy of the project and the guys running it, but my recent experience trying to use it as a dropbox replacement has been bad: machines that stayed "syncing..." forever, deleted files that reappeared after a while, renamed folders resulting in duplicated folders with both the old and the new name...

I think it's just that the software is not yet mature and production-ready, and perhaps I was throwing too much data at it at once. Seeing the project progress, I'm confident that when it becomes more stable it will be a very nice piece of software, but it's not there yet.

Completely agree. Compare to BitTorrent Sync, which is proprietary but works perfectly out of the box. [1]

[1] Okay, this is true for BitTorrent Sync versions < 1.4 -- more recent versions have been less reliable, particularly with regards to cross-platform syncing.

I've actually found the opposite. One of my coworkers and I are disappointed by a lot of syncing software out there, and began using SyncThing.

We haven't experienced any bugs so long as we both keep our versions in sync. Once one has a later version than the other, all bets are off. That's a problem that's tricky to solve correctly though.

Aside from that, we were disappointed by the constant needs to restart SyncThing when saving configuration changes, some lack of UX features (e.g. being able to move where your sync share is stored -- I made an issue on Github and was told to edit the config file manually and restart SyncThing -- I asked if a pull request would be accepted if I added the feature myself, and got a vague response.)

Overall though, SyncThing has worked excellently for us.

IME they are helpful at diagnosis and troubleshooting, as long as the user is willing to put similar effort to providing details and debugging.

Not sure what is causing your bad experience. Are you using an up-to-date version? I have been deploying it across 4 different computers and it works like a charm.

Yes, I was using the last version in all machines (some mint, some arch).

I was trying to sync seven machines in three different physical locations, some with slow internet connectivity; only one machine in each location had its syncthing ports open to the internet through NAT. I'm not sure it's a supported use case: syncing in the same LAN generally worked fine, but changes often didn't propagate correctly to other locations; for instance a folder name change sometimes was not propagated to the remote locations, and after some time the machine in which the rename had happened detected that the rest of the machines in the syncthing group had a folder which didn't exist locally (the folder with the old name), and downloaded it from its peers. The result was that the folder was duplicated everywhere, with both the old and the new name.

The same happened sometimes with deletions: sometimes they didn't propagate over LAN boundaries, and this resulted in the deleted files reappearing after some time.

A machine staying "syncing..." forever also happened sometimes, requiring a manual restart of the service. I couldn't reliably reproduce it though.

I like the project and I'm sure if I reported the issues in github they'd have given support. I wish I had the time for properly reporting and investigating, but I really needed a syncing solution I could trust for a small business and I needed it ASAP, so I settled for owncloud (which for the moment has not given me any surprises). I will try to find some time to reproduce the bugs in my personal machines and report them, but I really didn't have the time when I found them.

hey,

maybe the active spideroak project with nimbus is something for you

https://github.com/SpiderOak https://github.com/SpiderOak/nimbus.io

I use Syncthing for most of my cross-device sync'ing, but I haven't yet found a good FOSS solution for sync'ing locally (local folder to external hdd, mp3 player, another local folder, etc).

There were a few posts on the Syncthing forums requesting it as a feature but they didn't really get anywhere:

https://forum.syncthing.net/t/possibility-to-sync-local-fold...

In the mean time, I'm using GoodSync for sync'ing locally:

http://www.goodsync.com/download

It works quite well for that purpose, but since it's proprietary I'm not comfortable with blindly trusting it to sync my important files over the internet.

Still waiting for that perfect open-source p2p sync app that can also do local sync'ing.

What about git-annex[1], duplicity[2] or bup[3]?

I personally use duplicity to make versioned backups of my homedir (on OSX, debian and Arch machines) and keep versioned dropbox-like backups via owncloud. My homedir backups are GPG encrypted automatically.

1: https://git-annex.branchable.com/

2: http://duplicity.nongnu.org/

3: https://github.com/bup/bup

duplicity with gpg is good idea for backup and security, but probably difficult to deal with large amount of binary data(pictures,family videos) up to, say 2TB.

No? If only does deltas, presumably you aren't changing all the files very often. My largest duplicity backup is, ~800GB. With 24h of hourly snapshots, 1 month of daily snapshots, and 1 year of monthly snapshots, the actual backup currently takes ~5TB. It only increases in ~1-10GB/day

Unison File Synchronizer is made for you!

http://www.cis.upenn.edu/~bcpierce/unison/

Worth noting that it's not maintained anymore http://www.cis.upenn.edu/~bcpierce/unison/status.html

Actually your link says just that nobody is paid to maintain it.

"It will continue to be maintained and supported for the foreseeable future, and we occasionally release new versions with bug fixes, improvements (sometimes large ones), and contributed patches."

> I use Syncthing for most of my cross-device sync'ing, but I haven't yet found a good FOSS solution for sync'ing locally (local folder to external hdd, mp3 player, another local folder, etc).

Seriously ? How about Rsync?

I love Syncthing. I used to use Dropbox and then ownCloud on a VPS, but so far Syncthing has been the perfect replacement to regain complete control of my data.

Also: http://sparkleshare.org/ but it seems unmaintained.

How does it handle conflicts?

http://docs.syncthing.net/users/faq.html#what-if-there-is-a-...

  > When a file has been modified on two devices 
  > simultaneously, one of the files will be renamed to
  > <filename>.sync- conflict-<date>-<time>.<ext>.

I haven't had too many. About the same as Dropbox, I think. From memory, I believe it appends -conflicted-BLABLA's-copy or some such to the filename.

Conveniently, it has a built-in backup system you can enable that keeps the last 'n' copies of files. I've got mine set to 5 so even if something goes horribly wrong, I can fetch from the backup.

I would still worry without offline backups. Malware or a malicious actor could delete the built-in backups from _all_ sync'ed devices by modifying each file 6 times.

any good?

Yes! But it's decentralized. An alternative to Ubuntu One would be ownCloud (https://owncloud.org/).

Another alternative is Seafile https://www.seafile.com/en/download/

And yet another alternative, nodejs based:

http://cozy.io

Centralized is just a degenerate case of decentralized.

Stand up a dedicated server running syncthing and poof!, you have a centralized syncthing.

synch Thing is really a good option. It has a good community that is actively supporting the product and the features include only syncing certain files is great. I was a former Bit Torret Sync user but switch due to its open source code.

Yes. It's very good. I've been using it for about 4 months now between a Raspberry Pi (Raspbian Linux), an old laptop (Ubuntu Linux), my main computer (Windows 8.1), and my girlfriend's computer (Windows 8.1).

It's usable if you have a small amount of files. For anything serious (say 100k files or 500GB of data) is blows through CPU and RAM to the point of being unusable. I've been actively testing several and there is currently no actually good open-source solution in this space.

I've got about 150GB (but well over 100k files) and it only really lags on the Raspberry Pi. One thing I could suggest is that you change how often it rescans your directories for changes (I set mine to 6000 seconds versus the default 60). That might make a difference.

I've explored this pretty extensively and it's currently just the way it is. It happens from first sync so the rescan interval makes no difference. Your collection probably requires 300-400MB of RAM to sync so it should be on the cusp of failing on a raspberry pi.

Here's the bug report that was closed without any fix:

https://github.com/syncthing/syncthing/issues/468

And user reports of this show up pretty often:

https://github.com/syncthing/syncthing/issues/1841

https://forum.syncthing.net/t/arm-linux-cpu-usage-99-for-man...

https://github.com/syncthing/syncthing/issues/1841

A sync tool that uses on the order 2MB of RAM per GB of files or 4kB per file isn't really usable. A cheap NAS these days can easily have 6TB of RAID6 storage but it won't have 12GB of RAM. Clearly something in syncthing is keeping way too much stuff pinned in memory and the developers so far haven't focused on that. There's no apparent need for this memory usage either as there's already an on-disk database with the state of the sync folder.

That's really interesting, thanks.

It uses inotify, so "scanning" for changes more frequently should have only a minimal impact on performance. Of course if you have a lot of actively changing files, you don't want it spending a lot of time transferring files that are still in the process of being modified, so reducing the frequency might help.

I've never noticed performance issues with syncthing, excepting my old original Nexus 7 tablet which does seem to suffer a bit due to its poor hardware. I keep my photos volume (~250GB) synchronized between three desktop systems and it's never presented a problem with those.

Given that the grandparent mentions issues with larger volumes, there may be some sort of "tipping point" beyond which performance degrades, and I just have not encountered it.

Regardless I'd encourage people to give Syncthing a try. The project is under very active development and the issue the grandparent mentions may be resovled by now, and of course that issue might be environment specific and not generally applicable.

> It uses inotify

Actually it doesn't. There are some add-ons to do it but the base tool does periodic full scans.

> I've never noticed performance issues with syncthing, excepting my old original Nexus 7 tablet which does seem to suffer a bit due to its poor hardware. I keep my photos volume (~250GB) synchronized between three desktop systems and it's never presented a problem with those.

250GB between desktops with multiple GB of RAM probably works. It will soak up plenty of resources though.

>Regardless I'd encourage people to give Syncthing a try. The project is under very active development and the issue the grandparent mentions may be resovled by now, and of course that issue might be environment specific and not generally applicable.

The issue is pervasive and apparently quite deeply rooted in the way syncthing is built as it's indeed under very active development but this hasn't improved.

Thank you for the correction re: inotify, I did not mean to mislead people. Sadly I can no longer edit the incorrect information in my original comment.

I second this. Tried to use it to sync a picture directory between my desktop and NAS device and after the original sync, it never managed to complete any further attempts. It's great for smaller directories, it's horrible at lots and lots of small to medium sized files.

Yes. It's easy to install and runs on Android. It's quite fast in detecting differences and then syncing 2 nodes. It's meant to permanently run in the background though. It cannot be not easyly automated to, eg, sync 2 nodes and then shut down. In case if concurrent edits, you end up with multiple versions of the file on disk. You have to somehow detect those conflicts (find) and merge the files yourself. It doesn't help you much with that.

Reply on news.ycombinator.com