Syncthing graduation day
forum.syncthing.net
forum.syncthing.net
Yesterday I installed it again on my main PC (part of a migration from a different linux distro) and I was impressed: The notifications when adding new devices arrived with a very small delay, syncing just worked, file monitoring is now built in, the error message related to that (inotify limit) was helpful.
Syncthing came a long way and is great software, congrats for reaching 1.0!
What type of issues?
And related, sync % reflecting ignored files. For example I've ignored a bunch of transient things like node_modules so my backup server sync is permanently at 70% even though it's actually up to date with all the files I want synced.
1. An "stignore.txt" at the root of a shared directory. This gets automatically synced to every device.
2. "#include stignore.txt" in '.stignore' that I need to manually write on every new device.
After the initial setup, which wasn't that hard after all, but took more effort than doing nothing and continue on Dropbox I was amazed. My setup became better as now my phone is a real client that syncs files, not just a file browser that can "star" and cache some files, and there's more granularity on what's shared.
See: https://www.dropboxforum.com/t5/Error-messages/Dropbox-clien...
https://www.dropbox.com/help/desktop-web/system-requirements...
I'm aware of https://github.com/syncthing/syncthing-lite but the consensus seems to be it is not quite there yet. I should perhaps give it another go.
I've around 70 servers with syncthing on them and i am using it with Route53 for use as CDN.
One thing i am missing is that, i want to sync in a way that it's aware of the network topology and syncs from the nearest network first.
The downside is that all your data lives on every server.
Is there a smart way to sync only needed data using p2p for CDN application?
It creates a big peace-of-mind. Debian's version was built with iNotify for a very long time, so I never got big CPU spikes. I think iNotify is merged into master last year or so.
Then I found Resilio Sync and I've been using it for about 3 years now without any problems. Now I have a feeling that Bittorrent Inc will abandon the project after the acquisition. Syncthing is looking promising.
I use it to backup photos / videos that I capture on my phone, as well as to sync my KeePass database and Org Mode files which I edit on both my phone and laptop, often concurrently. My largest folder is 46 GB across 9,416 files.
The SyncThing Usage Data dashboard at https://data.syncthing.net/ has really interesting statistics: apparently at the 50th percentile, users are syncing more than 5,300 files and over 12.3 GB of data.
I'm using Syncthing for misc files but accidentally dropped a large git repo in it about a month ago... All the tiny files caused it to really chug.
Not that this is really Syncthing's fault.. file system operations over many files just tend to be way slower.
I sync documents and wallpapers accross my computers, I didn't see any corrupted or truncated file, it has been completely reliable from that point of view.
Also, as I said before, if the firewall of the place I connect is not extremely hostile, I can connect to it from anywhere in the world (it also has a auto-fallback to Tor). If you want to go completely dark, you can run your own tracker too. :)
My file stats are: 641 files, 85 folders, ~980 MB size.
1. That is neat.
2. Could you please say how you did this?
I might be wrong. Maybe they can compete with dropbox by being cheaper!
Their license structure is currently worthless to me. I have a license, but that is only for home use. They only offer subscription licenses for business. Subscription licenses are harder to purchase in some organizations, plus everyone that I want to offer files to needs to buy a license as well. This is a large downgrade from e.g. Dropbox, where one can just hand out download links, open file submission requests, etc.
Anyways, I'll use this opportunity to say that it has always been great to work with the guys at Syncthing. They responded quickly to every issue that we have ran into. For example, the switch from FontAwesome to ForkAwesome was done for due to a Debian issue:
...or maybe I'm making all these up from all the things I read while researching syncthing, who knows :)
Regardless of all, it's working very well and it's dependable now, and it's the point I think :D
[0]: https://snapshot.debian.org/package/?cat=s
Addenda: Actually, forgetting all little details about it is a reliable sign of an application working well and just disappears to the background IMHO.
It has worked surprisingly fantastically. I thought I would need a server to use as a "master" but syncthing actually supports merging folders automatically. It does not need a static IP address for either computer and it will opportunistically sync my laptop and desktop when they are both on the same network.
I had no idea before I tried using it that syncthing could support such a setup so well. High praise.
That means if I have several terabytes that I want to sync between devices, I'd need to have enough space in every device, is that right? (and each device would get a 1:1 copy of the folders). I read that when there are conflicts, the apps create a new "filename".sync-conflict file so you can manually resolve conflicts.
What I'd like is a secure way to backup all my files to some cloud provider (S3?) across all my devices, and have an MRU cache in a folder where I can configure the size (for instance, keep at most, say, 512mb of the files I most recently used). For things like photos, it would be better if the sync thing would get thumbnails from the cloud, so I would not need to download the whole thing just to find photos.
GDrive / GPhotos is the thing I use now. The combo has a lot of bells and whistles and I'd be fine having a simpler product where I could have more (perceived?) control on how things display, when syncing happens, etc (in lieu of other things like the nice A.I. photo search, integration with GSuite of apps, etc.). Kinda like using a file explorer on my local desktop.
Plus there are several folder types, particularly send or receive only.
See https://docs.syncthing.net/users/foldertypes.html
However,it's not a cloud backup service. If you need online storage then best option is to get an online service.
I suppose the people involved don't use the platform, and I realize iOS is more annoying to develop for than Android, but it's a pretty major downside and it's tempting me to use a cloud provider instead. I'd much rather just pay $15-$20 or something for an iOS app.
Maybe now with the 1.0 release it will be stable enough that a third party can step up and make an iOS app?
I really dislike that the Syncthing website appears to have mastered modern "we won't tell you what it is" marketing speak. Did I miss something and this is now a game with scoring for speaking around topics?
"Syncthing replaces proprietary sync and cloud services with something open, trustworthy and decentralized. Your data is your data alone and you deserve to choose where it is stored, if it is shared with some third party and how it's transmitted over the Internet."
Oooooooh, new shiny with buzzwords! What's it do? Is it a replacement for Chrome's cross-system syncing? Firefox Sync (which I think can already be self-hosted actually)? Password managers? Maybe hosted Exchange? Wait, it's not?
Oh, wait, it's a multi-system file syncing package. Well why didn't you say so while you were telling me I deserved to have control and blah blah blah. Cripes, it even says "a large number of features have already been implemented" then provides a list that implies that running on Windows, OSX, Linux, the BSDs and Solaris is one of those features.
There are several places where what it does is implied, but to actually get to a statement of "For this guide let’s assume you have two machines between which you want to synchronise files." I have to go to the bottom of the site, click Docs, then either click on Introduction and Getting Started (in the documentation structure) or click on the "getting started guide" link within the first paragraph of the Introduction.
grmbl get off my lawn
Edit: I should note, I'm not talking about the 1.0 announcement on their forums - that's for folks already actively involved with it. I'm talking about the very spare https://syncthing.net
The first two sentences of the readme are:
> Syncthing is a continuous file synchronization program. It synchronizes files between two or more computers.
That copy should absolutely be somewhere on the main website.
Yes. Specifically, it should be in a large font, roughly centered, at the top of the page. You should load the page, and immediately read those words.
I used syncthing to implement part of a project at work and it does a pretty good job, it deserves a website that actually tells you what the hell it is.
Also: top-left logos should link to the root (syncthing.net) not to a sub-root (forums.syncthing.net). When I land on your forum site and I want to know what syncthing is, I expect to find the answer by clicking the top-left logo in the header.
I cancelled my YouTube Premium, and started seeing ads on YouTube for Android; and one of the first few was an ad for some app called Killi or something. It jumps straight to how to find it on the app store and google play, and then explains how to set it up, all the while you're wondering what on earth it is.
IIRC, Nexium (antacid of some sort) when it was first introduced had ads everywhere full of people talking about the "Little purple pill" and how much it had changed their lives. There was nothing in the ads that indicated anything about what it did, just "ask your doctor if Nexium is right for you."
Had I been a bit more connected at the time I think a real push to start talking about "The purple pill that cured my X" would have been amusing, with X being something like uncontrollable flatulence, incontinence, explosive diarrhea, etc. Unfortunately, that was back in the days when social media was LiveJournal and Usenet was already well on its way into decline.
Apart from that I don't recall any major problems - which is very positive for anything related to file synchronization.
Over the past year or so, file watching (via inotify/etc) has been built into syncthing itself. I enabled watching on all my computers and syncthing's CPU usage has gone down considerably as a result, while user happiness has increased; I spend less time banging on syncthing's pipes and more time doing stuff with the files it synchronizes.
However, the case insensitive file system in Windows makes it annoying to use with *nix systems.
I've also been waiting for that one feature :/
While it would be convenient to have configurable encryption on endpoints, syncthing is composable, and thus this issue can be worked around. If you have a platform that doesn't allow you to compose tools easily like Android, that's limited technology.
https://en.wikipedia.org/wiki/EncFS
EncFS is not perfect in that if your adversary has the opportunity to watch your encrypted files updating on the regular they can (allegedly) break the encryption, but they can't do it from a single snapshot.
It's a decent protection against an airport sniffing your phone traffic while you're on the public wifi, it's not going to stop the NSA if they decide you're a person of interest. In other words, if this is a concern for you EncFS is an acceptable baseline protection.
What I'd like is integration with on of Perkeep or Syncthing from my freeNAS box to my local devices.
But if two nodes that have added each other's keys cannot reach each other directly (e.g. are in separate private networks/behind firewalls/etc) but both can reach an external relay server, it may act as a dumb pipe bouncing encrypted traffic between them.
https://docs.syncthing.net/users/relaying.html
There is a public pool of such relay servers (https://relays.syncthing.net/), and by default syncthing will reach out and connect/announce its itself there so other nodes looking for it could contact it via a relay.
You can host your own relay server and configure your clients to use it exclusively: https://docs.syncthing.net/users/strelaysrv.html#client-conf... or if you only want syncing when your nodes are on the same local network you should be able to just not configure any relays at all.
Syncthing is great - a folder that syncs, I have a VPC in the cloud, so for the data I really care about, it's mirrored offsite more or less in real time in case my house burns down.
I like a lot the announcement: it's very easy to stay perpetually in beta to escape commitment. Their reasons for graduating are sensible and I wish more developers of software on version 0.x since a long time would replicate it.
However, I'm surprised too see that by default the whole interface is available on 127.0.0.1:8384. It is possible to set a password but it's a bit hidden in the settings and the documentation does not insist on it.
This makes me wonder, how bad is it for a malicious program running locally to have access to this API?
Pretty rare someone manages every outbound connection but doesn't configure a password on the tool handling syncing all of their data across the network. Still, would be better if it prompted for a password the first time a user visited the page to set up the client.
What exactly is your threat model?
On a shared computer where you perceive your peers as a threat, adding a password to syncthing web UI is probably a minor entry in a long list of things you have to do to secure your processes and data, and to ensure the kernel and programs you're running are trustable (in practice, you can't verify that with 100% confidence; they can be backdoored, out-of-date with known security flaws, etc etc).
Ensuring security in such a malicious (where your co-workers/peers are apparently a nasty bunch) shared environment is so hard that to begin with that I personally wouldn't put any sensitive data there at all.
Having an API that can reconfigure syncing being available without authentication for anything running on the machine feels backwards to me. More so for a project that focuses on security.
If anyone from the project reads this, may I recommend either encouraging users to set a password or mentioning the implications on the page about Security Principles[0]?
If your threat model is you can't trust the programs running on your computer, then maybe you should switch to a different OS, for example Debian, and stick to the package manager.
Pity that I don't know of any browser that readily supports HTTP over a unix socket (nginx can proxy_pass it, but then you're back to a TCP listener).
But a multi-user install could reasonably lock things down in that way, though the defaults definitely favor "works out of the box on a single-user machine".
I use it for syncing my phone (everything!) to a pc, and keepin several ps's up to date with each other. These days, it just works. Works so well, in fact, that I have begun experimenting with up-sync to a production server. So far, things are looking real good.
But you can combine it with an encryption layer yourself by using any of the projects listed here: https://nuetzlich.net/gocryptfs/comparison/
That's pretty much the only feature I miss with Syncthing.
> I really love to use syncthing as a anti-bit-rot check system (i already do^^).
Bit-rot is something I want to (manually) protect against, without setting up a NAS running ZFS. How would you use Syncthing?
By the way if you're on linux btrfs has self healing capabilities (and any of the cheap 2/4 bay 64bit synology nas can use the btrfs self healing and coupled with raid protect your data from rot). Not a very big expense for a lot of safety.
Think a NAS will be a future investment.
As the other commenter said you can also built it yourself but frankly configuring Samba and its permissions and updating your mdadm config when changing a drive and all the other things gets boring and borderline dangerous really quick, just know that if you buy a pre-made NAS what you're paying is not really the hardware (you can get better for cheaper if homemade), it's the software.
I like DSM (synology's software) because it doesn't get in your way, it lets anyone use easily yet still offers the advanced options. If you've never had a NAS or don't want to spend hours in config files every now and then, I recommend it.
(as usual the question for devices like this is not so much "how easy is it to set up" but "how easy it to recover if say a drive died and you need to repair your array")
https://github.com/stephenh/mirror
Disclaimer it's my basically-well-functioning hobby project. And obviously wouldn't recommend it for doing actual deployments with it.
Syncthing does many-to-many remote servers synchronization (but you can also do 1:1 or 1:M). It leaves reconciliation of conflicts to the users. If you're lucky, a copy of the file in question is created that has as "sync.conflict" suffix. It ignores symlinks (or copies them as is). It is actively maintained & has a much wider user community. It works on windows ... as well as android etc. It's written in Go. Synchting can also be run in the background to watch folders and keep everything in sync as files change. (This actually is the main use case.)
You can use syncthing for almost everything unison does when you want to sync servers, workstations, phones etc. You cannot use syncthing to keep a memory stick in sync. IMHO unison is still better at syncing two directories with frequent merge conflicts. It's a pity it got trapped in the ocaml hole. I personally replaced unison with syncthing for most use cases.
Also good point about OCaml and not having iOS/Android support.