Gitlab Project Hit by Spambot
gitlab.com
gitlab.com
We have all sorts of protection: rate limiting, crippling non-paid accounts, detecting if a trial signup comes from a VPN/Tor (+5 to "suspicious" score!) etc... But they still manage to find ways. It's a never-ending battle. And the saddest part is - all this hard work is completely invisible to our existing paying customers :(
If your app has an email-sending module of some sort it WILL be abused. Even a trivial "reset password" form is a target.
My support goes to the GitLab team. Good luck and no hard feelings.
[1] https://www.jitbit.com/news/5354-spammer-attack-post-mortem/
here's the trade-off. Make you decision.
India, Midde East, Turkey, Russia, China, South East Asia (except Singapore), All of Eastern Europe, Africa and island nations.
It immediately reduced the amount of fraud and spam. Sure you might miss out on the <1% of your revenues but its a tradeoff I'm okay with.
or is this another one of your alt nick you periodically sign in from?
Having worked in the South African tech scene and for a UK company, every spam/dos detection I've seen has been from everywhere you mention except Africa and island nations.
Like in this case (GitLab) - create a fake "issue" with spammy links in it and then "notify" thousands of users.
In our case - we are a helpdesk ticketing app, so bad actors sign up for a trial, then add "users" to their account and then, say, create "tickets" on users' behalf (customizing the email template by inserting spammy links).
hCaptcha has bad UX, their dataset is way harder to decipher and commonly takes me three or four tries to get around on Cloudflare, if it doesn't keep failing with a server error which I've had happen on multiple, distinct occasions.
At least for Google, I was done in one turn when lucky, if I had no other choice (e.g. banking).
At least hCaptcha has a decent chance of letting me through if I answer correctly.
We are still in the early days of the internet. It is a frontier territory where crime is rampant.
I don't want the government prosecuting people for speech, even mass, automated speech. There are much better solutions than resorting to violence.
Protocols and system designs which make abuse impractical and thus prevent spam before it starts are surely desirable, although they often come with their own tradeoffs (user friction, privacy, false positives, maintenance costs, etc.). If spammers can be deterred by improving along other axes all the better. But the general problem of spam is that computing resources allow for massive amplification of malicious actions, and that's never going away — it will be an arms race between spammers and their victims forever.
As someone who battles spam on the small business front lines, I hate it. It fills me with fear and a sense of loss-of-control, that spammers may be commandeering our systems at any moment, possibly causing us to lose money, time, and reputation, cutting us off from vital services.
It takes hard work for me to overcome that and to recognize the humanity of spammers. Even now, I'm wondering, "Is sneak a spammer?" Rationally I don't believe it, but that's the kind of resentment that builds when people are left unprotected.
I hate spam; I just hate violence more.
I don't consider solutions to problems that involve violence to be solutions. Violence is, after all, the last refuge of the incompetent.
GitLabSec team-member here, ^ this 100%.
Our Trust and Safety team works really hard to keep up with spam. For those interested in how they do so, you can read more here[1]. We are also working on making our spam detection engine more effective at mitigating high-volume spam incidents such as this.
For current, working product improvements to detect and mitigate spam you and others can check out MRs labeled "spam fighting"[2]. We've had quite a few internal conversations about strategies and best-practices. For more information on how to contact our Trust & Safety Team, including tips on how to deal with abuse in your own instance or steps to suggest spam/abuse related features, see our handbook[3]
[1] https://about.gitlab.com/blog/2020/10/29/how-we-work-to-dete...
[2] https://gitlab.com/gitlab-org/gitlab/-/merge_requests?scope=...
[3] https://about.gitlab.com/handbook/engineering/security/secur....
That would require the project's moderator to clear the newly posted issues (whatever is possibly a spamming route there) before they propagate. As this is an additional friction point, some bad actors may be intercepted there (say, trying to clear a million issues at once or other equally ridiculous actions).
Of course, by itself it won't stop someone from clearing a single issue to a million of "users". But may still help in id'ing such actors.
Also, you could try to detect headless browsers like Headless Chrome.
If you want to appear to be a legitimate comcast, charter, centurylink, frontier end user on a last mile broadband connection in a house somewhere in the US48 states, that's a standard service that fraudulent organizations make use of now.
Usually using software that people have been tricked into installing on their computers.
We're actively working on mitigating this spam attack: https://twitter.com/gitlabstatus/status/1358777539624198145
Looks like the immediate issue was resolved a few minutes ago.
Gitlab recently added a band-aid of a feature to require admin approval for new sign-ups, which comes with its own obvious problems. But even the UI for that is clunky-- you have to click a button to reveal some options, find the option to delete the spammer, type (read: paste) the users handle as a failsafe, after which Gitlab takes you back to the main user view requiring you to click again on the unverified users tab.
Last time I talked about this problem someone from Gitlab claimed they were working on a feature to send an email with a simple link to click to approve/deny a user. That feature still has not shipped.
They did fix an unrelated catastrophic bug of keeping around artifacts on each branch even if you set an expiry. On a gitlab instance with even moderate development rate and moderate-sized artifacts this fills up the hard-disk with artifacts that the admin must manually delete (or manually create their own script to delete them, which is still a pain). However, their fix was to add a checkbox that defaults to the buggy behavior. Per project. So for every new user's project I have to manually tell them to turn this anti-feature off to keep from filling up the hard-drive with artifact spam. Or else write a damned script to go in and turn off this feature periodically.
The community edition of Gitlab feels a lot like the early Gnome 3 touch-screen stuff. Great in theory, but one quickly gets the sense that no one is dogfooding-- if they did these kinds of problems wouldn't exist.
At first glance, #1 seems tricky. But, I could see this as an opportunity to bring my users back to the site by providing very specific content teasers.
#2 feels possible in certain circumstances. If you can register the addresses when the user registers, you could avoid problems like changing addresses.
#3 seems like it would be problematic for the email provider who probably doesn't want to build a feature that reduces the ability to bill you. But I would pay for it.
If you had all this, couldn't this prevent the problems described here and incentivize the spammers to find a different, easier target?