1,839 karma · joined June 12, 2012
The author mentions this as being a feature of Gerrit in the post.
I stopped reading exactly there, I knew that something was amiss. We followed that workflow at my last job, it worked well.
* https://git-scm.com/docs/git-update-index#_untracked_cache
The underlying filesystem needs to have certain features enabled (mtime of directories) to be able to support the untracked cache. There's no guarantee that all will, so it does not enable it.
I suppose this could be enhanced to be an opportunistic feature. However, then there is some variability in what git is doing versus there being one default behavior.
Yeah, I think it may be because it's situation dependent. For me, it always consisted of recompile all binaries and deploy them. And even then, I'm not sure people would enjoy a blog post just telling you to recompile all the things. It's not very sexy, honestly.
In my situation, it wasn't that big of a pain to do so. A lot of these libraries are within stdlib, so a security bug would incur a new release of Go. So an update should be pretty easy.
If's an external library (i.e., one picked up from GitHub for example), it's pretty much the same. Get new source, recompile binary with patched library, ship to all the things.
There are tradeoffs, absolutely. The alternative would be to write something in an interpreted language. That means we need to push a runtime environment up to the node, and that may include managing any libraries that are needed (think Rubygems). Managing these kinds of setups can be a pain, especially if a node has multiple projects with different dependencies. I found that shipping a single binary was often easier for me.
But it goes back to it being an item in the toolbox, depends on what you need and how your systems are built.
I guess I was trying to call out that, depending on the API you are supporting, it can get hairy. If you are designing it from scratch, when starting the Go project, you can structure your JSON in a way that's easy to work with.
If you're coming in to something that was easy to use in a language like Python, it may not be that easy in Go.
If you start having to Marshal/Unmarshal JSON, it'll get pretty annoying pretty quick depending on the JSON format.
I see it being used for system tools, to some degree, because it requires no runtime at all (static binary). It also seems to be pretty popular for different newer data stores. Again, if it's pure Go it'll run anywhere... so it makes development (and portability) much easier.
I like its concurrency model, to a point. That said, I hope they really improve some things about in Go 2 (like working with channels and goroutines). I'm not sure when that will be.
I use Go for a lot of things, most of which are command line applications. Whether it be interfacing with some sort of REST API, taking actions on the local system, or some sort of network client for devices on my LAN. It works well for quite a few cases.
It seems to fit well with my mental model quite often, so I use it. And it's a simple language, so it was easy to get started with it. At the end of the day, it's a tool like many others. I think it's up to the creator to choose the tool that feels best to them.
That said, while it is extremely unlikely it shouldn't be discounted as impossible.
Honestly, it's not that easy. The StartSSL process is much, much, much easier.
There are a myriad of problems you need to solve when running an internal CA for your organization. Who has access to generate new certs? Is it automated, and what security controls are around that system? What audits are there around certs being cut? Who can revoke certificates? Then you get down to the details of CRL/OSCP...
While running your own PKI is great and definitely recommended, it's much easier to say "just set up PKI" than to actually do it and maintain it.
We operate all of our datacenters in a multi-master configuration across the WAN, so latency is something we need to be mindful of. We were also in the middle of an emergency situation that required a migration, so we needed a solution that would allow us to evacuate quickly.
In the end we decided that we wanted a provider who supported a VPC-like network configuration, and was roughly within the same latency profile as Linode in respect to US-WEST-1 and US-WEST-2.
If timing hadn't been a concern, we may have chosen differently. We felt we didn't have the luxury of time.
That would indicate that Linode's password hashes may have been cracked offline.
* https://www.pagerduty.com/blog/security-monitoring-alerting-...
Most of this is still valid. There may be some differences as we've improved our configuration over time.
We use OSSEC for host-level intrusion detection. This fired off quite a few alerts as the malicious party began to log in as root on the serial console, amongst other things.
We also have supplemented it with other tools, such as an in-house wrapper around nmap, to alert us to hosts that don't match their expected network configuration. So when ports get opened incorrectly, someone is alerted usually within a minute.
I'm a PagerDuty employee and am the same individual who made this post on the last HN thread:
* https://news.ycombinator.com/item?id=10845985
Unfortunately, there are some facts in Linode's post that are not correct.
>On July 9 a customer notified us of unauthorized access into their Linode account. The customer learned that an intruder had obtained access to their account after receiving an email notification confirming a root password reset for one of their Linodes. Our initial investigation showed the unauthorized login was successful on the first attempt and resembled normal activity.
This is almost correct. Someone got in to our account on the first try. They knew the password and a valid TOTP token. Although, Linode's email isn't what notified is, it was our intrusion detection system.
>On July 12, in anticipation of law enforcement’s involvement, the customer followed up with a preservation request for a Linode corresponding to an IP address believed to be involved in the unauthorized access. We honored the request and asked the customer to provide us with any additional evidence (e.g., log files) that would support the Linode being the source of malicious activity. Neither the customer nor law enforcement followed up and, because we do not examine customer data without probable cause, we did not analyze the preserved image.
This is partially correct. We informed Linode that we saw suspicious activity within their network, and reached out to them to inform them. We provided any and all logs we had. We also informed them that we passed the info on to law enforcement, in case they wanted to proactively preserve the data. The knew we had no further information, and as such didn't ask for anything additional.
>On the same day, the customer reported that the user whose account was accessed had lost a mobile device several weeks earlier containing the 2FA credentials required to access the account, and explained that the owner attempted to remotely wipe the device some time later. In addition, this user employed a weak password. In light of this information, and with no evidence to support that the credentials were obtained from Linode, we did not investigate further.
The story behind the mobile device is totally incorrect. The user did not lose their device, the device had been restored (intentionally wiped) 9 months prior to the compromise. The user got a new device, and never set up MFA on their new phone after wiping the old one. The device was, and still is, in the user's possession. The device has not been powered on in a long while.
The user who was compromised was no longer in possession of their MFA secret. They deleted it, intentionally, with no backups existing.
If anyone here is going to be at Velocity 2016 in Santa Clara, or at Monitorama PDX 2016, I'll be giving talks on how PagerDuty was compromised back in July. This includes full details of how this happened, including the details of the mobile device referenced above. There are some details in my talk that don't line up with the blog post provided by Linode. :)
There are some issues I've seen while trying to run systems there. Some of the network configuration there is very strange, and I've seen some crazy performance issues. Their API is very painful to use as well. Lastly, they run an agent on your node that can gain someone root access (worse than Linode's ability for someone to do that). Finally, their control panel has a basically unlimited session lifetime. I don't think I've had to log in once within the past 60 days.
I've found that systems running in Azure reliably perform worse than AWS. Systems in Azure with about the same CPU and RAM have worse performance by anywhere from 2x to 10x. I'm using this based on seeing things like GC run times in both Go and the JVM. The systems report 0.00% CPU steal, no idea where the bottleneck is.
Their network also leaves some to be desired. One of the biggest pain points is that they drop ICMP Echo / Echo Reply on the edge of the network. So doing network troubleshooting across the WAN is challenging. Another issue is that they seem to often either drop or de-prioritize UDP packets within their Fresno location. This causes some issues with software that uses UDP for communication. With that are the weird, and confusing, mix-match between NetworkSecurityGroups and Endpoints, with only one of them being configurable in the UI.
The last thing is their API and the SDK (at least Ruby). Their API is an XML behemoth with incorrect documentation (e.g., the example URL using a wrong path in the docs), and severe performance problems. There are times where the API takes over a minute to respond to a request, sometimes taking longer to respond with an HTTP 500. Their Ruby SDK, at least, isn't so much as an SDK as a library that's meant to be consumed via IRB.
Lastly, the nodes all run an agent called WALinuxAgent. This allows Azure to take action on your node without your approval. It can also do things like add new users to your node, and give then full sudo access. This is also done without a reboot, so you have no indicator that someone just took this action on your system. Scary!
I've also seen this agent get weird responses from the endpoints it talks to causing it to think it should reprovision your node. It proceeds to then rewrite your SSH host keys, vomit an exception, and then exit. It's brilliant.
Trying to get help from support is impossible. I've had issues with the quality of the responses given, but also issues with them just never responding to open issues. AWS's support team should be commended in comparison.
I worked at Linode for just under three years, and worked on quite a few different things there. I started on support and moved on to a development role (including writing ColdFusion). I left Linode on good terms. California is much more enticing than NJ, so I wanted to relocate. Plus I was interested in doing more of an Ops role, instead of working including customer-facing web applications. I'm still enjoying it. :)
I think there are lessons that can be learned whenever a company has some sort of security incident. This is especially true if they are willing to publicly disclose details of the incident. We've wanted to provide what limited information we had, but wanted to wait until we had confirmation that Linode was the vector.
While there is some relief in finally determining what we believe to be the vector of our attack, it's very unfortunate that Linode engineers are dealing with the fallout right now.
The honeypot user would not have been able to access the account had the credentials been valid, but based on the information given by Linode we did see someone attempt to log in as that user only once around the time of the compromise.
Using the access gained within the Linode Manager, the attacker reset the root password on a few systems, and used Lish to gain root access. We were alerted to this activity and fully revoked the attacker's access within 60 minutes of the first node being compromised. Working with Linode support, we discovered which user account was being used and completely deactivated the user. We also isolated the VMs, and performed forensics on read-only copies of their disk images.
In our situation the attacker knew one of our user's passwords and MFA secret. This allowed them to provide valid authentication credentials for an account in the Linode Manager. It's worth noting that all of our active user accounts had two-factor authentication enabled. An interesting data point was that the user who had their account compromised was no longer in possession of the MFA secret themselves. Their cell phone had been reset (thus deleting all data) 8 months prior. The user could not log in to the Linode Manager if they wanted, so it was our determination that the key could not have been obtained from the user and was more likely on Linode's side.
We also have evidence from access logs provided by Linode that the attackers tried to authenticate as an ex-employee, whose username ONLY existed in the Linode database. It was absolutely unique and was not used elsewhere by the employee making the username an accidental honeypot. This was another piece of data supporting that Linode was the source of our compromise.
We immediately reached out to them not only to inform them of their compromise, but to assist them in investigating it. We were confident that the Linode database had been breached, and that the secret key used to encrypt information in the database had been compromised as well.
In addition to reaching out to Linode, we also worked with a third-party security firm to audit our work done during the incident. Likewise, around the same time we reached out to law enforcement to assist in investigating the attack. I believe our public disclosure includes this information[1]. This was in the middle of July 2015.
We did not get confirmation in July that there was a breach of the Linode Manager or any associated credentials.
In the end, we migrated away from Linode because of this breach (even before it was publicly disclosed) in Aug 2015. We also never were able to confidently disclose that Linode was the vector due to lack of confirmation from their end. While all of us who responded to the incident were confident they were the source, we now thankfully have the data to confirm it.
[1] https://www.pagerduty.com/blog/july-2015-security-announceme...
It takes a bit of work and determination, but it comes together in the end.
One problem with building on AWS is their lack of network diversity. If there is a network cut, causing congested links, they will do little to try and alleviate congestion to save on cost. Issues in Oregon have caused network degradation between WEST-1 and WEST-2 that last for days with no improvement.
That doesn't seem like the kind of comment to start a meaningful conversation. I must ask, what data do you have behind that statement.
I think both languages offer their own set of pros and cons.
For me, Go has been my go-to because of the maturity of the language. And, it has been implemented in systems that run at a very large scale by people within Google/YouTube as well as CloudFlare.
Would I consider it? Honestly, maybe one day. But that depends on how the language, and related things, mature over the next few years. And I think it's after the libraries have filled-in, and there is experience running systems built with it at scale.
Go is allowing me to solve the problems I have right now with some well-built packages. Is it perfect for everything, no. But it does well in a lot of areas.
To echo others, aim for low-priced and not free. That'll give you some operating revenue to avoid it going the way of the dodo.