ppierald

I just got back from a trip to Death Valley. While it was extremely hot (116F), it was very beautiful. The mountains are where you probably want to go for hiking and camping. The valley floor is where all the heat gets trapped in. I highly recommend Death Valley. Many people stay tuned for the short-lived super blooms of wildflowers that pop up from time to time in spring after perfect conditions allow. It is supposed to be quite spectacular.

Bummer for the company and I wish them well as they navigate the incident response waters. What I found interesting from the article was

If users created or changed their password after March 14, 2012, it is hashed with a function called bcrypt. If users have not changed their password since then, it is uniquely salted and hashed with SHA-1.

They could have migrated all accounts regardless of whether they came back to the site or not, changed their password or not, etc. Just use:

password_hash = bcrypt(sha1(raw_password))

We use it to "set a timer for 10 minutes" when cooking something and "what's the weather for tomorrow" to know whether to bring a coat or not.

I guess they are solving similar problem, but this is automatically sending the email to the recipient. Also the integration with GPG may be preferable to some people if they don't have an extensive community of Keybase peers.

I would agree with this sentiment, however, there are a number of things that you can to to make the job of the attacker harder, or to notice internally when there is inappropriate usage. Service abstractions to encrypt/decrypt the data will limit the surface area where you might store keys. Logging usage to know what operations were made. Also, you can use a system like vault such that the keys are not extractable to do all the crypto operations offline thus massively increasing the chances you would detect such a breach / unauthorized access.

vault (https://vaultproject.io) uses Shamir to generate shares for operators to unseal the vault. In the latest release (1.0beta), vault seal keys can be wrapped with something like AWS KMS which allow for operator-less unsealing of the seal keys, but trade that off with potential operator access to the seal key itself.

Collusion between operators is a real problem of Shamir-based key systems. If you have a crypto system that depends on Shamir keys and a few of your operators leave the organization or become untrustworthy for some reason, then you need to revoke / resplit the origin key data material.

Additionally, the output of the ssss command itself is not secure, so you should consider having that data going through GPG to give each operator a GPG / keybase'd output which has never been seen in cleartext by anyone but themselves.

Long story short, key management continues to be really hard and needs to be thought through from begin to end with operational procedures in place to handle the real life situations that occur (employee collusion, employee join/depart, breach).

In the 90's this was rare and a big lure for candidates. Now it's table stakes.

We're still removing your code. Thanks buf. :)

Makes me think of the recent XKCD on voting machine software: https://xkcd.com/2030/

There is a lot of commentary about the use of vault as an alternative, number of secrets needed, etc. I think the inclusion of Secrets Manager is a great addition for AWS and will definitely help people get better control over their secrets, however, vault contains richer functionality than just secrets key/value storage.

It can provision users to backends like SSH, databases, cloud providers, and such. Use is audited, can be revoked, and has a TTL associated.

Additionally, vault contains a full "crypto-in-a-box" implementation that allows for sign/verify, hmac/verify, encrypt/decrypt, random number generation, and other functions.

So I applaud AWS for doing this and hope the will continue developing KMS/HSM/Parameter Store/Secret Store/??? in the future and innovating, but evaluating Secret Store vs. Vault simply on price may be a short sighted comparison.

Disclaimers: Employer is an AWS customer using vault

Go nearly anywhere in Argentina and you will find mate. People walk around with a thermos of hot water and sip all day. It is a shared cultural experience too like saying hello. You see a friend, greet them, then offer them some mate.

Our SF-based company has an office in Mendoza, Argentina. You will not find baristas and coffee (though Starbucks just opened there), but plenty of mate for everyone. Due to this connection, we have plenty of supplies for mate in the office, but it is still not as commonly adopted here as coffee.

> Finally, we’d like to assure our customers and partners that we take security at Mailgun very seriously.

So very seriously that they don't even use https for their blog...

"Between 7:20 AM and 8:04 AM PST we observed connectivity issues with an external provider outside of our network that impacted internet connectivity between some customer networks and the US-EAST-1 Region. Connectivity within the Region and between Regions was not impacted. The issue has been resolved and the service is operating normally."

Are people reporting issues coming from the US or from other parts of the world. We are not seeing any issues from the US, but other customers and monitoring points are reporting sporadic blips.

$58 in 2014, $120 today? Follow the money.

My anecdotal evidence shows a dramatic shift of my 20-something and 30-something year old co-workers to Oakland from SF due to affordability. Oakland is close, cheaper (though rising), diverse, and close by BART. This is probably putting a good deal of supply back into the SF market, but putting pressures in the East Bay markets (Berkeley, Oakland, Alameda).

Thanks! That was a super helpful response!

I always thought the term Tofu was "mojibake": https://en.wikipedia.org/wiki/Mojibake

Wikipedia disambiguation page: for "Tofu" mentions: Slang for the empty boxes shown in place of undisplayable code points in computer character encoding, a form of mojibake

https://en.wikipedia.org/wiki/Tofu_(disambiguation)

I would be interested in the details of the storage mechanism of the global pepper. Is this in an HSM? For AWS customers, something like KMS? There are then huge operational and redundancy issues to think about. Failovers for your HSM. Handling the possibility that AWS might not be available or corrupt the key, other cases. These things are easy to whiteboard, but when the rubber hits the road and you need to think about all the operational edge cases, things get hard quick.

The u= parameter in the OPs article also is vulnerable (even if http/https are whitelisted, file:// blacklisted, etc) to the #10 vulnerability on the OWASP Top Ten 2013 list, namely Unvalidated Rediredcts and Forwards. https://www.owasp.org/index.php/Top_10_2013-A10-Unvalidated_...

I can easily get you to click the link to drive-by malware, adult sites, pharma, phishing, etc. because the site doesn't ensure where the link is actually going to.

Developer productivity increases by an order of magnitude.

Sue Decker, ex-Yahoo CFO & President sits on the Board of Directors for Birkshire Hathaway. I'm sure she has plenty of insight to the value of the company, and the complexities of its business.

My undertanding was that the Customer Master Key was stored in an HSM, but your customer-generated keys were not. I might be wrong about that. So if true, AWS employees would not have access to your root key material, but the definitely the intermediary key material. It's a cost trade-off

I totally bought a password and it was the best $2 I spent all weekend. I have no intention of using it, but more of a pat on the back telling her to "keep it up" and that people approve. Last thing she needs is telling her she's doing the wrong thing here.

<script src="..." is very dangerous. At best, you can vet the src and check to see if it's benign or not. Often times, that vendor and their "1-line of javascript to get our whiz-bang service" in turn loads other javascript files. I don't see how cryptographically signing the bootloader solves anything in this case. Compromised analytics or vendor javascript will still lead to total site pwnage if I'm reading this right.

Another good article on git-signing: http://mikegerwitz.com/papers/git-horror-story

I can imagine a scenario where a malicious employee is intentionally injecting malicious code (backdoor, whatever) and wants that commit tied back to someone else (their enemy, boss).

The one thing that is appealing to an "AWS shop" and something that GitHub and other services have a harder time emulating is the integration with IAM (user authentication and provisioning) and the CloudTrail (audit trails). For any reasonably sized organization, ensuring that users get provisioned / deprovisioned is a tricky, error-prone task. Having IAM as a single toggle is nice. Using their SAML integration into something like Okta, OneLogin, or other IaaS-providers ... even better. Getting user activity into centralized logging via CloudTrail (Splunk, Sumo, ELK, et al.) is best practice for security teams. AWS makes all of this pretty easy, but at a price.

Agreed. Just want to point it out to those that jump in head first without understanding the consequences of that action. You might get fired from your job for doing this or severly reprimanded. You might unwittingly cause a production data breach for some reason you never even imagined. So it's a message of be careful and understand fully what you are doing.

It's important to know that your content, intellectual property, and possibly confidential user data including full http request/responses are being proxied by this service. Their privacy policy is vague about what they will do with this information other than to say they will collect and process it:

https://meetfinch.com/privacy

"In running and maintaining our website we may collect and process the following data about you:

Information about your use of our site including details of your visits such as pages viewed and the resources that you access. Such information includes traffic data, location data and other communication data. Information provided voluntarily by you. For example, when you register for information or make a purchase. Information that you provide when you communicate with us by any means."