98 karma · joined June 29, 2012
The Internet Archive is looking for a datacenter & network engineering to help us with our physical datacenters (one is a converted church in SF!) and networking stack. We have a global site pushing 100s of gigabits out of on-prem and bare metal, adding 150TB a day of new data.
More info and application at https://workforcenow.adp.com/mascsr/default/mdf/recruitment/... , but please email me at kevin.oconnor@archive.org for more info or a quick chat about the position!
I've done this in past roles on AWS with their i3, etc. family with local attached storage and didn't use EBS.
A couple notes: - the verification email went to my spam folder on Gmail - acknowledged is misspelled on this image https://kintaba.com/images/collab_splash.png
We don't have a normal CDN, we have Fastly. They are really incredible at what they do, and this would not have been possible with our previous CDN partners.
Creating a "custom database program" is not a small task.
We like to use boring technologies that we know work well. We were already using Cassandra, had some experience with Redis, and had a lot of confidence in our CDN.
That said - we use Redis in at least 2 (soon to be 3) capacities across the site for different services (outside of the monolith) and it works really well.
https://github.com/reddit/reddit-mobile/compare/e3f1119e3358...