Hacker News Reader

How many reads do you do on that server? 192GB + 8 SSDs is a pretty serious setup. Just the disks themselves should be able to push 500K+ random IOPS.

It's actually very much over built. I had the budget, so I built for growth. This setup should easily last us through this year, assuming 4x growth over last year. I am actually putting together another similarly spec'd machine with 384gb of ram (why not, it's cheap) because the current secondary is on 15k disks with only 24gb of ram. Still more than enough to handle the load through this last year, but probably not this coming year, at least without a bunch more ram.

In regards to actual iops, not sure what this thing can peak at off the top of my head, but we'll easily be doing 100 queries a second this year, with a considerable portion of those queries pulling out ~1mb documents.

Playing it conservative, so I am moving towards gzipping those large documents (never need to access anything but the full data, > 90% of accesses are directly served to clients that can handle inflating the data). For now they will stay in mongo, but I am building out an evaluation of using a flat file structure and just letting nginx pass them out.

"just letting nginx pass them out": proxy cache could work as well... Volume of data is much less here, but I let nginx cache gridFS stored photos...

One thing to note that I left out above, I also use the same severs for mysql. Our mysql working set is something like 30gb now, so I have a decent chunk of that ram apportioned to mysql.

Additionally our mysql db sees many more queries than mongo, so the overbuilt hardware is a bit less overbuilt when taking that into consideration :)

670 gigabytes is a puny database size. You should be able to press so much power through a system with a disk system like the one you have. I would seriously consider a Postgres setup on a data set of that size. Additionally, I would probably just store the JSON data directly inside postgresql.

See comment below. Definitely a small database size. You hit the nail on the head. It's going to grow fast this next year, and I'd like to put off sharding as long as possible, hence gzipped storage of the data.

postgis isn't a good fit for the data we store in full fidelity, since it's not just geo data but also sensor data (heartrate, cadence, power in watts, temperature etc). However I'll be storing a point reduced version of the full track in postgis, so i can move to using actual intersection queries for matching tracks, instead of the current brute force approach (check everypoint in every track sharing a bounding box) that works now. All bets are out the window though with 2-4x the traffic and data we currently have, using that brute force approach.

I already run another beefy postgis setup (192gb ram, though spinning disks not SSDs) for serving OSM maps, and eventually OSM routing hence the ram.

> postgis isn't a good fit for the data we store in full fidelity, since it's not just geo data but also sensor data (heartrate, cadence, power in watts, temperature etc).

Dumb question: why not?

because it's faster to serve the full fidelity track, with all sensor data for every lat/lng pair, straight from disk as a gzipped file. I can store a point reduced version in postgis for actual geo operations, since that only needs lat/lng pairs, and not at 1hz sampling.

Pulling 21k points (1.8mb of JSON) for a single recording out of disk into memory, turning it into JSON, sending it over the wire to the client, is slow (500ms just to query, pull off disk and turn into JSON). Serving the same data pre-compressed, is 2ms. This isn't even considering what it takes nginx to compress it on the fly for every request.

So it's because you don't want to recreate the track from point data on each request. I can see why that's advantageous.

OK, new dumb question. It sounds like you're serving gzip'd JSON straight off the disk with nginx (serving gzip'd files direct from disk is one of my favourite nginx features). Where does Mongo come into the picture?

>data that isn't updated frequently

How often did you update your data then? In my current project I am seeing locking issues in my way soon...

I think I peaked at 5% lock utilization this year, so I haven't seen any real issues.

Our actual track data isn't updated frequently. Mostly it serves as an archive for a user, and is only seen by 1-2 other people. Most people use our service to store all their activities, which for the most part are really boring. They are interested in aggregate metrics like "I've ridden 200 miles this month".

A smaller portion of our data is from planning a route using google maps, which has much more modest storage requirements, since it's optimized data (one point every mile if it's a straight line) instead of 1hz logging from a GPS unit. This stuff is edited, but I'd say only 10% of planned routes are ever modified, so actual updates on the track data are small.

> using mongo as a very simple key/value store for data that isn't updated frequently, which could easily be replaced by flat file storage, is painless. YMMV with other use cases.

This is our use case as well and MongoDB has been fine. We had some initial pain as we learned the product but it's great for this use case. Currently sitting around 1TB of data.

> 192gb of ram with 8 RAID10 512gb SSDs probably masks performance issues

I would hope so.

Which data centre are you hosting in? Even when I was colo'ing back at 500px that much RAM wasn't "cheap". "Fairly priced" is the way I would put it.

We build our own machines and host them out of a carrier neutral facility in Portland, OR. $200/mo unmetered unlimited 100mbit connection and $1100 for a full rack with redundant 20 amp feeds. It would cost us something like $5k/mo to run our 9 servers on amazon.