Hacker News Reader

Comment by mrtksn | Hacker News Reader

If data needs to be processed, an option would be to use MongoDB for collecting the data in bulk and later decide how you need to structure it for your needs.

When it comes to "read" the data, you read it only from processed database that can be anything.

Since you can do your processing completely independently of your web server, your are not necessarily pushing your computational load from the DB to the web server.

View on HN

> If data needs to be processed, an option would be to use MongoDB for collecting the data in bulk and later decide how you need to structure it for your needs.

OR, i can open a file stream, serialize my data to JSON, entity by entity, and dump all to a file.

The good old file.

replied to the same question here: https://news.ycombinator.com/item?id=14805812

Because we want to do partial updates, searches, indexing etc.

Your position can be applied to all databases. Why not abandon them all and just use CSV ?

You could always do that, dump the data to s3, and use Athena :-)

https://aws.amazon.com/athena/

And then be locked-in to Amazon. From ubiquitous files to ... a single service provider.

Then again, if you're going to push into Postgres you might as well store the temp unstructured records there too

Well, that's also an option :) But I believe this is a relatively new option.

Personally, I never got into the hype with the NoSQL and for me the only use is as a convenient API for storing JSON files during prototyping so that I can later decide how my data should be structured.

Definitely tech debt for the future but fast iterated design that needs the database stuff fixed has more chance for success than the rigid design with the perfect data architecture.

It that point it's harder for me to understand why I'm not just writing out JSON files or something.

You definitely can but MongoDB provides convenience over storing and managing JSON files on you filesystem by your own efforts.

Sure, but it is also another service to keep running. Depends on what scale you're operating at.

Any RDBMS is just a toolset to manage files on some filesystem. If you are dealing with some config files you can spare yourself the trouble and just load them from the filesystem.

NoSQL or classic RDBMS, these tools give you ways to manage your data on the filesystem(or in the memory?). You can spend time to write a code that processes files in a folder or you can use the software that does it for you. You can write a code that relates two files in a folder or you can use a software that already can do it and you can just ask that software for the outcome.

I don't really have to write that code though; what language doesn't already have file-manipulation libraries?

Well I guess some people wonder why people are using ready software when you can achieve the exact same thing with a text editor and a compiler.

That's not the same thing at all; you're suggesting an approach that probably takes more code and definitely takes more setup.

You are underestimating the complexities of file management

Again, depends on scope and scale. If you have a handful of items to process it's not a big deal.

No, I really can't think of any situation where installing, managing, updating, maintaining, and crying over a mongodb cluster would be easier for a flat-file datastore than say, a file system, or S3. In fact if one is on AWS, then nothing beats just dumping them in S3 and processing them as necessary. If I'm a startup and I really wanted some querying then PostgreSQL looks great, with Amazon handling the maint and backups.

You can even query flatfiles in s3 using SQL via AWS Athena.

Reply on news.ycombinator.com