Hacker News Reader

Comment by danso | Hacker News Reader

I had the pleasure of working with NPR's data news chief, Brian Boyer, who taught me a lot about actual good software practices.

I agree with his preaching the power of flat-files. Not that flat-files should be used to do things that they inherently can't...but that too many projects (or hobby apps) don't consider them and then spend as much time figuring out how to keep their server from crashing. I find it pretty amazing that they have only one small EC2 instance for their news apps (this is separate from the NPR.org site overall) just to do cron jobs.

Flat files, of course, require good planning...not least of which involves an accurate gauging of how often an app's data needs to be refreshed. But I like that kind of planning and thinking more than I do the kind it takes to maintain a stable server.

View on HN

Recently I re-engineered an ecommerce site that takes many things from the "flat-file" montra. It's amazing how little database interaction you actually need for many sites.

I think the database-driven content comes primarily from kitchen sink packages like Wordpress. More often than not, it's better to get an overall look at your structure and decide what needs to be dynamic and what can be "static".

When talking about it internally, we refer to "flat-file" as "generated". Simply meaning that it's dynamically created by a task or user interaction.

FYI I'd be totally interested in more info about how you achieve this in an e-commerce environment. Blog post?

The file system is the original "database". I solved an interview question recently using a few Linux utilities, pipes, and flat files. It was a bash one liner that impressed the interviewer, but then he asked me to solve it the "real" way.

Both databases and file systems use B-trees to implement fast read / writes, it's just that SQL databases enforce structure. OTOH, using flat files moves data checking into application space and losing certain ACID properties.

Similar trade offs are made between NoSQL and SQL, dynamic and static languages, but I digress...

I would imagine "writes" are still being written to the database, which then triggers the re-generation of the appropriate flat-file(s).

No, given a bunch of Apache logs I needed to find the top 10 queries that met conditions A and B.

1. grepping for conditions and extracting the query with sed

2. appending a char to a flat file (i.e. increasing query count by 1)

3. sort by largest files and get the top 10

Parsing log files[0]:

  for ./*.log -type f -exec echo 1 >> $(grep cond_A {} | grep cond_B | sed -E "s:.*(query).*:./results/\1.txt:") \;

Finding top 10 results:

  ls ./results/*.txt -Sl | head

[0]: Untested code, and it only grabs one line per log file instead of grepping all matching lines in the log file. I'd have to move the grep to the outside and loop the `echo 1 >>` command, but you get the point.

You probably could have done the same thing in one line:

    cat *.log | grep condA | grep condB | sed 'some regex to get rid of dates, etc' | sort | uniq -c | sort -nr | head -n 5

Or something like that... Not very efficient, but it would work in a pinch. I actually do something like this all the time for large datasets. In the time it would take me to write something better, this set of commands is already done.

Those that don't know awk are doomed to complex pipe lines.

Maybe I shouldn't be, but I am kind of surprised that you can do an ecommerce site using mostly generated files. The two things that I wonder about are (1) displaying cart/user information on every page (is it all javascript?) and (2) csrf in the forms.

Also, I would assume that javascript becomes a requirement for the functioning of the site; I always considered ecommerce to be the one part of the web were you wanted to be able to operate without javascript, so as not to turn anyone away.

I found this blog post that says that in 2010, 2% in the US had JS disabled:

http://developer.yahoo.com/blogs/ydn/posts/2010/10/how-many-...

You can also integrate with a third-party shopping cart then load a small iframe to show "2 Items in the Cart." Not my favorite way to build a site, but it is very inexpensive to build, and for small vendors the site becomes entirely static. Even if you build it in Rails, you can use page-level caching so after the first hit it's served as a static html file.

Page level caching is one of the most underused features of rails. Most applications show anonymous users identical content, and can tolerate that content being cached briefly. Setting this up gives you slashdot protection as well as access to a last chance degraded mode if there's a chain of catastrophic failures further into the app stack.

Exactly, this is the general idea of what I do in the ecommerce app I mentioned.

I guess I should have been more specific about my strategy for "generated" content in an ecommerce environment. The traditional ecommerce elements that should be real-time are; such as carts, pricing, availability, etc. Everything else is generated on update via our [custom] inventory management software.

The closest analogy I can give is Varnish's Edge Side Includes. It's not exactly the same, but it's very similar.

Serving flat files...isn't that what a caching layer is for? I don't understand why someone would go to extra trouble architecting an app around flat files on the back end, rather than building it the easiest way and sticking Varnish or nginx in front it.

> I had the pleasure of working with NPR's data news chief, Brian Boyer

I found the unexplained use of the word "Boyerism" in this article to be confusing and unprofessional. I have nothing against the use of slang, or the expectation that readers will need to Google terms: this is the reality of our culture in the age of the Internet and excellent search. However, the casual and unexplained use of a private group's inside joke reveals a lack of awareness of how search affects interacts with culture. Is it a deliberate attempt to confuse and snub the larger audience, or is in unintentional?

I think you are bothered too easily.

I agree with you...I didn't even realize that Boyer wasn't introduced in the OP by name. I mentioned his name because he is well known in news circles and frequently evangelizes about flat files. The OP should have said who Brian Boyer was even if the NPR apps blog is meant as insider baseball for news devs.

Reply on news.ycombinator.com