HNHacker News
TopNewBestAskShowJobs

antonovka

1,517 karma · joined July 29, 2009

submissionscomments
antonovka··on This shouldn’t be the image of Hack Day
Ridiculous cultural relativism can be applied to almost any transgression.

If you don't like naked males, don't watch Torchwood.

If you want to be part of the technology community, but feel uncomfortable with strippers and lap-dances, too bad?

antonovka··on This shouldn’t be the image of Hack Day
The way that you're presenting it is, however. "Don't hire strippers because it makes the girls feel uncomfortable." is the subtext here.

So? You're saying male strippers wouldn't make the men feel uncomfortable?

antonovka··on This shouldn’t be the image of Hack Day
I won't deny that there are few women around here. I think there were 14ish in my program? But the problem won't be solved if you run around trying to make special rules for the girls.

"Don't hire strippers. No lap dances." aren't special rules for the girls.

Technology is possibly one of the most meritocratic fields that there are.

Yes, clearly there are no social issues and sexism in technology, which is why we get blog posts about it from women who take issue with blatant examples of sexism in technology.

Sarcasm aside, I can't recommend the above video highly enough.

antonovka··on Fat ELF binaries for multiple architectures on Linux (and possibly others)
That's way, way, way more complicated than:

1) Build a binary, put it on a DMG

2) User clicks download link, drags to any directory they want.

Trying to build packages that continue to work across releases and as libraries get updated and packages change -- and operate in a compatible manner with the built-in packages -- is surprisingly difficult.

Why don't more people do it? The only 3rd-party packages I come across are RedHat RPMs, and half the time, they're broken. I almost never come across 3rd party apt/yum repositories run by corporations, and the few times I have, they've been very broken (remember Ximian? http://en.wikipedia.org/wiki/Ximian)

antonovka··on This shouldn’t be the image of Hack Day
Some of the disturbing arguments here demonstrate just how far we have to go before women are celebrated as equals in technology.

Here's one techtalk I found to be quite enlightening: http://www.youtube.com/watch?v=TmROmy5jT80

Google TechTalks - Dr. Cornelia Brunner Associate Director, Center for Children & Technology Education Development

Anyone have others they would recommend?

antonovka··on This shouldn’t be the image of Hack Day
Wouldn't you agree that becomes a problem when it masks how people actually feel about a subject?

No -- I think the fact that it's considered culturally inappropriate to express clearly racist and sexist opinions is a benefit -- it's the first step to stemming the spread of that cultural virus.

Side note: I move to officially recognize mentions of Asperger's when discussing social situations as HN's custom version of Godwin's law.

It was an intentional reference to an endemic lack of social graces in technology culture and the oft-referenced syndrome. It was not an ad hominem argument.

antonovka··on This shouldn’t be the image of Hack Day
I wonder how many people are bordering on asperger's levels of social ineptitude and insensitivity due to the fact that it's really easy to be insensitive when you're on top.
antonovka··on This shouldn’t be the image of Hack Day
Culturally held sexism is still sexism, and such culturally established prejudice and undue partiality is not unique to Taiwan.

We shouldn't accept sexism or racism, period.

antonovka··on This shouldn’t be the image of Hack Day
I enjoy "fine women". Even at strip clubs. This wasn't a strip club.

When the "fine women" present at a technology event are paid to attend (and gyrate), we alienate intelligent, fine women (for all values of 'fine') who might actually want to be there.

antonovka··on Fat ELF binaries for multiple architectures on Linux (and possibly others)
I'm not so convinced that a centralized distribution authority that introduces a significant turn-around time (months or even years), patches the software as they see fit, and enforces their own licensing ethos is the best model for programmers to get their software to end-users/consumers.

Of course, there are two good reasons 'fatelf' has not been implemented before, and it has nothing to do with packages:

http://www.google.com/patents/about?id=b3wiAAAAEBAJ&dq=5...

http://www.google.com/patents/about?id=MQQnAAAAEBAJ&dq=5...

antonovka··on Martin Fowler describes four kinds of "technical debt"
You found articles on 30+ year old dead-obvious technology solutions to long-solved problems to be more interesting?

To respond in-kind -- maybe this indicates that you just haven't reached the level of expertise necessary to appreciate the nuances of the topic addressed here (regardless of whether you agree or disagree with his opinion) -- and moreover, perhaps you lack the perspective necessary to differentiate between architectural astronauts and people presenting real issues.

antonovka··on GitHub's Unicorn Setup
> The Erlang virtual machine has what might be called 'green processes' - they are like operating system processes (they do not share state like threads do) but are implemented within the Erlang Run Time System (erts). These are sometimes (erroneously) cited as 'green threads'.

Erlang processes' implementations share internal mutable state, despite the fact that the inter-process messaging API available via the runtime does not expose this. They have to -- internal cooperative context switching can't be implemented without shared mutable state.

They're threads.

Going back to Ruby, sure, threads might be theoretically better than processes, but in practical terms, other things are the real bottleneck, so it ends up not really mattering that much.

I'm not sure I follow. Is this something like "ruby is really, really slow, so no need to worry about concurrency" ?

antonovka··on How a 1-Engineer Rails Site Scaled to 10 Million Requests Per Day
Ruby isn't the issue here - most of the hardware spend (tiny as it is) went toward master and slave databases. Our database isn't huge, but it isn't small either. The working set doesn't fit in 32 GB of memory - lots of memory and fast disks are critical (even with caching and all that) and that takes 2 non-cheap servers.

7 machines is getting into part-time sysadmin territory.

My estimate of peak load was somewhat generous in averaging it across only 5 hours -- could you elaborate on the actual peak load?

I'm surprised that something that's orders of magnitude slower than the alternatives isn't at least part of the issue, given the amount of front-end stack required, hired employees, and 3 6-core (6?) 40 GB front-end webservers required for 10 million hits/day.

I worked on Java web apps for 8 years before starting Ravelry and I definitely don't regret choosing Ruby for this project :) Did I have to buy 1 more server than I would have had to with Java? Yeah, I think so. ...but the trade-off was definitely worth it.

The JVM truly doesn't require Java, Spring, or J2EE, and it's an incredibly impressive piece of technology to just discard.

antonovka··on GitHub's Unicorn Setup
In other words, threads are a nice boost to Erlang, but not really the secret to its success.

I think you missed my meaning. On FreeBSD 4.x and Linux 2.0.x, threads were implemented entirely in user-space. They didn't allow for true concurrent execution of multiple threads on multiple CPUs, but they were still very much threads.

Likewise, erlang's lightweight processes are exactly the same -- threads. The fact that they can be modeled to arbitrary numbers of operating system threads is irrelevant to the nature of the model -- one in which execution of a thread can proceed independently of others, implemented via context switching of execution state across those threads, able to access shared state.

However, in the fork model as described in the original article, processes are the only form of concurrency. A blocked request will, in turn, block a process, and unlike threads, far fewer processes may be run concurrently due to their significantly increased resource constraints. Furthermore, those processes are much more limited in their ability to implement low-cost inter-thread communication via shared mutable state.

If the processes actually relied on their own internal M:N scheduled green threads, then at least that part of the concurrency problem would be (mostly) solved, and fewer processes would be required. IPC is still an issue, and of course, there's the multi-decade demonstration of the high difficulty in implementing a 'performant' general-purpose M:N scheduled thread system.

antonovka··on GitHub's Unicorn Setup
Erlang is not really an argument for threads.

Solaris 2-8 implemented a general purpose M:N thread scheduler mapping user-space threads to kernel threads. This is the exact same solution Erlang implements for the less general purpose use-case of actor messaging.

Is Solaris' M:N thread implementation somehow not 'threaded'? If it is threaded (and it is), then how is Erlang's implementation not also an argument for threads?

To elucidate rather than rely on Erlang as an example, the argument for threads over processes:

1) Extremely low-cost alternative to IPC. Threads allow (but do not require the use of) shared mutable state. This is a much, much cheaper way to communicate between concurrent entities. You can achieve the same effect with processes and shared memory, but it's significantly more complex to implement and subject to the disadvantages listed below.

2) Extremely low memory foot-print. A thread costs a stack plus minor OS book keeping. A fork(2) can leverage COW pages, but almost invariably the number of non-shared pages will be significantly higher than with a thread. If an operation blocks a thread, it's cheap to create more. If an operation blocks a process, you'll hit resource constraints far more quickly trying to fork() more.

Of course, leveraging shared memory and other operating system tools, you can turn a fork(2) implementation into a thread alternative -- but then, you'd have ... threads. This is what Linux's clone(2) syscall was intended for -- a thread-implementation friendly fork(2) alternative -- and the pthread library was built on top of it.

... for a while, you could call setuid() in a Linux "thread" and the new uid would be a thread-local change, because the thread was actually a 'process'.

This is effectively not an argument against fork(2) (although it's an expensive route to the same solution offered by threads), but rather against scaling models that will block an entire OS process for the sake of a single request.

antonovka··on GitHub's Unicorn Setup
If they are using http caching and ESI properly, the average request time would be significantly lower, and the requests per second would probably be > 500/s on that box.

500 req/s is pretty abysmal for a 8 core 16 gigabyte server. I realize that some of that has to do with Ruby performance, but yikes -- that is frighteningly bad scaling, regardless of the cause.

antonovka··on How a 1-Engineer Rails Site Scaled to 10 Million Requests Per Day
Yes. http://en.wikipedia.org/wiki/ColdFusion#ColdFusion_and_Java
antonovka··on How a 1-Engineer Rails Site Scaled to 10 Million Requests Per Day
In-process libraries that are orthogonal to your domain aren't a golden ticket. SOA lets you scale the parts that need to, one service at a time, be that caching, search, storage or domain logic.

SOA is also considerably more complex to architect, configure, and administer. If you don't need it yet, why pay for it?

I disagree that in-process libraries are actually less complex architecture, it just hides the complexity within a single process.

I'm not sure what you mean by "hide". Using these tools you can scale up a much larger order of magnitude before having to worry about breaking out services.

We're talking about the difference between 1-2 servers and 10-20, and that's a fairly substantial difference in man hours, architectural complexity, physical plant, etc, especially early on in a startup.

Chances are good you'll never even need to grow to that scale, either -- we ran a game site with 4 million in yearly revenue (and quite a bit more than 10 million requests/day) on about 8 mid-range servers.

antonovka··on GitHub's Unicorn Setup
I think we all understand the difference between threads and processes and where context switching fits in.

You might, but I don't think that's the general case.

From the github dude: "Our Ruby application code spends very little time blocking on external resources." That is why using threads in this particular case won't help.

It hits a database, loads content from memcached, and waits on HTTP requests to complete, does it not?

I'd be surprised if the usage modeled a pure event-driven sendfile() based static-only file server (such as lighttpd), for instance (where threads vs. processes is moot).

Moreover, a fair amount of effort has been expended on a complex architecture to push only a very specific type of requests to Unicorn.

antonovka··on GitHub's Unicorn Setup
Erlang can use native threads, but they do not map to its lightweight processes.

That's not what I said; I said: "[Erlang] maps lightweight erlang processes to operating system threads."

Erlang processes are green threads mapped to OS threads. They're called processes in erlang, but they're M:N scheduled threads.

So on a four-core system, you might have one thousands of Erlang lightweight processes running on just four OS threads in one OS process.

Or you could have thousands of Erlang lightweight processes mapped to 185 OS threads (not that it will, but it could). The fact that they're green threads is irrelevant in terms of scalability -- they're still threads, but even less resource intensive than a direct mapping to OS threads. M:N scheduled green threading is also very hard to do in the general case, which is why you don't see more of it (see the abandoned attempts to implement generic M:N OS:green thread scheduling in FreeBSD, for instance).

You could also be using Scala, where an actor only maps to a thread when it's blocked inside of its react loop, but in that case will consume a whole thread.

What's your point? :)

antonovka··on Google Code Now Supports User Clones for Mercurial
I very much appreciate their use of the word 'clone' instead of 'fork', and the focus on maintaining the idea of canonical repository and project:

  User clones aren't forks, in the traditional sense,
  where a fork has little intention to contribute back
  to the original project. Rather, the entire purpose
  of a user clone is to allow users to contribute to
  projects without requiring official commit access
  permissions.
I was thinking of migrating some projects to github for this feature, but instead I think I'll just switch to Mercurial on Google Code.
antonovka··on How a 1-Engineer Rails Site Scaled to 10 Million Requests Per Day
Of course, using Java on bare metal is going to have more performance/$ than ruby on Xen. That's not the point. The point is to use tools that maximize human capital at the expense of server costs.

That assumes that the tools maximize human capital more than the tools that maximize both human capital and minimize server cost. Do they?

I don't know what you mean by reducing moving parts. The only parts involved in handling the requests for this setup are Nginx and HAProxy. All the other technology parts (Tokyo Cabinet, memcached, Sphinx) are used to scale the database, something your stack still has to achieve.

Lucene, ehcache -- in-process libraries to achieve the same effect but without all the moving parts. Or, instead of lucene, PostgreSQL's built-in FTS.

You also don't need nginx, haproxy, and a complex architectural division between static/non-static content.

The tradeoffs between using bare metal vs VM and using Rails vs Java are well known at this point.

I'm not sure I agree, given the constant repetition of "Ruby is more productive" as a justification for poor performance and complex architecture. Is it more productive than Scala? Clojure? Groovy? What about using even just using JRuby?

antonovka··on GitHub's Unicorn Setup
You seem hung up on the number of concurrent requests. An 8 core machine can only do 8 things at once, no matter if you are using threads or processes.

You can't assume that the threads are 100% CPU bound. If they were 100% CPU bound, you would have a point -- 8 cores, 8 CPU-bound processes, no processing time left over -- that is where event based and hybrid thread/event-based architectures excel.

In reality, webapp threads will sleep -- waiting on the network, waiting on disk, waiting on the database. When they sleep, the the processor has nothing to do. If the OS can schedule another thread while one sleeps, work can move forward.

If you can run 500 threads to completion in 10ms, then you can serve 500 requests within 10ms.

If you can run 16 (or 32) processes to completion in 10ms, then you can only serve 16 (or 32) requests within 10ms.

antonovka··on How a 1-Engineer Rails Site Scaled to 10 Million Requests Per Day
That may be partially true[1], but I'd challenge you to re-implement this non-optimized case (ie, this is a rough webapp, no caching, etc) in another runtime on similar hardware and see:

1) Whether it can support anywhere near the same level of concurrent requests with similar response times.

2) How much complexity (nginx + unicorn + memcached + puppies) is required to achieve this in comparison to a servlet engine (eg, tomcat) and your webapp.

[1] Most web applications are read-heavy, low on writes, and scaling up write capability generally requires scaling up the database. You can grow quite a bit with simple caching and monolithic database scaling before having to tackle more complex distributed data architectures.

antonovka··on GitHub's Unicorn Setup
Each box can handle plenty more than 16 workers, but we currently don't need that many, so it's pointless to run them. Our frontend machines generally sit about 60% idle. We wanted to have plenty of headroom on our new setup.

At those numbers, it sounds like you'd max out at roughly 32 workers (so 32 concurrent requests), or am I missing something important?

It's true that running Ruby code is generally more expensive (dollar-wise) per given task than other languages, but we made a conscious decision to make that tradeoff for the productivity gains that Ruby affords us.

It's really easy to measure performance, but productivity claims seem difficult to support and highly susceptible to confirmation bias.

I think that Ruby, Scala, and Clojure (just for instance) would stack up against each other very well in terms of productivity, but I wouldn't know how to prove it.

antonovka··on How a 1-Engineer Rails Site Scaled to 10 Million Requests Per Day
You can support 10,000 requests/second hitting a servlet engine, with dynamic templates, and backend DB calls, on one quad core box?.

5-10k, depending; a local instance of a recent webapp I wrote can run 5.9k requests/sec on a simple page JSP-templated page backed by a database request. It's running on tomcat, using servlets with a lightweight REST API, and postgresql.

The webapp I'm working on right now has the following performance profile for a page that fetched the user-list from the backing database. Not as fast, but 2k req/sec is not bad, and I haven't done any profiling on the new stack we're using at all yet:

  Server Software:        Jetty(6.1.x)
  Server Hostname:        localhost
  Server Port:            8080

  Document Path:          /users
  Document Length:        5855 bytes

  Concurrency Level:      4
  Time taken for tests:   0.508 seconds
  Complete requests:      1000
  Failed requests:        0
  Write errors:           0
  Total transferred:      6045474 bytes
  HTML transferred:       5855000 bytes
  Requests per second:    1968.41 [#/sec] (mean)
  Time per request:       2.032 [ms] (mean)
  Time per request:       0.508 [ms] (mean, across all concurrent requests)
  Transfer rate:          11621.09 [Kbytes/sec] received

  Connection Times (ms)
                min  mean[+/-sd] median   max
  Connect:        0    0   0.1      0       1
  Processing:     1    2   1.7      2      22
  Waiting:        1    2   1.7      2      22
  Total:          1    2   1.7      2      22

  Percentage of the requests served within a certain time (ms)
    50%      2
    66%      2
    75%      2
    80%      2
    90%      3
    95%      3
    98%      4
    99%     12
   100%     22 (longest request)


I'd like to give you a job.

Something in your tone tells me you're not serious ...

Maybe I've been working with heavyweight frameworks too long, but that seems an order of magnitude off at least.

I couldn't say; I do have local 3rd party spring-based webapps that take 5 minutes just to start up, not to mention 300ms+ to render a single page, so I wouldn't be surprised.

antonovka··on How a 1-Engineer Rails Site Scaled to 10 Million Requests Per Day
Same as anything else -- you start by adding a few more front-end servers (or VMs), or scaling up the ones you have just a bit more. You don't have to do nearly as much work, and not nearly as soon.

If load levels continue to grow, leverage ehcache to handle caching directly in the VM (if you're not already), begin splitting responsibilities across different service implementations, etc.

antonovka··on How a 1-Engineer Rails Site Scaled to 10 Million Requests Per Day
The summary: An surprisingly large investment in a complex platform development and extensive hardware purchases.

Let's be really, really generous and say peak load on 10 million requests is 5 hours, and pack all 10 million requests into 5 hours. That's 2 million requests/hour, 33,333 requests/minute, 555 requests/second.

You can easily handle 5-10k requests a second with a few ms response time -- including dynamic templating and hitting a backend database -- on a 2.66 GHz 4 core Xeon 5150 with a couple gigabytes of RAM, using a servlet runtime and a JVM-based language.

Basically, you could take that entire solution stack and compress it down to a few items with far fewer moving parts, and spend half as much on engineering and capital expenditures.

My hope is that efforts like MacRuby bring this level of free performance to the Ruby world, and that languages like Scala and Clojure make the JVM more attractive, too.

antonovka··on GitHub's Unicorn Setup
The argument encompassed two aspects of the model:

1) "Threads are out -- processes are better than threads."

2) Process-per-connection architectures.

The first is demonstrably false -- for instance, look at Erlang, which maps lightweight erlang processes to operating system threads, providing SMP scalability at a low cost without running into Github's issues with mongrel "thread-killing". More broadly used, look at Servlets and the Servlet 3.0 support for async comet-style event-based request handling. Each request is handled on a thread as necessary. Inter-thread communication (where necessary) is cheap, and this scales just fine.

If you use the fork() model and a long-running request blocks the entire process, comet is basically a non-starter. This is why people are interested (and implement) lightweight threads, coroutines, and restartable request implementations.

For a conceptual challenge, consider how you would implement a live web chat system that can scale up to a considerable number of clients, with extremely low resource usage, and instant message distribution (no polling). Locally, we implemented this with async servlet support and M:N scheduled scala actors -- a blocking HTTP request doesn't hold a thread or process hostage in the web server or the application, and we can scale up to enormous number of live clients on one machine.

The second was merely a lack of understanding of the model. In implementations where fork() is used as an alternative to threads and multiple connections are not handled per sub-process, you quickly run into scaling issues with subprocess memory utilization. In cases where subprocesses handle multiple connections via an event mechanism, you're just using fork() instead of threads.

I'm wondering how many servers github is using to keep up with load -- If each 8 core 16 gigabyte (!!!) server can actually only handle 16 concurrent requests via a pool of 16 workers, that's an incredibly poor (and expensive) scaling model.

antonovka··on Ask HN: OSX Package Management
I'm fine with the App system on the Mac. It's ultrasimple. Package management is a teeny bit simpler, but it's not enough for me to bother over.

I'm not sure it's actually simpler; as a software author, my experience has been:

If I want to release something for the Mac, I post a dmg containing a dragable application; and even the most non-technical user can download and install it immediately. Sparkle provides automatic updates via my website; users gets the latest versions whenever they want them, I don't have to do anything complicated.

If I want to release something for the UNIXes, I provide a set of autoconf scripts and a make-based build system (or one of the recent alternatives, or a Python egg, or a Ruby gem), hope that the package management systems pick it up during their next release cycle, wind up in an argument with Debian over OpenSSL+GPL licensing compatibility, and see my software get shipped with half the featured turned off and a ridiculous number of local hacks applied, including all-too-regularly replacement of a working library dependency with something that doesn't work (but Debian likes it more).

The average end-user doesn't see my updates for months or years, the software is missing features and locally modified, and I have to wonder: Why couldn't I just give them something that they could drag to their applications folder?

← PreviousPage 5 of 12Next →