If you don't like naked males, don't watch Torchwood.
If you want to be part of the technology community, but feel uncomfortable with strippers and lap-dances, too bad?
1,517 karma · joined July 29, 2009
If you don't like naked males, don't watch Torchwood.
If you want to be part of the technology community, but feel uncomfortable with strippers and lap-dances, too bad?
So? You're saying male strippers wouldn't make the men feel uncomfortable?
"Don't hire strippers. No lap dances." aren't special rules for the girls.
Technology is possibly one of the most meritocratic fields that there are.
Yes, clearly there are no social issues and sexism in technology, which is why we get blog posts about it from women who take issue with blatant examples of sexism in technology.
Sarcasm aside, I can't recommend the above video highly enough.
1) Build a binary, put it on a DMG
2) User clicks download link, drags to any directory they want.
Trying to build packages that continue to work across releases and as libraries get updated and packages change -- and operate in a compatible manner with the built-in packages -- is surprisingly difficult.
Why don't more people do it? The only 3rd-party packages I come across are RedHat RPMs, and half the time, they're broken. I almost never come across 3rd party apt/yum repositories run by corporations, and the few times I have, they've been very broken (remember Ximian? http://en.wikipedia.org/wiki/Ximian)
Here's one techtalk I found to be quite enlightening: http://www.youtube.com/watch?v=TmROmy5jT80
Google TechTalks - Dr. Cornelia Brunner Associate Director, Center for Children & Technology Education Development
Anyone have others they would recommend?
No -- I think the fact that it's considered culturally inappropriate to express clearly racist and sexist opinions is a benefit -- it's the first step to stemming the spread of that cultural virus.
Side note: I move to officially recognize mentions of Asperger's when discussing social situations as HN's custom version of Godwin's law.
It was an intentional reference to an endemic lack of social graces in technology culture and the oft-referenced syndrome. It was not an ad hominem argument.
We shouldn't accept sexism or racism, period.
When the "fine women" present at a technology event are paid to attend (and gyrate), we alienate intelligent, fine women (for all values of 'fine') who might actually want to be there.
Of course, there are two good reasons 'fatelf' has not been implemented before, and it has nothing to do with packages:
To respond in-kind -- maybe this indicates that you just haven't reached the level of expertise necessary to appreciate the nuances of the topic addressed here (regardless of whether you agree or disagree with his opinion) -- and moreover, perhaps you lack the perspective necessary to differentiate between architectural astronauts and people presenting real issues.
Erlang processes' implementations share internal mutable state, despite the fact that the inter-process messaging API available via the runtime does not expose this. They have to -- internal cooperative context switching can't be implemented without shared mutable state.
They're threads.
Going back to Ruby, sure, threads might be theoretically better than processes, but in practical terms, other things are the real bottleneck, so it ends up not really mattering that much.
I'm not sure I follow. Is this something like "ruby is really, really slow, so no need to worry about concurrency" ?
7 machines is getting into part-time sysadmin territory.
My estimate of peak load was somewhat generous in averaging it across only 5 hours -- could you elaborate on the actual peak load?
I'm surprised that something that's orders of magnitude slower than the alternatives isn't at least part of the issue, given the amount of front-end stack required, hired employees, and 3 6-core (6?) 40 GB front-end webservers required for 10 million hits/day.
I worked on Java web apps for 8 years before starting Ravelry and I definitely don't regret choosing Ruby for this project :) Did I have to buy 1 more server than I would have had to with Java? Yeah, I think so. ...but the trade-off was definitely worth it.
The JVM truly doesn't require Java, Spring, or J2EE, and it's an incredibly impressive piece of technology to just discard.
I think you missed my meaning. On FreeBSD 4.x and Linux 2.0.x, threads were implemented entirely in user-space. They didn't allow for true concurrent execution of multiple threads on multiple CPUs, but they were still very much threads.
Likewise, erlang's lightweight processes are exactly the same -- threads. The fact that they can be modeled to arbitrary numbers of operating system threads is irrelevant to the nature of the model -- one in which execution of a thread can proceed independently of others, implemented via context switching of execution state across those threads, able to access shared state.
However, in the fork model as described in the original article, processes are the only form of concurrency. A blocked request will, in turn, block a process, and unlike threads, far fewer processes may be run concurrently due to their significantly increased resource constraints. Furthermore, those processes are much more limited in their ability to implement low-cost inter-thread communication via shared mutable state.
If the processes actually relied on their own internal M:N scheduled green threads, then at least that part of the concurrency problem would be (mostly) solved, and fewer processes would be required. IPC is still an issue, and of course, there's the multi-decade demonstration of the high difficulty in implementing a 'performant' general-purpose M:N scheduled thread system.
Solaris 2-8 implemented a general purpose M:N thread scheduler mapping user-space threads to kernel threads. This is the exact same solution Erlang implements for the less general purpose use-case of actor messaging.
Is Solaris' M:N thread implementation somehow not 'threaded'? If it is threaded (and it is), then how is Erlang's implementation not also an argument for threads?
To elucidate rather than rely on Erlang as an example, the argument for threads over processes:
1) Extremely low-cost alternative to IPC. Threads allow (but do not require the use of) shared mutable state. This is a much, much cheaper way to communicate between concurrent entities. You can achieve the same effect with processes and shared memory, but it's significantly more complex to implement and subject to the disadvantages listed below.
2) Extremely low memory foot-print. A thread costs a stack plus minor OS book keeping. A fork(2) can leverage COW pages, but almost invariably the number of non-shared pages will be significantly higher than with a thread. If an operation blocks a thread, it's cheap to create more. If an operation blocks a process, you'll hit resource constraints far more quickly trying to fork() more.
Of course, leveraging shared memory and other operating system tools, you can turn a fork(2) implementation into a thread alternative -- but then, you'd have ... threads. This is what Linux's clone(2) syscall was intended for -- a thread-implementation friendly fork(2) alternative -- and the pthread library was built on top of it.
... for a while, you could call setuid() in a Linux "thread" and the new uid would be a thread-local change, because the thread was actually a 'process'.
This is effectively not an argument against fork(2) (although it's an expensive route to the same solution offered by threads), but rather against scaling models that will block an entire OS process for the sake of a single request.
500 req/s is pretty abysmal for a 8 core 16 gigabyte server. I realize that some of that has to do with Ruby performance, but yikes -- that is frighteningly bad scaling, regardless of the cause.
SOA is also considerably more complex to architect, configure, and administer. If you don't need it yet, why pay for it?
I disagree that in-process libraries are actually less complex architecture, it just hides the complexity within a single process.
I'm not sure what you mean by "hide". Using these tools you can scale up a much larger order of magnitude before having to worry about breaking out services.
We're talking about the difference between 1-2 servers and 10-20, and that's a fairly substantial difference in man hours, architectural complexity, physical plant, etc, especially early on in a startup.
Chances are good you'll never even need to grow to that scale, either -- we ran a game site with 4 million in yearly revenue (and quite a bit more than 10 million requests/day) on about 8 mid-range servers.
You might, but I don't think that's the general case.
From the github dude: "Our Ruby application code spends very little time blocking on external resources." That is why using threads in this particular case won't help.
It hits a database, loads content from memcached, and waits on HTTP requests to complete, does it not?
I'd be surprised if the usage modeled a pure event-driven sendfile() based static-only file server (such as lighttpd), for instance (where threads vs. processes is moot).
Moreover, a fair amount of effort has been expended on a complex architecture to push only a very specific type of requests to Unicorn.
That's not what I said; I said: "[Erlang] maps lightweight erlang processes to operating system threads."
Erlang processes are green threads mapped to OS threads. They're called processes in erlang, but they're M:N scheduled threads.
So on a four-core system, you might have one thousands of Erlang lightweight processes running on just four OS threads in one OS process.
Or you could have thousands of Erlang lightweight processes mapped to 185 OS threads (not that it will, but it could). The fact that they're green threads is irrelevant in terms of scalability -- they're still threads, but even less resource intensive than a direct mapping to OS threads. M:N scheduled green threading is also very hard to do in the general case, which is why you don't see more of it (see the abandoned attempts to implement generic M:N OS:green thread scheduling in FreeBSD, for instance).
You could also be using Scala, where an actor only maps to a thread when it's blocked inside of its react loop, but in that case will consume a whole thread.
What's your point? :)
User clones aren't forks, in the traditional sense,
where a fork has little intention to contribute back
to the original project. Rather, the entire purpose
of a user clone is to allow users to contribute to
projects without requiring official commit access
permissions.
I was thinking of migrating some projects to github for this feature, but instead I think I'll just switch to Mercurial on Google Code.That assumes that the tools maximize human capital more than the tools that maximize both human capital and minimize server cost. Do they?
I don't know what you mean by reducing moving parts. The only parts involved in handling the requests for this setup are Nginx and HAProxy. All the other technology parts (Tokyo Cabinet, memcached, Sphinx) are used to scale the database, something your stack still has to achieve.
Lucene, ehcache -- in-process libraries to achieve the same effect but without all the moving parts. Or, instead of lucene, PostgreSQL's built-in FTS.
You also don't need nginx, haproxy, and a complex architectural division between static/non-static content.
The tradeoffs between using bare metal vs VM and using Rails vs Java are well known at this point.
I'm not sure I agree, given the constant repetition of "Ruby is more productive" as a justification for poor performance and complex architecture. Is it more productive than Scala? Clojure? Groovy? What about using even just using JRuby?
You can't assume that the threads are 100% CPU bound. If they were 100% CPU bound, you would have a point -- 8 cores, 8 CPU-bound processes, no processing time left over -- that is where event based and hybrid thread/event-based architectures excel.
In reality, webapp threads will sleep -- waiting on the network, waiting on disk, waiting on the database. When they sleep, the the processor has nothing to do. If the OS can schedule another thread while one sleeps, work can move forward.
If you can run 500 threads to completion in 10ms, then you can serve 500 requests within 10ms.
If you can run 16 (or 32) processes to completion in 10ms, then you can only serve 16 (or 32) requests within 10ms.
1) Whether it can support anywhere near the same level of concurrent requests with similar response times.
2) How much complexity (nginx + unicorn + memcached + puppies) is required to achieve this in comparison to a servlet engine (eg, tomcat) and your webapp.
[1] Most web applications are read-heavy, low on writes, and scaling up write capability generally requires scaling up the database. You can grow quite a bit with simple caching and monolithic database scaling before having to tackle more complex distributed data architectures.
At those numbers, it sounds like you'd max out at roughly 32 workers (so 32 concurrent requests), or am I missing something important?
It's true that running Ruby code is generally more expensive (dollar-wise) per given task than other languages, but we made a conscious decision to make that tradeoff for the productivity gains that Ruby affords us.
It's really easy to measure performance, but productivity claims seem difficult to support and highly susceptible to confirmation bias.
I think that Ruby, Scala, and Clojure (just for instance) would stack up against each other very well in terms of productivity, but I wouldn't know how to prove it.
5-10k, depending; a local instance of a recent webapp I wrote can run 5.9k requests/sec on a simple page JSP-templated page backed by a database request. It's running on tomcat, using servlets with a lightweight REST API, and postgresql.
The webapp I'm working on right now has the following performance profile for a page that fetched the user-list from the backing database. Not as fast, but 2k req/sec is not bad, and I haven't done any profiling on the new stack we're using at all yet:
Server Software: Jetty(6.1.x)
Server Hostname: localhost
Server Port: 8080
Document Path: /users
Document Length: 5855 bytes
Concurrency Level: 4
Time taken for tests: 0.508 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 6045474 bytes
HTML transferred: 5855000 bytes
Requests per second: 1968.41 [#/sec] (mean)
Time per request: 2.032 [ms] (mean)
Time per request: 0.508 [ms] (mean, across all concurrent requests)
Transfer rate: 11621.09 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.1 0 1
Processing: 1 2 1.7 2 22
Waiting: 1 2 1.7 2 22
Total: 1 2 1.7 2 22
Percentage of the requests served within a certain time (ms)
50% 2
66% 2
75% 2
80% 2
90% 3
95% 3
98% 4
99% 12
100% 22 (longest request)
I'd like to give you a job.Something in your tone tells me you're not serious ...
Maybe I've been working with heavyweight frameworks too long, but that seems an order of magnitude off at least.
I couldn't say; I do have local 3rd party spring-based webapps that take 5 minutes just to start up, not to mention 300ms+ to render a single page, so I wouldn't be surprised.
If load levels continue to grow, leverage ehcache to handle caching directly in the VM (if you're not already), begin splitting responsibilities across different service implementations, etc.
Let's be really, really generous and say peak load on 10 million requests is 5 hours, and pack all 10 million requests into 5 hours. That's 2 million requests/hour, 33,333 requests/minute, 555 requests/second.
You can easily handle 5-10k requests a second with a few ms response time -- including dynamic templating and hitting a backend database -- on a 2.66 GHz 4 core Xeon 5150 with a couple gigabytes of RAM, using a servlet runtime and a JVM-based language.
Basically, you could take that entire solution stack and compress it down to a few items with far fewer moving parts, and spend half as much on engineering and capital expenditures.
My hope is that efforts like MacRuby bring this level of free performance to the Ruby world, and that languages like Scala and Clojure make the JVM more attractive, too.
1) "Threads are out -- processes are better than threads."
2) Process-per-connection architectures.
The first is demonstrably false -- for instance, look at Erlang, which maps lightweight erlang processes to operating system threads, providing SMP scalability at a low cost without running into Github's issues with mongrel "thread-killing". More broadly used, look at Servlets and the Servlet 3.0 support for async comet-style event-based request handling. Each request is handled on a thread as necessary. Inter-thread communication (where necessary) is cheap, and this scales just fine.
If you use the fork() model and a long-running request blocks the entire process, comet is basically a non-starter. This is why people are interested (and implement) lightweight threads, coroutines, and restartable request implementations.
For a conceptual challenge, consider how you would implement a live web chat system that can scale up to a considerable number of clients, with extremely low resource usage, and instant message distribution (no polling). Locally, we implemented this with async servlet support and M:N scheduled scala actors -- a blocking HTTP request doesn't hold a thread or process hostage in the web server or the application, and we can scale up to enormous number of live clients on one machine.
The second was merely a lack of understanding of the model. In implementations where fork() is used as an alternative to threads and multiple connections are not handled per sub-process, you quickly run into scaling issues with subprocess memory utilization. In cases where subprocesses handle multiple connections via an event mechanism, you're just using fork() instead of threads.
I'm wondering how many servers github is using to keep up with load -- If each 8 core 16 gigabyte (!!!) server can actually only handle 16 concurrent requests via a pool of 16 workers, that's an incredibly poor (and expensive) scaling model.
I'm not sure it's actually simpler; as a software author, my experience has been:
If I want to release something for the Mac, I post a dmg containing a dragable application; and even the most non-technical user can download and install it immediately. Sparkle provides automatic updates via my website; users gets the latest versions whenever they want them, I don't have to do anything complicated.
If I want to release something for the UNIXes, I provide a set of autoconf scripts and a make-based build system (or one of the recent alternatives, or a Python egg, or a Ruby gem), hope that the package management systems pick it up during their next release cycle, wind up in an argument with Debian over OpenSSL+GPL licensing compatibility, and see my software get shipped with half the featured turned off and a ridiculous number of local hacks applied, including all-too-regularly replacement of a working library dependency with something that doesn't work (but Debian likes it more).
The average end-user doesn't see my updates for months or years, the software is missing features and locally modified, and I have to wonder: Why couldn't I just give them something that they could drag to their applications folder?