HNHacker News
TopNewBestAskShowJobs

jondot

2,256 karma · joined August 4, 2011

http://paracode.com
submissionscomments
jondot··on Crunch – Go-based toolkit for ETL and feature extraction on Hadoop
Actually yes! Because I'm using Pig to "abstract" I/O you can use Pig to load data from anywhere (any format), it streams it into Crunch, and then again, Pig to save data to anywhere (any format).
jondot··on Introducing Raspberry Pi B+
No :(
jondot··on Hybrid apps: Do not copy. Do get inspired. Avoid reimplementing native UI
Incidentally i spend the last 2 weeks going through various Android devices, and iOS devices, various javascript frameworks and various CSS frameworks.

I'm happy to see others having opinions on this subject.

Essentially i did the research for these combinations:

[Android devices]x[iOS devices]x[js frameworks]x[CSS frameworks]

It was hard and tiresome but the conclusion - same as yours - iOS works very well for cordova/phonegap. Android is a no-go!.

Among the frameworks I checked: backbone, ember, angular, ionic, topcoat and various other CSS frameworks.

All were slow on Android. Also discovered 4.4 introduces worse performance (acknowledged by google) which is solved with crosswalk. However crosswalk brings many bugs to the table itself.

Really disappointing - currently I'm opting out and going Native.

jondot··on Show HN: Hacker News Redesign
fixed, sorry :)
jondot··on Show HN: Hacker News Redesign
Its a good question - what are the habits of a HN reader?
jondot··on Show HN: Hacker News Redesign
Thanks, the original post is my friends, I just called her to come online and give back explanations / background if anything would like to know more
jondot··on Scala 2.12 Roadmap
Hi, thanks for taking the time to reply. My personal opinion of SBT is one that represent almost every developer I've discussed it with. Jumping to the bottom line - an experience like: rubygems, npm, lein (clojure), and that kind of build and package family is much preferred - one example is Buildr.

It's hard to pinpoint a developer 'instinct', but I think a build/dependency tool like rubygems or npm is one of the first things a developer bumps into, and becomes a central part of the experience _very early_, even before the experience of the language. With that in mind, SBT present a cryptic (some times even non-scala) like feel to it.

jondot··on Scala 2.12 Roadmap
Honest question: with tooling (IDE) and build tools (SBT) to my understanding remaining as-is despite of developer feedback, and Java 8 presenting itself as a nice comeback (in my opinion) - is Scala adoption for big companies (Facebook, Google, Twitter) still going up as it were in the high times of it?
jondot··on OpenRoss – fast, scalable, on-demand image resizer
But the nice thing is - using a storage solution where you don't pay for what you don't use.
jondot··on OpenRoss – fast, scalable, on-demand image resizer
As a healthy feedback - the correct architecture sits right below your nose, since you already have all of the components.

You should pre-process images in your scraping cycles, and not when a client comes to request it.

In this way, your "scale" is always predefined, bounded, expected, and much smaller - defined by your scraping scale and not user scale.

Good luck!

jondot··on Bit.ly's shady business model
Sounds like your mistake. If your business relies on link shortening, you should identify it early in the process and build your own.

I don't use personally and every now and then I examine the reasoning of delegating parts of my products to 3rd party providers.

jondot··on Developers cannot upload new APKs to Google Play for the last 22 hours
Google issue https://code.google.com/p/android/issues/detail?id=67226
jondot··on Grunt and RequireJS are out, it's all about Gulp and Browserify
And again I'll remind people what happened in Java land:

ANT -> Ivy/Maven -> Gradle

I'll state it clearly: Grunt. is. Ant. It's a mess to follow a build script and a ritual to make it work in a real life project.

I expected a tool like gulp to come sweeping in and it did, I'm extremely happy about that and started migrating away from the horrible tooling that is Grunt.

I never understood how people can tolerate Grunt. Long live gulp and common sense!.

jondot··on Show HN: Sneakers – Fast background processing for Ruby
I understand. You're completely right about Sidekiq.

For RabbitMQ - you can have jobs persisted to disk on all nodes in the cluster which is like a backup. And of course transactional messaging is a problem that needs special attention.

jondot··on Show HN: Sneakers – Fast background processing for Ruby
Right now Redis cannot be clustered production-ready. I wish. As I stated in the Wiki, you'll have to pry Redis from my dead body, I am very happy with it, and for me its a true swiss army knife and I've used it as such.

Even though it doesn't have clustering - it's rock solid in production and I haven't experienced a drop in one of my Redis servers in around 3 years.

That being said, if you are building a system where reliability is an explicit requirement you can't take those risks.

jondot··on Show HN: Sneakers – Fast background processing for Ruby
More or less the same. Plus it will use all cores on MRI, so you can keep using gems with C-Extensions (i.e. you don't have to run rbx or JRuby to max all cores).

Discussing reliability - this is something that sadly Sidekiq will never give you, by virtue of the fact that it uses Redis. RabbitMQ can be clustered in active-active mode, which means you have won over reliability here by just using a cluster.

When comparing queue systems, comparing Sidekiq+Redis to RabbitMQ is a bit unfair - because RabbitMQ was born to do this. And that's why if you're doing proper background jobs and messaging it's better to pick the right tool.

That being said, I do keep using Sidekiq for small Rails apps for the typical background emailers, denormalizers, etc. But I keep an eye open for when I realize that I'm doing proper messaging - in which case I'll switch over to something like Sneakers.

jondot··on Show HN: Sneakers – Fast background processing for Ruby
I can definitely say that RabbitMQ (and the awesome bunny ruby library) is doing the heavy lifting.

Batch handling - is actually "prefetch" in RabbitMQ/AMQP. Error reporting - via logging, and a "dead letter mailbox" (called a dead-letter-exchange in RabbitMQ) which is a great enterprise integration pattern for properly handling errors and retries in jobs.

RabbitMQ does the heavy lifting in another surprising facet - the UI. The management UI is excellent and at some point in time I decided not to compete with it by creating my own UI (you have to pick your fights :).

Here's some more info about the management plugin for RabbitMQ.

http://www.rabbitmq.com/management.html

jondot··on Show HN: Sneakers – Fast background processing for Ruby
Sorry about that :)
jondot··on Show HN: Sneakers – Fast background processing for Ruby
tdumitrescu "auto-scaling" is exactly what Unicorn gives you. You can scale up or down a running pack of worker processes by sending signals (kill -USRX) to the supervisor.

The sad news is, I've gotten some feedback that I believe may be true - self-daemonizing processes is a bad practice and that we should let the OS handle daemonization. And this kind of autoscaling is a bad practice in of itself because of it. This is why I've started to deprecate this feature (passively by just including a notice for now).

The question between number of processes and number of threads is excellent. It is mostly based on the workload - and the good news is that it's all scientific. You first need to understand the peak job run time (always try to upper-bound your jobs with timeouts) which can be had by some trial runs.

If it takes 200ms per job (I/O bound), it means each thread can do 5 units of work per second. If you need 1000 jobs/sec - you need around 200 at worst of these little guys to do work. Now, you can divide those into 4 processes on a dual-core machine (2 per core is a good rule of thumb). You end up with 50 threads per worker which is pretty relaxed.

The punchline is - if you need 1000req/s - with Sneakers the question isn't "can the broker support 1000req/s" anymore, because RabbitMQ should virtually look down and laugh at those numbers :).

jondot··on Show HN: Sneakers – Fast background processing for Ruby
djur - thanks :)

Yes, you nailed it. For MRI I had a bit of a different challenge. I already solved this problem a year and a half ago, and with the benefit of being able to use JRuby performance was a bit easier to reach (by dropping to "bare" Java amqp driver and Executors) - https://github.com/jondot/frenzy_bunnies

jondot··on Show HN: Sneakers – Fast background processing for Ruby
Thanks danso! I'm happy that you find it useful. I also hope to integrate conclusions from relevant discussion here back into the Wiki.
jondot··on Show HN: Sneakers – Fast background processing for Ruby
Not hard because most things are abstracted (though - there's a trade off between too much abstraction and peformance).

But I feel it might be missing the point, RabbitMQ was used to remove the typical bottleneck from the broker which exists on most Ruby background frameworks, and using Postgres would bring it back. All in all, RabbitMQ will give you a good, transparent HA story with active-active which is useful when you must not lose messages (Postgres will give you just failover).

Edit: to cover @artellectual's response - I think OP meant to swap RabbitMQ with Postgres as the backend but if OP meant "is it possible to use existing Rails models / etc" - then yes, @artellectual is completely right - easy to do.

jondot··on Show HN: Sneakers – Fast background processing for Ruby
Thanks man :)
jondot··on Show HN: Sneakers – Fast background processing for Ruby
Yes. I dislike reinventing the wheel, I actually bumped my head against making Sidekiq use a different backend.

Three problems - one - it is too tightly coupled to Redis. Secondly, Celluloid (the actor model behind Sidekiq) proved to be some of the problem here. Lastly, I wanted both a process and thread processing model which doesn't exist there.

Then, I tried rigging Celluloid to perform all these but after failing and speaking with Celluloid commiters I understood that it is under "major revision" and that they are not happy with some of the core elements right now (threadpool).

THEN, I opted to reinvent the wheel. Even here, I've chosen serverengine as the basic process management infrastructure to work on - a production tested core, to build Sneakers on.

jondot··on Show HN: Sneakers – Fast background processing for Ruby
Author here - Hi! :)

Sneakers is a high-performance background-job processing framework based on RabbitMQ.

It uses a hybrid process-thread model where many processes are spawned (like Unicorn) and many threads are used per process (like Puma), so all your cores max out and you have best of both worlds.

It's being used in production for I/O intensive jobs as well as CPU intensive jobs.

On a recent 2012 MBP it reaches 7000req/s for a silly microbenchmark while Sidekiq keeps at the hundreds (600-700req/s).

jondot··on Linux Mint 16 "Petra" MATE released
Naive question: visually, this looks like all Mints before it. As an Ubuntu and OSX user, I'm not sure I'm liking the 'retro' Linux UI style.

For a distro like Mint or Ubuntu to hire a professional design team for the OS to provide an aesthetic and holistic UX comparable to OSX / Android - is the stopper just budget or is there something technical in X/Mir that stops that from happening?

jondot··on Scaling Scala vs Java
I actually spent a part of the weekend re-evaluating Scala build tools for the who-knows-how-many times now.

Gradle did come up and I was astonished to discover Groovy is now over its performance problems and may very well be a worthy platform to develop web services on!. Only if Groovy had a clear road map and profound backing it would have been the true "Scala killer".

I think it's worth giving Groovy a second chance by its own, and I've set up some time to do that.

And lastly - after looking at gradle as a build tool for Scala - it looks OK, however I don't think I will ever withstand all of the hate and incompatibility from the Scala community because I've deviated from SBT (which is another problem by itself).

jondot··on Scaling Scala vs Java
Every now and then comes such an article and nudges me towards Scala again. It can really be an elegant and succinct language that can be all-encompassing.

But although I have a production running Scala project, I cringe every time I have to work with Scala just because of the tooling. It's sad to see that SBT is cryptic and IDEs are slow and misleading.

When I work on the JVM I prefer Clojure. I think lein makes an exemplary development, build, and dependency management tool. I wish Scala had that.

jondot··on Ask HN: What makes you go for it?
Hey Tam, I'm exactly the same. Maybe we can form a support group for such people (i'm serious)?
jondot··on Goodbye node-forever, Hello PM2
Nice work!.

This is a very good addition - I'm using forever extensively in production over many services for the last year and a half.

While it's great, it has a few corner cases which looks like you guys also bumped into and solved.

Can't wait to give this a go.

← PreviousPage 3 of 7Next →