2,256 karma · joined August 4, 2011
I'm happy to see others having opinions on this subject.
Essentially i did the research for these combinations:
[Android devices]x[iOS devices]x[js frameworks]x[CSS frameworks]
It was hard and tiresome but the conclusion - same as yours - iOS works very well for cordova/phonegap. Android is a no-go!.
Among the frameworks I checked: backbone, ember, angular, ionic, topcoat and various other CSS frameworks.
All were slow on Android. Also discovered 4.4 introduces worse performance (acknowledged by google) which is solved with crosswalk. However crosswalk brings many bugs to the table itself.
Really disappointing - currently I'm opting out and going Native.
It's hard to pinpoint a developer 'instinct', but I think a build/dependency tool like rubygems or npm is one of the first things a developer bumps into, and becomes a central part of the experience _very early_, even before the experience of the language. With that in mind, SBT present a cryptic (some times even non-scala) like feel to it.
You should pre-process images in your scraping cycles, and not when a client comes to request it.
In this way, your "scale" is always predefined, bounded, expected, and much smaller - defined by your scraping scale and not user scale.
Good luck!
I don't use personally and every now and then I examine the reasoning of delegating parts of my products to 3rd party providers.
ANT -> Ivy/Maven -> Gradle
I'll state it clearly: Grunt. is. Ant. It's a mess to follow a build script and a ritual to make it work in a real life project.
I expected a tool like gulp to come sweeping in and it did, I'm extremely happy about that and started migrating away from the horrible tooling that is Grunt.
I never understood how people can tolerate Grunt. Long live gulp and common sense!.
For RabbitMQ - you can have jobs persisted to disk on all nodes in the cluster which is like a backup. And of course transactional messaging is a problem that needs special attention.
Even though it doesn't have clustering - it's rock solid in production and I haven't experienced a drop in one of my Redis servers in around 3 years.
That being said, if you are building a system where reliability is an explicit requirement you can't take those risks.
Discussing reliability - this is something that sadly Sidekiq will never give you, by virtue of the fact that it uses Redis. RabbitMQ can be clustered in active-active mode, which means you have won over reliability here by just using a cluster.
When comparing queue systems, comparing Sidekiq+Redis to RabbitMQ is a bit unfair - because RabbitMQ was born to do this. And that's why if you're doing proper background jobs and messaging it's better to pick the right tool.
That being said, I do keep using Sidekiq for small Rails apps for the typical background emailers, denormalizers, etc. But I keep an eye open for when I realize that I'm doing proper messaging - in which case I'll switch over to something like Sneakers.
Batch handling - is actually "prefetch" in RabbitMQ/AMQP. Error reporting - via logging, and a "dead letter mailbox" (called a dead-letter-exchange in RabbitMQ) which is a great enterprise integration pattern for properly handling errors and retries in jobs.
RabbitMQ does the heavy lifting in another surprising facet - the UI. The management UI is excellent and at some point in time I decided not to compete with it by creating my own UI (you have to pick your fights :).
Here's some more info about the management plugin for RabbitMQ.
The sad news is, I've gotten some feedback that I believe may be true - self-daemonizing processes is a bad practice and that we should let the OS handle daemonization. And this kind of autoscaling is a bad practice in of itself because of it. This is why I've started to deprecate this feature (passively by just including a notice for now).
The question between number of processes and number of threads is excellent. It is mostly based on the workload - and the good news is that it's all scientific. You first need to understand the peak job run time (always try to upper-bound your jobs with timeouts) which can be had by some trial runs.
If it takes 200ms per job (I/O bound), it means each thread can do 5 units of work per second. If you need 1000 jobs/sec - you need around 200 at worst of these little guys to do work. Now, you can divide those into 4 processes on a dual-core machine (2 per core is a good rule of thumb). You end up with 50 threads per worker which is pretty relaxed.
The punchline is - if you need 1000req/s - with Sneakers the question isn't "can the broker support 1000req/s" anymore, because RabbitMQ should virtually look down and laugh at those numbers :).
Yes, you nailed it. For MRI I had a bit of a different challenge. I already solved this problem a year and a half ago, and with the benefit of being able to use JRuby performance was a bit easier to reach (by dropping to "bare" Java amqp driver and Executors) - https://github.com/jondot/frenzy_bunnies
But I feel it might be missing the point, RabbitMQ was used to remove the typical bottleneck from the broker which exists on most Ruby background frameworks, and using Postgres would bring it back. All in all, RabbitMQ will give you a good, transparent HA story with active-active which is useful when you must not lose messages (Postgres will give you just failover).
Edit: to cover @artellectual's response - I think OP meant to swap RabbitMQ with Postgres as the backend but if OP meant "is it possible to use existing Rails models / etc" - then yes, @artellectual is completely right - easy to do.
Three problems - one - it is too tightly coupled to Redis. Secondly, Celluloid (the actor model behind Sidekiq) proved to be some of the problem here. Lastly, I wanted both a process and thread processing model which doesn't exist there.
Then, I tried rigging Celluloid to perform all these but after failing and speaking with Celluloid commiters I understood that it is under "major revision" and that they are not happy with some of the core elements right now (threadpool).
THEN, I opted to reinvent the wheel. Even here, I've chosen serverengine as the basic process management infrastructure to work on - a production tested core, to build Sneakers on.
Sneakers is a high-performance background-job processing framework based on RabbitMQ.
It uses a hybrid process-thread model where many processes are spawned (like Unicorn) and many threads are used per process (like Puma), so all your cores max out and you have best of both worlds.
It's being used in production for I/O intensive jobs as well as CPU intensive jobs.
On a recent 2012 MBP it reaches 7000req/s for a silly microbenchmark while Sidekiq keeps at the hundreds (600-700req/s).
For a distro like Mint or Ubuntu to hire a professional design team for the OS to provide an aesthetic and holistic UX comparable to OSX / Android - is the stopper just budget or is there something technical in X/Mir that stops that from happening?
Gradle did come up and I was astonished to discover Groovy is now over its performance problems and may very well be a worthy platform to develop web services on!. Only if Groovy had a clear road map and profound backing it would have been the true "Scala killer".
I think it's worth giving Groovy a second chance by its own, and I've set up some time to do that.
And lastly - after looking at gradle as a build tool for Scala - it looks OK, however I don't think I will ever withstand all of the hate and incompatibility from the Scala community because I've deviated from SBT (which is another problem by itself).
But although I have a production running Scala project, I cringe every time I have to work with Scala just because of the tooling. It's sad to see that SBT is cryptic and IDEs are slow and misleading.
When I work on the JVM I prefer Clojure. I think lein makes an exemplary development, build, and dependency management tool. I wish Scala had that.
This is a very good addition - I'm using forever extensively in production over many services for the last year and a half.
While it's great, it has a few corner cases which looks like you guys also bumped into and solved.
Can't wait to give this a go.