spotman

database rule #1, do not use your database as a work queue.

this article does a great job discussing why not, and finishes with a sane implementation that would work, but would fall over and shutter to a halt with a moderate amount of load. ( which is mentioned in the article too )

doing queue workloads in rdbms is a recipe for index contention, bugs, or both.

also not op, but share same thoughts largely.

for me makepkg/PKGBUILD and abs keep me coming back to arch.

tldr: mysql is getting worse for small, in memory workloads with low concurrency.

for the average medium to high traffic mysql install, i find 5.7 to be much faster than 5.6 or mariadb where concurrency and varying workloads prevail.

ymmv, do your own testing to see which db is a good platform for your specific workload. maybe it's mysql 5.6 maybe it's mariadb, maybe it's postgres.

for me mysql 5.7 brings much awaited niceness, saner defaults and even though not all workloads are better on it i think oracle is really trying to iterate. likely they will fix this regression and there would be little reason to use it over 5.6 or older.

Some systems do up to 1mbps. MIDI for example is nowhere near this. It's not the right protocol for media or bandwidth intensive applications but for control systems or reading sensors its often enough.

Disclaimer: I work on this project w/ the author.

The author writes about how he used this system to solve a case of a slow query. However, this is really sort of the wrong query to illustrate why we built this. It's more of a "oh hey look since we already have this data super fast in this other system, lets just use it."

What the system does is allow us to arbitrarily summarize at whatever granularity we want, any second, to any second.

For example we can ask it to summarize January 4th at 11:22:02 to March 1st at 16:22:09, and so however many writes happened during that time at any second are tallied up very quick, because of the way the data is stored. So even though lets say during this 3 month~ period you have 10,000,000 writes for one user some that say "add 3" or "remove 4", you will only query at a second granularity around the edges, and the middle you will query whole day, or month buckets.

Another way of explaining the point of this system is that it allows us to really quickly (well under 1ms) summarize any data in the past N days (N usually equals 120) from any start to any stop point we want, using any granularity we want, and we can do it concurrently and generate this for 1000 users at a time and it all can happen in 30-50ms. Considering that the underlying data is recorded at second granularity, and can be retrieved at that granularity and automatically expired when this granularity is not needed, redis is the right tool for the job here. To accomplish this in pure sql you would have to log all writes you make to the table, and query this change logging table in a similar way. So not only would this add a lot of overhead to the database as a whole, but a lot of churn to remove the lower granularity when its not needed.

We will be polishing the system more over time, and more blog posts will come out when we release some of the client code ;)

CAN Bus is great. I highly recommend reading the CAN Open Spec if your going to be implementing new systems using CAN Bus.

https://en.wikipedia.org/wiki/CANopen

Many more things besides auto's use CAN. One of my current projects involves exercise equipment that internally speaks CAN Bus. There is great support in Arduino for interfacing with stuff like this.

Imagine that! Working backups! nice job not losing data digital ocean.

From my experience, I have found that a properly designed schema in an RDBMS is going to outperform a graph database for OLTP style queries _if_ your graph traversal is light.

So, if you just need to build a system that finds similarities and you only need to look at a level or two of relationships, its going to be more simple to use an RDBMS, possibly.

But, if you need to do a lot of graph traversing, this is where and RDBMS system is going to get tricky.

One caveat with graph databases is that there is different kinds with different strengths. Some are fast and in-memory, some are distributed and meant more for OLAP. Some are distributed with reads, and are meant for OLTP but are slow for writes.

While there is similar tradeoffs with different RDBMS systems, if your use case is to get something up and running, I would start with an RDBMS and keep it simple until you can't.

Finally, your final paragraph may be one path forward, start with an RDBMS, and if you can't leverage to do what you need quickly but its working for most of your use case, you can extract the relationships to a graph database and use it alongside.

If you give up, yes your screwed. Don't give up if you want this career to work.

Some times things can be discouraging in this industry, so look at this as training for in the future when you have to get through a discouraging or trying time.

Sigh. Some instances can not get to level 3 at all. Go go gadget google cloud!

Having said that Kudos that this is on their status page!

This has a feel of it solved a race condition due to scheduling something on a few ticks later. So it might accomplish not only running something on the main queue but also sneakily after something else finished. For example if you call something like this at the end of view did load, other code that also runs on the main thread may run before it. It's async.

Having said that, it kind of feels hacky. I have certainly seen this in various codebases and I can't comment on react native but just with standard iOS programming it often points to a lack of understanding of lifecycle or similar order of when things are happening.

It's hard to know where it's going and concurrently learn the right pieces of that thing in the right time to not be late or early to the party...

So, learn the thing you can either learn the deepest or the fastest, or both... because nothing helps learning stick than being fascinated or having fun with it.

Sometimes you can learn on the side and sometimes it's healthy to bite off just a little more than you can chew if you are determined and learn some parts of it in real time.

Personally I think go, swift, javascript/react are going to be around for awhile and see many new projects in these. But I think this is on a rolling cycle; every 5 years seems like so many changes. Just seems like yesterday that rails was the coolest thing since sliced bread. Then there is java which just is everywhere. Rust is also very interesting, and seems to have a great community, job wise not many in comparison to either rails or js.

Finally however elixir seems cool. And slightly railsy. That might be next on my list.

With many modern cars, especially luxury ones, they come with a way to put a sim in it, so that they can provide you with ways to locate your own car (to remember where you parked), provide a wifi hotspot, integrate google maps, among other things.

The fact your even asking this means your ahead of a lot of your peers.

None of this comes overnight. Many of us start out incredibly messy, and there is nothing wrong with that.

In fact it might give you an edge over folks that start out very elegant and by-the-book because you are used to dealing with things like race conditions and they expect none.

Remember this, the best code is never written the best it can be the first time around, and in a lot of cases never.

Code is a moving target, stay light on you feet, avoid religion of certain methodology ( to an extent ) and paddle towards things that you are proud of.

You will be fine and likely do great if you stick with it.

Finally, there will always, always be someone who knows more to you. There is no end to what you don't know. The sooner you make peace with this the better!

Yes, you will need an SOW, (statement of work) that articulates very clearly what you aim to accomplish. Near the latter part you should have another section ( there is more than one way to accomplish this however ) that clearly states what the "acceptance criteria" is for meeting the goals - ie - spells out usually in bullet points reachable goals for the project that when met everyone agrees your end of the deal is met.

Along with this it should also cover your rates, payment schedule, terms, and what things cost if they fall out of scope.

Finally I find it useful to add a "prerequisites", and "disclaimers" section that say that work can't begin until you have things like write access to their git repo or access to their aws account ( or whatever ) and that the current SOW doesn't cover getting it into the App Store or bugs that come after the acceptance criteria are met.

Put places on this document for your company or self, and the other party and make sure you properly distribute and record the signed docs.

IANAL but have one, and the more you get into consulting and bigger contracts it's good to run things by a lawyer if it's something that's high value, with a lot of effort, or even remotely puts you at risk.

Be even more careful signing documents clients give you;)

Have fun

Except for Peter Theil, one would guess. Heh.

Some of the Blairs brand of hot sauces claim up to 16 million on the scoville scale. I have tried the 4am sauce which claims 4M on the scoville scale, and it was not a laughing matter immediately, more crying, then laughing days later. (I love this stuff though, and would do it again!)

Having said that the stuff in the millions is pretty darn hard to find, they don't stock it or even sell it regularly.

Info: http://www.ushotstuff.com/hotSauceHeatScale.htm

My point is that if they are advertising rock solid distributed data, I don't think they have achieved that without putting it through some more paces.

I agree, actual numbers are great.

We also agree, jespen is great.

But I am saying, another path, to a similar level of confidence jespen provides, is some real critical stress testing.

Sure, it will be out of the scope of what most people need, but it will give them the confidence to say "rock solid distributed data".

2 billion writes over 8 years? Cool if so, but many database systems both MySQL, and others are capable of getting this in a day. I know this because I maintain some high availability MySQL systems that see about 4 billion writes per day across a similar amount of hardware.

Have you run this through Jespen or done any actual load testing or deep testing for failures related to machines dying, network partitions or other?

I encourage you to do so, because (not trying to sound rude) this kind of reads like "hey my homemade car has 25,000 miles on it and I still use it". But you can build a performance and failure testing framework to really put it through its paces.

Anyhow , have fun and good luck~~

This is the story of many "enterprise" companies sadly. Some I work for, and while they are slowly changing, I would not say their own data centers (yes, read: data centers) are "little co-located hosting" companies. Usually they are entire departments, with entire budgets and multiple facilities, with many peoples jobs within that.

So while for the developer, moving things to AWS is a no-brainer, a time-saver, and a money-saver to the company, the amount of politics, and change, is so large these behemoths of companies are the last to consider it.

Security is a valid piece, but also a political move to keep the money from changing hands too rapidly.

Here in Los Angeles some locations that were less busy are now never a long wait and can even appear desolate at times.

Ones that were super duper line-out-the-door popular almost never have the line out the door anymore it seems.

I don't really eat here hardly ever, but walk by 2 almost every day.

Still having said that they are still routinely busier than some other options near them.

I agree with your sentiment, but another piece of the puzzle here should be considered:

Playing fast and loose with health care, that may be key to determining whether you have some life-threatening disease, is a big deal.

This is not like taking some VC money for the next social network, and having no chops to pull it off. Peoples lives could come into play.

Take a job, or a gig, or a ticket that is beyond your skill level, and don't give up until you solve it.

Fair game sure, but unfortunately confusing. If there was another dart that was an actual programming language ( Is there ? ) you would have to qualify every time which dart programming language you meant. I imagine searching for things online would be tricky.

Performance is great, but what is more interesting is maintaining that performance in the midst of failures and things going wrong.

I don't know anyone who chooses InnoDB and says "gee I wish it were faster". While it isn't the fastest show in town, it is a known quantity, and how it breaks is well understood.

So if I was going to use a different storage engine, would want something more than its a bit faster, in some cases.

Having said that, what a wonderful amount of work, and don't stop hacking on it!

Four thousand dollars to ride a single-wheeled skateboard, marketed as a hover board, for a total of 16 minutes, seems like a scam.

Shouldn't a hover-board, hover?

Having said that, with a small marketing and price adjustment (with a large impact), I could see it being a neat form of transportation.

Can you elaborate on the cognitive overhead vs a GC language like golang or java? (With regards to memory)

To me sure there is things you think about but generally memory performance is less of a mental burden.

Also, swift 4 is supposed to support a rust like memory model. Not sure if it's opt in or how it will work but my impression is that it's memory model is about to support some new use cases to make it appropriate for more memory critical tasks.

I don't work at google but have many friends that do. (so take this with a grain of salt)

But just because they may put all their code and commits in a giant common repository, does not mean that its hosted, scaled, and deployed as a monolith in current day.

From the grapevine, I know that some of their stack is c++, and some is java, as an example.

Also, google losing all their code, made me laugh. Maybe that is a possibility, but seems like they have to have so many backups and failsafes. But I get you said that to illustrate a point :)

You say:

> By making services depend on each other through a protocol like REST (or Thrift, gRPC or whatever), you can then replace that module by another one that respects the same interface... and you can reimplement it in whatever technology you want. I can have a module implemented on Elixir running on FreeBSD talking to another one running on OpenBSD implemented on C, talking to another one running on Linux implemented in node.js...

I said:

> It is compartmentalization between teams, and honoring stable internal functionality between teams that it makes the biggest difference.

Maybe we agree more than disagree?

You also say: > Similar to SOA, but SOA was much broader in scope. You had service composability, discoverability, etc. That's in most cases overkill for most companies.

Not sure if the normal way of thinking of micro-services does not include this. If so, that is a new definition to me. To me micro-services mean services which perform less functionality, therefore you need more of them. If you have a platform that has 40 functions-as-a-service running in docker containers, you will need discoverability, for example.

Oh microservices. Here we go again.

Yes, it's easy to get burned. Yes, it's often the absolute wrong tool for the job as a young startup. Further, its all too easy to just light cash on fire once you get some real serious traffic hitting all these things that may autoscale individually. We have seen it all before, and we will see it again.

But don't throw the baby out with the bath water. In fact, I think the healthiest approach is to drop the "micro" and think of classic Service Oriented Architecture (SOA).

When you hire your 50th engineer, and you don't have this yet, its not the end of the world, but slowly moving from a monolith to a SOA, is pretty much a very well travelled path, and at a certain scale is how you maintain sanity.

It's like the old saying, the first code you write is never the best. Try the 2nd or 3rd rewrite, and you finally know what you want. So start with a monolith, but don't back yourself into a corner, and move one obvious service out at time.

The main benefits are not even what the article gripes about. It is compartmentalization between teams, and honoring stable internal functionality between teams that it makes the biggest difference. Sure scaling CAN be easier, but it can also be harder. Things like managing connection pooling becomes easier with an SOA. But not to the tune of doing it concurrently across 40 services, that is irresponsible to design without knowing exactly how they are all going to get hit when you finally get insane traffic.

One thing the author said that I agree should be more obvious to people is that somewhere in-between hiring engineer 12 and 50, you will end up running your monolith a bit like an SOA, where even if its the same codebase, you can run in different clusters/pools and service a specific part of the application. This is a wonderful way to get the benefits of SOA, with out the nightmare, furthermore, you will quickly learn which of the parts of code, when your rewriting it the 2nd time, you actually want to divorce from the monolith, and you already have patterns, load balancers, and metrics for what that code should do.

When you hire your 200th engineer, it's easy to get on HN and say SOA is the only way to get the job done. But people need to realize the importance of the journey of getting there. Otherwise your just taking a shotgun to the skateboard your cruising down the road with.

Sanity is the key takeaway. Always optimize for sanity.