spotman

When we see things like instagram, snapchat, spotify, sound cloud, uber, mobile banking apps, IoT apps, and more becoming native, then this narrative is more interesting.

Just with the IoT stuff, and the fact that many of these apps use cameras, bluetooth, wifi, gps sensors, and more, I think that this is still years away, and by then, what else will native SDKs be capable of that browsers have not caught up.

Browser tech is just getting far enough along where there is things like offline data that works good, but it hasn't caught up yet. I think "app-ocalypse" is a little too strong of a scare statement in my humble opinion.

The article mentions google inbox as an app that shares code, but if I am not mistaken, this is still an app store app, so while they may be able to dynamically change how the app works server side, they still are in the app store, billing it as a native app, so this seems to work against an app-ocalypse.

Agreed. This is one of those areas that most mobile developers are better off with a little less rope to hang themselves.

The expression hurry up and wait comes to mind. I would rather have a thread sleeping while it waits for slightly less perceptual performance than a thread spinning and chewing through my battery because it's waiting as fast as it can.

One way to think about this that maybe overlaps with the theme of this article, but also stands on its own with specific regard to database modifications is that you often need to have the ability to have multiple versions of everything alive at one point in time.

So maybe that's a schema A and a schema B, or maybe you have applied schema B which only app version 1.1 is optimized for , but version 1.0 is what is in production immediately following your database migration. So you can't make changes in schema B that would immediately render app 1.0 broken, which means you need to not box yourself into a corner with future assumptions as much as possible.

Ultimately if downtime is not an option you end up writing these capabilities in at every layer. Whether it's an api endpoint or code talking to a database you often have to make carefully thought out changes incrementally to ensure that things can all operate simultaneously, and often this ends up having metadata about the versions of everything as an option for taking different code paths.

This article touches on this in the way that if suggests making your app deal with an available database and one that is not available. Same with a field in a schema or a payload. To make your code less brittle instruct it what to do in both cases.

Cool, makes more sense now.

At a high level it would be interesting to make it as durable as possible in situations where say F=2 but you have 30 nodes.

As I understand it, in this hypothetical case, the majority of the system would work fine if 3 nodes fail, but there could also be some percentage of transactions which are unfulfillable due to this scenario while there should be many that can still be answered with confidence. ( if I'm understanding correctly that data does not go into every node with a configuration like this and is consistently hashed across )

So was just curious how a prolonged period of this may turn into an unbounded problem when trying to digest your page on it.

Ultimately it sounds like it's headed in a good path and embracing this partial-fail-but-that-doesn't-mean-show-is-over with specifics is a good thing to get ironed out and would certainly be a key thing to get right early, even if it's largely semantics and client behavior.

Good luck, will try to play with it soon

Thanks, that answers my question.

Considering more than one exact txn I imagine will hit a single specific node often, at large scale with a single mode down even if that means 5% of transactions block, you are basically growing a queue of waiting work indefinitely with the only upper bound being how much ram you have. Meanwhile 5% of clients will be waiting and this node may take awhile to come back if it needs something.

Once your out of ram / etc constraints the 5% of the system that is not functioning turns into 100% real fast because your capacity to handle the other 95% takes ram or other resources you now have dedicated to waiting.

If what your saying is also each client can only have one blocked transaction that is relevant but doesn't prevent a consumer to spin up as many clients in succession trying to get through.

I would suggest that you have at minimum strict timeouts for the the transactions, in conjunction with an immediate fail if the node is answer is not available right now. So a client would never wait more than X seconds or the transaction is aborted and if necessary rolled back.

What this would create is a system with predictable failure cases when things go pear shaped. You could calculate in advance how much overhead it would add when something goes wrong as you can have a determinant time factor when having clients wait instead of indefinitely.

Furthermore , what if a node never comes back. Somehow there seems to need a transaction failure that is handed back to the client whether it's node is down, node is gone forever , or node simply timed out.

At the end of the day even if your system is able to handle N thousands of transactions pleasantly waiting and can still answer other requests indefinitely that is a great accomplishment, but in practice may not be ideal for many workloads. People and computers tend to both retry interactions with data that are slow or failed and the combination of just taking on more and more work and hoping everything flushes out when things become healthy is a recipe for a thundering herd, and better served by something like an async work queue type of pattern.

Btw I say this w/o looking at your code , just home page, so possible these things exist and it's not clear what the bounds and failure cases are yet.

Keep on hacking on it!

It helps but doesn't quite answer my question yet.

With your situation #1, what if this is very common transaction and therefore you have 100 of these all waiting. What about 1000, 5000 etc. what system resources are used to let these transactions wait indefinitely ( if I understand your semantics with specific regard to blocking )?

Some systems handle this as a failure that is communicated to the client rapidly. Other systems let N clients actually wait indefinitely but at the cost of taking up a thread / file descriptor, etc. in systems that have finite amounts of threads for example this would then be communicated in his paradigm as a such of upper bounds as to how many requests one could have waiting.

So just trying to get a feeling for how this could have infinite amount of waiting transactions due to partial failure and still keep taking requests.

Thanks for the reply, this stuff is always interesting

> The trade-off is that in the case of failure (node failure or network partitions), operations may block until the failures have been resolved.

Curious how this works at scale. For example if a node is down and N requests start blocking, is there an upper bound to N? What happens when this is reached? Does N map to a goroutine? A file descriptor? Is there timeouts?

Seems like possibly a node going down essentially means the entire show going down if requests are filling up and not being resolved, you would think there may be a tipping point in which other requests may not be able to be served as all resources are blocked.

Possibly there is just a sane timeout and these requests also just fail, leaving the client to know something is amiss?

This is one of the most important points in the comments here.

In LA opening a restaurant with only 50k is irresponsible mathematically. I say this having taken restaurant management courses with the goal of one day opening one myself.

It is true that restaurants are one of the most difficult businesses to start. It takes time to find your groove, and 50k no matter what neighborhood your in is not a lot of time. Most people in this situation would start with a food truck or save more.

Seems like a net positive to notice some attempt that likely has little/less/no malice, that may aid you in preparing to increase your security to be more prepared for when someone that actually has malice does the same thing.

Just want to say, I hear you and in no way am I a fan of the drug war. I should have read more about toxicity levels before commenting.

The reason my initial gut felt this way is not due to any documentary or scare tactic it's purely personal experience.

I have two close friends that have taken the recreation too far. One now has permanent bladder issues and the other was lucky to get their habit in check in time. So my gut reaction was not very scientific, apologies.

I am a huge fan of legalization and science. Let people make their own mistakes and if there is a medical path that helps people I'm all for it!

Just hope that this is also part of the discussion ( harm prevention ) because there is some real risks with abuse, that personally I have witnessed first hand.

Having said that science over fear for sure, I am a fan of science over fear. I still feel more positive light than negative and am hopeful people use his for good!

This makes me nervous. Ketamine has major issues with bladder damage when taken repetitively.

Possibly with a low enough dosage this would be manageable, but speaking from experience from watching people that have been on the wrong end of the stick with this one. it can do some really serious long term damage in that department.

This is newer and shinier, and not all use cases overlap, but at the end of the day, yes its someone else managing the execution of your processes.

Having worked on GAE in a very large scale, my gut thinks that while this has many different use cases, it will make sense for small to medium size projects, and there will be a tipping point surrounding cost and this will appear less like a silver bullet after some horror stories.

At small scale this may even save money, if you truly don't need your code running all the time, and can deal with latency of starting your code. For projects that think terms of N thousands of transactions per second, the math breaks down real badly. It would cost approximately $2,500 dollars to run a 100ms job a billion times in a month on a machine with only 1.5GB of allocated to this process. You could pay $4350 per year w/ reserved pricing (is there even reserved pricing w/ Lambda? Couldn't find it.) for a c3.4xlarge, which has 30 gigs of memory and 16 cores, which could easily process many more times work than a billion 100ms jobs per month. So lambda seemingly works out well over 6x more than rolling it yourself on EC2 with my overly conservative napkin math.

The other pain point these type of systems produce is that instead of scaling a server here or a few servers there, your thinking in terms of processes, and time and time again this becomes a very rapid way to not only provision more horsepower instantly, but also run up the bill instantly. Code that might have ran on 1000 processes on traditional infrastructure because you had to tune it to fit in that infrastructure (and I don't mean fine tuned, things like, setting your database pool correctly and having basic concurrency), can accidentally be scaled up to 10,000 processes and appear that everything is fine. Unfortunately have seen this quite a bit on GAE and lead to some very costly mistakes.

Processes are harder (in my humble opinion) to have a good feel for what costs what, especially if your scaling up rapidly. Instead of adding a few servers and understanding how that effects the bill, going from 1000 to 1500 processes happens much easier, and you pay for that convenience.

My .02 is that this would be ideal if your either in a hurry and money is no object, or you don't plan on growing past a certain size, or possibly for workloads that are truly not running that often. I think this whole craze of lets replace everything with lambda is going to lead to rapid development (awesome) at high prices if your project takes off, which will leave developers rushing to get off of it if/when their project needs to scale with financial constraints. (not awesome)

theme forest is great, keep up the good work!

While I certainly shared your frustration, I do think that MS had some extremely talented folks there.

Some of the brightest people I have ever had the pleasure of working with came from MS.

Incompetence was far from the problem. Priorities out of order would be one diagnosis.

+1 for occasionally meeting up. My team works remote and while some of us do see each other more frequent, others are on the other side of the country.

So about two times per year, once for a tech summit we go to , and another we have our own summit, we take over a hotel and hang out. During this time we also gives talks and other things but it is half fun and half work.

I think occasional face to face time and bonding time is super helpful, but not everyone needs that every day, and I think in our case it had made these twice a year meet ups something we all really get excited about and look forward too.

Curious your opinion on what you would advise if not Martini or Gin?

I love martini (and lately gin) and definitely love and use go every day. What about these frameworks is not idiomatic go?

Is there another framework that is? Or you suggest just rolling your own using net/http by itself?

Cheers

I can echo this. Have some go processes that have currently been running for over 6 months.

I would like to understand why the word beggar is not equivalent to the word pan handler..

for some reason my brain is not making the jump here.

Is the word bum equally offensive?

Is pan handler preferred because it sounds like more skill is required? what about if there is no tact or pitch and someone is just begging for food?

my brain seems to think all variations of this act are not things you would want to be called, despite if your engaged in it. but is it not calling an apple an apple?

just curious, not disagreeing and would like to know more, thanks

Toyota got a lot of heat over their software implementation surrounding the whole stuck accelerator debacle.

They have set the bar pretty low, despite their myriad of experience.

Pretty sure a technology company that designs cpu's for a living can do better than toyotas example of what not to do, at least in an electronics sense.

They have already hired staff that has a lot of experience in this field, and already had all the engineering chops and bank account to fund this.

While they have a lot of experience to catch up on in the auto industry, I don't think it will be amateur hour.

Has Apple ever released a cpu that had the same quality control issues as Apple maps?

Thanks for explaining. This is a pain point I think everyone that does service oriented architecture has to some level.

Most shops I have worked in seem to mostly attack this via procedure. Ie - release is OK'd by QA, then deploy ticket is made that specifies what versions go where, and then operations is responsible for executing this procedure and ensuring its sanity.

If you have a lot of hands in the pot, (too many cooks), then this is bound to have issues when everyone is busy and things get overlooked.

There is a lot of automation that can help however. Dependency management in this sense I think has to be more elaborate than just "is X installed or deployed", but you might also want to ask "is X healthy?" Just because a package has been installed doesn't mean its working.

So that is in my opinion why there is no one-size-fits-all solution for this. Many releases are fully compatible with each other, and in simple applications probably even more so, but once you have many teams working on many services, it can be a nightmare.

A lot of places have granular feature management. So, frontend can fail gracefully if feature is not supported on backend. Often backend supports more than 1 fronted version set. You can use versioned api's (like putting a version in an http header, or namespacing features like /api/v1/x, /api/v2/x). (it's been said a huge distributed app like facebook has something like 500 different versions of components all running concurrently)

You can have your deployment automation do things like check to see if backend has required version (which would be something that you would have to put in hiera/equiv of what versions are required), and even more importantly if they are responding to health checks.

One project I currently work on has a very distributed architecture, and every application has to conform to a health check endpoint that returns its state and its version. So we have scripts that do either rolling deploys or green/blue deploys depending on what we are deploying, and we can define the order in which services are deployed. We can deploy them concurrently/rolling, or independent/rolling, or either of those flavors non-rolling, or green/blue.

So if we say we are deploying fronted3.1, backend4.2, and we are replacing frontend3.0 and backend 4.1, this would be seen by the deploy scripts (and operations team) as a single changeset for the environment. So if frontend3.1 needed backend4.1 to function, we would specify a independent rolling deployment, so for each service thats being deployed (starting with the backend), we would roll out server by server at a specified interval, and we would wait for that server to return a healthy health check with the specified version. If this fails the deploy is rolled back and notifications are sent.

^ with this working correctly, we have to have the understanding with the application developers that we will never release a backend that is not backwards compatible with any components that rely on this. (beyond a cutoff version, which those do exist too)

In practice this works quite well, but is not magic, and definitely requires some up front development of automation.

So, sorry for the long post, I think this is a long winded way of saying, my opinion is that this belongs in a configuration management, as the logic encapsulating "what version goes where" gets more complex over time, and so some code, logic, decision making, and failsafes all have to be carefully thought out.

Yeah, i look at os-package dependency management as largely a problem that should be left up to the os' package manager I suppose.

So, naturally just like any change, you would want to test the upgrade before rolling to prod.

What are you hoping for I guess? Im genuinely curious. Maybe you want puppet to complain before running that system X doesn't have version C of package K ?

How is that different than it failing on dpkg?

(its possible I'm not understanding..)

Yeah, i hear you, but I think that its a bit of a silver bullet, to have a 1 click install that gives you all of these things.

Just the 3 you mentioned here, privacy, reliability and sustainability are not going to happen if you are doing a 1-click install, and don't understand how it works.

Lets give a quasi specific example maybe..

lets say you have service X, is your ApiService 1.2, and it's in production.

Lets say you have machine 1 and machine 2, both running ApiService 1.2

In puppet, you could use hiera to know what version of ApiService a machine should have. You can use mcollective to interrogate these machines to verify that puppet successfully installed ApiService 1.2 on these machines.

Now, to answer the question "can i deploy ApiService 1.3 on machine 1", I think that that would be best answered by testing this in a development/staging environment with another machine that is not serving mission critical traffic.

Here you would do the testing you need, and ensure it works, both in a installation capacity (say ApiService 1.3 needs a newer version of libxml2 or something), and in a general "does it pass QA" standpoint.

It would be your package maintainers job to specify that ApiService 1.3 requires libxml2 version XYZ.

Puppet/etc (or, specifically when dpkg runs via puppet) would fail to install the package if its not going to have libxml2 version XyZ.

Being that it's your package, you would write it in a way that dpkg is able to install it on the target servers your targeting. So, you could even spell it out exactly in puppet to, "before installing ApiService-1.3.deb, install libxml2-2.x728.deb"

If what your getting at, is that if ApiService 1.3 on machine 1, relies on libxml2 being updated on machine 2, then that sounds a bit brittle to me and should be refactored to be more forgiving.

Maybe that helps make it more clear, if in doubt will look for more comments.

Sounds like you know enough about this process where you know it can be a lot of setup and careful maintenance.

So if you don't want to invest the setup and ongoing maintenance time, or get no pleasure from doing such , may I ask why you want a simple way to do all this at all?

Seems like you should just use fastmail or gmail or similar?

Configuring and running smtp isn't supposed to be super easy and fool proof. I think most systems that claim that will leave you with an unmaintained email solution that is likely not very secure and probably not being replicated or backed up, for starters, not mentioning all the actual things that go into making it all work reliably.

Check PBL too

Except database size. Will grow to gigs and gigs and gigs and gigs. Dspam can be very difficult to manage with a 400gb token database when you have a large system. ( last version I used was 3.9 maybe they have improved this?)

Spamassassin is old and outdated. It's like Apache and rspamd is like nginx. Give it a shot.

Yes but you can't start from scratch without them and do a moderate amount of traffic.

If you have the same clean ips from pre dkim/domainkeys days then don't lose them, or it may be an uphill battle which I would be surprised if you didn't engage dkim to aid in fighting at that point.

It's not just corporations that would not think your legitimate but everyone really.

These sort of unspoken rules you mention are very largely due to combat spam. If anyone with port 25 open was trusted the amount of spam would be intolerable.

Seems sort of silly to want to use email without taking these steps to make it official as possible.

This is not some manipulative plot by google to get you to spend an extra 60 dollars a year, sorry.

You would architect your application to not be so brittle that deploying one wrong package takes the service down. For example if you have to make breaking changes try to make your backend smart enough to handle web2.2 and web2.3.

Tools like configuration management (chef, puppet, ansible, etc) can help you roll out the different versions , and interrogate servers to see what is installed on them.

So in the case of rpm/deb, if you need to update a package and it comes with dependencies I would let the package manager resolve them. But specifically if your packaging your own application components in this format you would not have control to tell server A to be dependent on server B's dpkg state. That would be complicated and brittle.

You want to use rpm/deb/etc to resolve local dependencies only. For setting up infrastructure that needs supporting infrastructure at a certain version to work well this must coordinated at some level even if by hand.

These days there is more than several handfuls of different ways to approach this. I recommend chef or ansible if your new to configuration management.

Edit: oh you use puppet already I see. Just have puppet update packages as needed, and consider using it for deploys. You can group systems together conceptually and do all kinds of things even blue/green deploys, rolling deploys etc. you can use mcollective to see what packages are where.