802 karma · joined May 24, 2011
- simplicity, which in turns creates less complex code and less crashes
- performance, which for my use case was faster than sqlite
- slightly opinionated, in that it forces you to not access objects from different threads, which can eliminate a whole class of bugs (you could and should do this or a similar approach to prevent these with whatever solution, but Realm forces it)
this is handled by the applications code, which inserts data primarily from a rest api into realm in a background thread. Not all data comes from rest however, some from apns or xmpp, depending on application state and the particular data.
Syncing is hard to get right. If realm offered it I would consider but I would not want them to do that at the sacrifice of making their product as solid as it is.
How does couch handle it? Does it have push and poll type of hooks?
Great job on the recent additions to the more fine grain notifications, with this I was recently able to reduce my own code, simplify and increase performance!
Core data is good and a powerful weapon, and certainly educational to learn its approach and how to use it effectively. I sometimes ponder if going through the that first has made me a more competent iOS developer.
Realm is the right tool for the job for me in most use cases. Don't see myself switching back anytime soon, keep up the good work!
Sounds like we are in agreement then - at high enough scale - this is solid advice.
> Then use a proper database that implements MVCC.
InnoDB does implement MVCC. MVCC is not a silver bullet.
>> Do not use transactions, which introduce locks. Instead, use applicative transactions.
> Or just use a database that handle transactions more efficiently.
Easy to say, hard to implement at this scale. If you do a lot of writes and reads concurrently to a hot dataset, it's really quite hard to beat this architecture. This is why its such a popular and battle tested solution for many extremely high scale applications with workloads like this. Not to mention extremely well understood.
>> Do not normalize.
> Bullshit. Normalize as much as is practical and denormalize as necessary. It's much easier to denormalize and it greatly simplifies any transaction logic to deal with a normalized model.
But we are talking about performance... Having something in a single table that is denormalized is always going to be faster than having an elegant data model with "Everything In It's Right Place"
>> Fields only exist to be indexed. If a field is not needed for an index, store it in one blob/text field (such as JSON or XML).
> This is terrible advice.
So facebook/friendfeed, uber, dropbox, and many more or wrong then. Ok.
This is really all best practice for running something like this.
Of course it flies in the face of best practice for running a smaller system. Is there tradeoffs? Absolutely! Would it be smart to do this if the need for this scale is not obvious? Probably not.
You end up having more logic in your application and coordination layers, but this is all pretty good advice for people at this scale, and certainly not bad at all.
I would monetize it in a similar fashion as Touch Tunes. You should look into what they charge a bar.
Having said that I think I would choose your solution based on your teams skills. If they are comfortable with golang there is no reason it could not be made to work.
But as long as you understand how the table is set and the trade offs I don't think it matters. Golang is going to have less batteries included but it has other advantages too. If it's worth it is really a question of speed of code execution vs speed of writing code.
Tesla has hired the brightest minds in the business, _plenty_ coming from years and years and years of deep experience building cars, supply chains, and more.
You can believe that he didn't get funding without having the team in place to build his company, even if he is the fire starter.
> "It's in alpha right now at about 80% of completion."
confused but curious thanks!
That's why this comes up.
It simply not as much of a question of is the idea great, it's we have been here before (and if not exactly this idea, there is a lot of overlap with other new databases), and it's quite hard to pull off.
Not everyone can be the tesla of databases, and there is a plethora of new options in the space. So answering this question seems to weigh a bit heavy in the discussion for good reason.
Sorry but if this is the first experience the team has with building or maintaining data stores then I would definitely not invest. There is so much to learn from experience.
Having said that, I imagine it's not, so it should be an easy question to answer one would hope.
Real debt is too.
Both have their positives of course. Just because there is cases of technical debt gone bad and not being repaid, there is horror stories the same with real debt.
Software often needs this phase to get started, or maybe tech debt is incurred to put a fire out / solve an emergency.
If you are on a high functioning team or maintain software that has been around for a long time, technical debt does reduce. It's not fair to name it any different in my opinion.
One of the projects I am working on lately is a data integrity and syncing platform with multiple PB of data in multiple data centers, and the dataset has been around for 10-12 years.
Some of the file names and path names I see have the most unnatural and unbelievable characters in them.
It can absolutely be quite a challenge. This point about rusts string system sounds like maybe could give me some very precise control.
I am in my 30s fwiw.
Seems like a reasonable purchase if somehow they can convert these people to snap chat users.
Also, I would say over half of my friends using bitmoji have given them money, including myself.
Furthermore, every big movie ( currently the new batman/superman ) has a deal with bitmoji.
But any well architected app that wants to work well without Internet will write many things to disk.
500 writes a second is very realistic when downloading some bulk content that will be used to power different states of the app. Now this might not be a continuous burst of 500 operations a second sustained, but many it has a handful of these bursts over several minutes. Given that, I would say that lots of mobile apps can have brief periods of tons of writes.
Save these for the things you can not do in house, like NaCL. Don't write that yourself.
But string padding, sorry. Any developer worth their salt would laugh at adding a dependency for this. It's irresponsible, and comes across as amateur hour.
This is a simple case of optimizing for responsibility. The inexperienced programmer does not know how to do this, because they have not spent enough time being responsible, and having to deal with the fallout of managing responsibility poorly.
An experienced programmer carefully manages responsibility. A simple function, that is easy to understand, easy to test, and easy to reason about, is something that makes more sense to either pull all the way into your codebase, or write yourself.
Years of doing this means that managing dependencies should never be just slap it into your packaging system and start using the function. If you are on the line for making the wheels turn for a large scale platform that is directly connected to a monetary transaction of some nature, you will quickly find yourself preferring to remain responsible for everything that you possibly can control. There is certainly enough that you can't control to keep you on your toes.
It would be extremely expensive to run a workload like this, and would not be very performant.
This is if we are assuming a very high transaction rate, lots of concurrency and likely highly volatile data. Big table is just not really for that. Either is Dynamo. Both of these are great at giant scale multi purpose databases however.
I moved one large project from RabbitMQ to NSQ over a year ago and haven't looked back. It has just been wonderful to work with and build on top of.
Anything you didn't like about NSQ or pitfalls to watch out for from your experience?
This is part of the game called software. In some worlds this is actually the goal; that the software works so well and is so reliable that your customers eventually don't all need to keep paying you.
So you find new customers and add new, never advertised before features and enterprise clients with SLA support contracts, etc.
It's very understandable that it's hard to monetize, but giving people the impression clustering was going to be included long term and taking it away is not going to score you points.
Furthermore you want to get all the folks doing a startup on a budget hearing a story that works for them. Most ( especially new or young ) cofounder programmer types rarely plan to not need clustering, even if the reality is most won't need or use it. Saying this is for the big kids only is a turn off at this point in the influx story.
Wish you the best of luck
Otherwise you will never be able to really understand what is going on and will not be able to optimize a complex app that needs to be performant at scale.
Learning both has its advantages and I think an ORM can indeed speed up development, most especially for the prototyping phase.
(0) http://techcrunch.com/2014/06/25/facebook-at-work/ (1) http://techcrunch.com/2015/01/14/facebook-at-work-ios-androi...
I know several people like this, my gut tells me it's not super uncommon.
It was not always like this , but the older I get the less sleep I seem to need. In my 20s would feel groggy on 5 hours of sleep, but in my 30s ( current age ) I feel groggy if I get more than 5 hours sleep.
I haven't been able to "sleep in" since I was probably 15 or 16.
Some people probably call this being a morning person, to over simplify it heavily.
Edit: if you do some light googling there is a substantial anoint of data on the internet that claims 8 hours of sleep is too much and 6 is closer to ideal for full grown adults. So getting 4-5 and a healthy dose of exercise does not seem in the realm of supernatural to me. Give it a shot! ( don't leave out the exercise either )
In addition to just energy and whether or not programming is a good fit for you, I think it really really helps to have passion, i.e. Drink the kool aid, work in something you want to use yourself or can't stop thinking about, or is uniquely challenging to you.
I don't know any developers that are excellent that got there by working a mundane/boring programming job. Having said that I also don't any who have not had to work at several of those before they find one that keeps them awake at night in an excited state.
If you aim to improve your skills and work isn't doing it for you, do some side projects or contribute to an open source project. You would be surprised how excitement and passion can beat sleep. Don't get too hooked on it though or you will never have a social life.
My company does 24/7 devops for some clientele. We are a team of 6 for reference. We have been doing this for many years since before "devops" was the term for it. Some of the platforms we have also built for these clients, and some we simply manage, or we have only built+manage the automation.
In terms of answering your question:
- expected duties. In terms of what we are selling the client there is a long specific list of what exactly we can provide the client. In practice however the list is exhausting, and intended 1/2 for CYA, and we will fix whatever issues arise. Some of that is seeing things that need work/adjustment/tuning/optimizing before problems arise, and then as you can imagine the whole being-on-call thing means that you must be an expert at resolving issues that no one saw coming. In terms of SLA stuff, we are supposed to be on-scene (digitally) in 15 minutes, but we are never that slow.
- how deep. Think of it in phases. When something needs fixing, you must quantify the urgency. If there is an urgent fix needed this is ground zero. You do whatever you can, if that involves waking people up, or checking out code bases that are new to you and patching some developers code in the middle of the night, its all on the table. In ground zero mode, you do whatever it takes. Once the bandaid/fix/solution is in place to "stop the bleeding" as I like to say, then there is follow ups, that may include working with the client to have them implement the elegant permanent fix, or if they are in over their head, sometimes that lands on us too. But it serves no-one to have a bandaid that is going to just get ripped off every night. So we see the problem from A-Z, even if our offramp is on B, or M. If it's not urgent, we will just wait for tomorrow to discuss with the client, it falls out of the scope of on-call.
- priority. Well, on-call work is billed and agreed different to ticket work since we are primarily a consulting company. So these are different buckets. But we also have our own products that we own 100%, and the priority is quite easy. If something is broken/down/about-to-break, it trumps everything else. Regular tickets are great, but they are meaningless if the platform for which your developing them against can't stay functional. On-call work rarely has a "queue" or even really a ticket system.
- there is no "complete in time" it's either done or you failed. I say this, having failed too, and it sucks, but it is what it is. If something breaks, and you can't fix it, you don't just go home. But.. sometimes you do, but you walk away from the scene with your tail in-between your legs, no one plans for that.
- managing other teams risk. Communication. Putting energy in ahead of time, and bringing things up before they break is huge. Also if you say "Hey you should turn left, there is a cliff!", and the client is insistent on turning right, this can do two things. A - they know and hopefully its recorded in an email or in a meeting that you wanted to turn left. B - if your absolutely certain they are going to run over a cliff, but your still on the line / have to support the darn thing anyway, you can quietly put a bunch of pillows at the bottom of the ravine, and prepare for the inevitable. When the car goes over the cliff, and everything almost grinds to a halt, and you manage to have been correct about it, but also managed to bail them out, you score a lot of gratitude from the client, and ongoing future relations.
This is all from the point of "consulting" mostly, but I have done this same type of work for many large companies directly on their payroll too, it all applies, and the bigger the company the more it's like consulting in some way also, because bigger companies are more compartmentalized. I have also done this in many small startups where your all just one small team.
Holidays and vacations are important, but they will never truly be the same after years of this. We are pretty good at it now, and we really do try to keep everyone up to speed with where "the bodies are buried" with all of our clients infrastructure. That is the hardest part. Everyone can look at a perfectly groomed wiki or set of 100% refined chef/puppet cookbooks/modules, but the world isn't perfect. So the hard part is learning how to take the punches with elegance, and people need a break. It really does take at least 3-4 people to have a not-insane 24/7 schedule.
We generally plan about 1-3 weeks ahead depending on the time of year and put it into our scheduling system, which is also the system that does our paging. We rotate weekends, and work it out amongst ourselves. Some people have things like date nights that we know about and try hard to not screw up.
Don't build your pager system yourself, you have enough stuff to do. I won't plug them because I don't want this to sound like an advert, but do yourself a favor and pay the money to a company that specializes in bubbling up alerts to notifications that can do phone/sms/email/ios/android/push/etc. These have really helped us manage the insanity & schedule.
If there is any advice, simply don't be frightened to say when you don't know something, even if your tasked with fixing it. There is little room for ego when your the one that ultimately has to be responsible. Being on call means you can't pass the buck, and the sooner you know what you don't know, the sooner you're able to learn it!
Edit: Typos