this is a major hassle when converting from avro (from kafka which uses a schema registry, so schemas are not shipped with the avro data) and storing in parquet which requires a schema in the file but you can 'upgrade' it with another schema when reading it. It would be great to have a binary protocol-like format (schema-less avro), and a schema-less columnar storage format.. which is I guess is what these guys are doing.
I really only stopped caring what people thought after I got divorced (I was 43 at the time). Before that, it was 'the big house', 'the bmw', 'running the cool business'.. ostensibly, showing off to everyone else how impressive I could be (I wasn't). I felt like a fraud because I hated what I was doing. Others still had more money, a bigger house, and a cooler car. It was an arms race and I was losing. But when all of that was stripped away after the divorce, it was again, just me.. along with my three young kids in my two bedroom apartment. I realized that all of that was stupid, and really didn't contribute anything to who I really was. I was forced to do only what was absolutely necessary.. to stop being selfish and focus on my kids and rebuilding my life. Although I was struggling, it was liberating. Now I could regain what I lost: money, sure..I was always pretty good at my job, but mostly my humility. I didn't come from much, which I used to wear as a badge of honor when I was younger. Losing a lot of 'stuff' made stop caring about where I lived, or what I drove. I could just simply 'focus' on what I personally felt was important, not someone else's arbitrary standard. Now I drive around my neighborhood (in my 12 year old honda) looking at all the late 30-40 year olds racing inexorably towards their collective mid-life crises, doing their home improvements, driving their Teslas that they can't afford, berating their kids because they're not impeccably perfect in every way and I think to myself : 'yeah, I know.. it's coming for you too'..
I should also qualify my jaundiced view of the future, with an actual belief that we'll eventually solve some of these problems.. particularly global warming, and energy creation. I have exactly zero faith that we are finished trying to kill one another for money.
I should add that, of course, there actually was terrorism.. just not in the US on the scale of 9/11.. (which I had the misfortune of experiencing firsthand in Manhattan).
I was born in 1967.. so I remember the 80's very well. First, let me say that it wasn't all great: AIDS was pretty scary (as someone else noted).. no one knew what the hell that was and it was killing a lot of people. There was also the ever-present spectre of nuclear war with Russia...Iran-Contra, 1987 stock market crash.. However, to your point, there was no exhausting news cycle, no exhausting bullshit social media, no in-your-face, constant idiocy from the political class (well, there probably was.. but we didn't hear about it), no 20 years of terrorism and associated wars, no worldwide pandemics that killed millions, no global warming; all of which has scared the living shit out of every person alive (ya know, if you're not completely insane). I grew up in the 80's.. went to high school, and college in the 80's. I remember always being optimistic about the future. Now I'm not so sure.. I have three kids all under 23.. I do not envy what the future holds for them. (which is probably what my parents said about me;)
I can attest to that statement. I work with many contractors from Ukraine and they're tough, determined, and extremely skillful. They can handle any problem, in any condition, at any time. In my experience, they will not stop until they're satisfied that the issue has reached a conclusion that they are satisfied with.
Moreover, storing a large binary file (in this case) in Dynamo is probably not the best use-case for it..most likely would have to convert to base64 in and out of it.
currently reading Dr Strogatz' book 'Infinite Powers'.. fascinating journey through calculus' origin story. Actually reading about Newton's method of integration using infinite series right now..
we use DynamoDB like a big hash table of s3 file locations.. we look up these locations via a key (at the time, it sounded like a pretty good use-case for it). I suppose we could have used some other managed redis or memcached thing, but being an AWS shop, it was, and is, pretty useful. I have to say, it's been pretty effortless to configure.. read/write units are really the only thing we've had to configure (other than the base index) The rest of it has been easy. It has about a 100 million entries that are read or written pretty quickly.
I was at Princeton once (maybe 15 or 20 years ago) for a meeting with one of the startups that spun out of one of their graduate programs.. The guys asked to meet me at the CS building (which was pretty cool to actually be in because I could have never gotten into Princeton as an undergrad!). Anyway, I'm walking down the hall and I see a door that says 'Brian Kernighan'.. I turned and asked them if that was 'THE Brian Kernighan'? they chuckled and said 'yeah..that's him'. I couldn't believe it.. I wish I had met him.. but it was pretty cool to see the guy just walking around.. talking in his office, etc. This guy had a huge part in inventing something that runs most of the networked world.. and there he was.
agree.. I really like scala and Li's blog and book have helped me immensely. I use it mostly for spark, but I've found it really useful now for other things. The concurrency really is pretty nice (compared to other languages). His API's are written in the same way as his blog and his book: zero condescension; he's not constantly trying to impress you. This seems to be the problem with the language.. the language allows you to really go nuts if you want to and the result can be an incomprehensible type system that takes a week of hacking just to figure out what it wants you to do. That said, json4s isn't terrible.
I generally understand the construction.. (you know, the canonical example of splitting a string of words by spaces, which returns a collection, which is then flattened). But there seem to be so many ways to use it differently.. resolving futures, spark dataframes, scala collections, all have their idiosyncratic ways of processing the results The dsls of the transformations are the most confusing for me. When I’m bored at work like I am now, I think of all of the ways that flatmap gets used and I try to refresh my memory on how it’s used. For some reason, it’s harder for me to grok than any other idiom.
another wonderful, honest, insightful post. Your book and blog posts have helped me immensely in the past two years deploying spark and 'big data' tools/processes at my company. I could not have done it without your help. Cheers!
healthcare insurance is the best example of a state-sponsored scam I've ever seen.. and it just continues unabated. I can guarantee that almost no one involved in that transaction actually knows what the per-unit cost for a 1 hour ultrasound actually is.. and when I say 'cost', I mean how it costs the hospital. Health insurers, hospitals, doctors are so awash in money they probably can't believe they're still getting away with it after all this time. Just look at all of the new hospitals and care facilities that are getting built. You don't do that if you're 'squeaking by'.. Hospital lobby is full of shit. You can probably trace all of this back to the HMO model where insurers began to add a lot of distance between the caregivers (i.e. doctors, hospitals, etc) and the patients. Like wall street did (with basically every financial instrument), they add layer upon layer to the cost chain, which adds an equivalent number of money-takers, to the value proposition. Insurance companies amortize the total cost across their subscriber base and 'poof', you've got our ever increasing health premiums. Just ridiculous and so obvious.. but since they're 'doctors' and 'care givers', we automatically just assume they're not motivated by profit and therefore beyond reproach...um, wrong.
it's all been said before here, but after working in IT for 30 years, I've literally forgotten more things that I've done than they could ever ask me. I've written threaded C++ code , built numerous IOS (objective C, Swift) apps for clients, written A LOT of java (many different app servers, most not supported anymore), Node, scala, spark, SQL (multiple dialects).. but I honestly can't remember most of the time how I did any of it (certainly not enough to regurgitate on a programming test). Headhunters are always amazed at the varied work that I've done.. but I've forgotten most of the low-level details which sucks because some of it was like 20 years ago. I've never been let go from any client, and I've been consulting continuously for big companies for over 25 years. But this is clearly not a sufficient for knowing answers to their specific question. I can't tell you offhand how to build depth-first-search, and I can't remember big O for most of the data structures that they would test me on). With three kids, I really have no time hack leetcode problems all day.. I completely understand that they need to filter, but I'm at such an odd place that it's hard for them to quantify my value because it may not line up to their identification process (which is also fair, I suppose). I somehow doubt that it's age bias, it's almost that I've done way too much and they can't rely on trusting that I'm not lying about any of it. I would have to interviewed by someone who puts the same value on my specific accomplishments with specific discussion around design, implementation challenges, etc. Not specific syntactical bullshit questions.. Most of the shit I look up anyway based on what the specific problem is. As someone else said, smaller companies tend to trust a little more in this area.. I basically learn what I need so I can solve their problem in the most performant way. I know I'm not an 'expert' in anything, but I've been able to quickly learn things and apply them so it accomplishes their goal. This isn't really 'testable' per se.. other than perhaps hiring on a trial basis and throwing some difficult problem at me. None of that is really scalable though. It's really a difficult problem.. the best guys I've worked with really didn't look good on paper. However, they always achieved the goals and worked well in chaotic, mostly unstructured scenarios where the goal wasn't well defined, and the technology choices suspect or basically untested. They figured it out.. 'figuring it out' under pressure is hard to quantify in an interview.
I never understood how anyone who calls themselves a 'project manager' of a software project, could not actually be a technical contributor. The best projects that I've been associated with have always been led by technicians who also could write, present, manage users, project plans, etc. They're not just 'coders'.. How can anyone define project strategy, tasks, and timelines without an implicit understanding of how it all works? These people (generally rare in my experience) can easily cut through bullshit by technical, and non-technical staff pretty easily. The projects that they lead are generally on time, with fewer problems. I've learned early in my career never to get too far from what it is people are paying us to do : deliver some business value through the act of writing software.
yeah.. I really don't see why everyone gets so bent about it. I can certainly understand the cognitive load of learning essentially all of the features so you can find the ones that work.. but really, picking a set of features that work (and ignoring possibly better techniques) seems to true for all languages. Java, Python, etc.. map.filter.reduce.. is better than for loops, but I still see those. C++ has a LOT of features and 40 years of both advancement in the language, as well as strong third-party library development.. so, compared to something like Go, it's a bit daunting to use.
completely 'normal'.. I've literally never been trained on anything that I've actually worked on. The natural reaction to this is basically the standard 'google all the things' when you're working with something new.
I used to rail against this practice of using CS-y questions as hiring filters.. but I think now I get it. I just got a take home test from a start up where I had to find all bounding boxes in a 2d array of chars, filter out the overlapped (or contained boxes), and return the largest non-overlapping box.. whew. I wrote it with a recursive traversal to find each bounding box (something I thought was clever), in scala (their language of choice, although I could choose any), within a few hours on a weekend, and I still didn't get the job. Evidently, missed some edge cases, but in their view generally worked for the most part. Obviously, this isn't remotely close to what I'd (ever) possibly do there (it's a web portal.. or something), but I can see why it was used. It's like the SAT, it's something that can be cheaply deployed with immediate feedback. You don't really have to guess if this guy/girl is going to hack the basics. Now clearly it doesn't really control for any sort of specifics of experience, other than how you think, and how you put together a specific solution. They need to filter on something, other than a gut feel, or a good conversation. They're in the business of software development, so I guess this is (at least) some objective measure of that. What might be interesting is to know how many FAANG wash-outs there are, despite doing really well on these tests. I wonder if it's close to what the normal attrition rates would be at other companies.. maybe none of that really applies.
I switched my major from 'business' (yawn) to comp sci 35 years ago (1986).. at my school, it was previously a 'concentration' in the Math department, which meant you got a BS in Math, concentration in Computer Science. It wasn't a full Bachelors of Science degree until about two years before I got there.