4,603 karma · joined March 5, 2008
I totally agree the options are all somewhat unpleasant. I've found dbt to be a nice step in the right direction.
This is a beautiful way of putting it, but IME rarely do actual debates actually rise to this level. Typically, they just amp up our confirmation biases and cause people to dig in.
I've had a few beautiful debates/discussions with friends in my life where we took sides on some issue and by the end, had both switched sides. These are great, because we're both cracking up by the end, but they've been incredibly rare. They require a way of exploring topics in a way that doesn't require/trigger ego investment and some adherence to the principle of charity[1].
Hell, I'm practically demonstrating the wrong way to do it in this very post. ;)
Faust is a stream processing library, porting the ideas
from Kafka Streams to Python.
It is used at Robinhood to build high performance distributed systems and
real-time data pipelines that process billions of events every day.
I did some evaluation of various Kafka ingestion methods a while back (including Faust), and didn't find Python to be a great fit, so I'd be curious to know if that has anything to do with it.True... I wasn't clear about that, but I was including it as another of these extrinsic goals.
If you're an intrinsically motivated software professional, you're likely motivated by a number of different factors: creating a great product, love of solving problems, writing elegant code, the fun of building new things, making users happy, etc... (exact levels those matter vary by person.) You're probably not intrinsically motivated by things like hitting sprint goals, achieving a certain velocity, meeting OKRs, etc... (Did you enroll in CS classes thinking "I'm so looking forward to hitting sprint targets!"?) Thus, all of these tend to amount to various forms of extrinsic motivation.
A problem with scrum, IMHO, is that it places these extrinsic factors front and center. Yes, the other things may still matter, but the stuff you focus on and talk about week after week are the various scrum related ephemera. (One may include OKRs, which you may not discuss as often but which may represent another extrinsic factor looming in the background.)
There's some evidence to suggest that extrinsic motivation may dampen one's intrinsic motivation, and so a pet theory of mine is that, to the extent scrum does that, it has a certain deflationary effect on the overall morale of an engineering organization. It might be subtle, depending on the particular org and how scrum is implemented[2]. I didn't even notice it until I went to a place that didn't use scrum.
[1] I'm not strongly committed to this idea. It's just something rolling around my brain.
[2] Obviously, some orgs are better than others. I'm open to the idea that the above is more a reflection of "scrum done badly" than scrum itself.
>>> a_tuple = ([1, 2], [3, 4], [5, 6])
>>> a_tuple[2] += ['foo', 'bar']
TypeError: 'tuple' object does not support item assignment
>>> a_tuple
([1, 2], [3, 4], [5, 6, 'foo', 'bar'])
vs >>> a_tuple = ([1, 2], [3, 4], [5, 6])
>>> a_tuple[2] = a_tuple[2] + ['foo', 'bar']
TypeError: 'tuple' object does not support item assignment
>>> a_tuple
([1, 2], [3, 4], [5, 6])
Edited to add, to understand what's happening here, check out the documentation on in place operators: https://docs.python.org/3/library/operator.html#in-place-ope...I used to try to type with my arms/elbows floating -- not resting on anything. This seems to be fairly standard advice. For example, you can see it here: http://ergonomictrends.com/proper-ergonomic-typing-posture-a... I've had various doctors recommend the same thing.
But when I try that, the muscles in my shoulders/back get incredibly inflamed, and tension/pain tends to spread throughout my body, eventually to my hands/wrists.
Eventually I gave up on that, and now my elbows are splayed out and rest on the chair's arm pads. My hands rest on wrist pads, and my keyboard is split about 18" (using a Kinesis Freestyle 2). Basically, my hands/arms are nearly always at rest, and no energy is required to keep them suspended in the air. I also try to avoid "reaching" for the upper rows with my fingers, instead I move my entire arm forward to some extent to hit those keys (which is more a 'push' than a 'lift', because my arm pads have a bit of give in them).
I've had no problems since adopting this setup.
First, the take home test was relevant to the sort of work one might actually do in a reasonable time frame. (Read a csv input file, hit an api per record in the csv, then merge that info and output an updated csv.) Could theoretically be done via shell script; my submission was ~100 lines of python.
Second, submissions were graded mostly for correctness. Style/structure and all the other stuff were nice but usually weren't deal breakers, iirc.
Third, the on-site included a session where we reviewed the submission with the candidate, asked questions, etc... This was pretty instructive, because we could point out a missed edge case or whatnot and see how they'd modify the code to handle it.
Fourth, names & personal identifiers were scrubbed from submissions, to avoid potential bias.
So, it was a better than average process, IMHO. That said, after reviewing a fair number of coding tests myself, I'd note that there are still a number of downsides to take home tests:
a) even if the instructions specify that one should take less than N hours, it's obvious that some applicants spend considerable time on them. So we'd see solutions of wildly differing quality; some with detailed unit tests, documentation, Dockerfiles, vagrant files (!), build scripts, etc... etc... some with just a single file.
b) Even though the primary criteria was correctness, there was some room for assessment of code quality, and subjective bias of the reviewer was undoubtedly a factor. A solution that missed an edge case but which was well structured would generally be fine, but "well structured" is obviously subjective. Or, although applicants could use "the language of their choice," those selecting, say, Java, were at a disadvantage. Most of us were Python/PHP/Ruby folks and the Java submissions looked generally bloated in comparison, and seemed to be graded more harshly.
Not sure what a perfect solution would look like. To resolve issue a) one could impose a strict time limit or use HackerRank or whatnot, but that would create time pressure and up the anxiety. For b) one could rely on purely objective criteria, though that may create other unforseen problems.
https://shop.americastestkitchen.com/special-issues/cook-s-i...
For example, I bought Words with Friends a long long time ago, and for a while it was great. But now they've added tons of gamification features like powerups that bypass game rules, social, awards, etc... to try to get you to engage more and pay more money. Now, various awards screens pop up and waste seconds of my life after every move.
I just want to play standard Words with Friends with my wife like I have for the last 10 years. But one can see that folks like me aren't making Zynga any (new) money, hence the other inducements.
https://news.ycombinator.com/item?id=21763789
Having an actual story to back the learning process might help making employees feel more comfortable with it.
I figured this would be great in a number of ways -- developer deepens their knowledge of some topic, possibly even has fun, practices their communication skills by sharing with others, and the entire team learns something. Although the idea was technically backed by management, it didn't really catch on. I think the problem was that the short term opportunity costs were too high when everything was on fire (which was often).
[1] 2-3 days max, probably.
[2] "Short" is the key word here.
https://github.com/jorgebastida/awslogs
Which allows you to stream Cloudwatch logs via a command like:
awslogs get <log group name> ALL --watch
which is cool because you can just specify the log group and get all streams without having to dig around for the specific stream in the AWS console.There's a python based command line HN client (https://github.com/donnemartin/haxor-news) that's nice, although I think the dependencies need updating (I get various conflicts if I try to install other packages in the same venv).
For twitter, rainbowstream is nice:
I believe this is what Rackspace was attempting a while ago -- leveraging OpenStack to provide cloud services, so you'd be free from lockin in the sense that you could move to some other OpenStack compatible provider.
I have a theory that SRP is the driver behind a lot of this, and that the way it's commonly understood is incorrect (in part because the idea itself is somewhat ambiguous and possibly incoherent). I'll try to go into more detail in a blog post.
After that, without the clear goal, I think we drifted a bit. Mentoring still had positive value, but I in general I think the value curve is logarithmic and after a while one reaches a point of diminishing returns.
SELECT AVG(stats.val)
OVER(ORDER BY stats.date ROWS BETWEEN 5 PRECEDING AND CURRENT ROW) AS avg_val
FROM stats
Regarding this point:What I've learned from this project I'm doing is that sql is very, very powerful in what it does. I've shifted more to using it for actions than I would have in the past, and pretty much all thinking I do is making as little code as possible.
It's an interesting tradeoff. I've been on the other side, in an environment that was super SQL heavy (also Oracle PL/SQL) where we had to deal with these huge complex queries, and it was incredibly painful, for various reasons. One, SQL by itself isn't that composable in the way code is[1], so there tends to be a lot of copy/paste/duplication. Two, giant queries can be hard to understand/debug. Three, SQL can be hard to test. I mean, you can set up integration tests against a database (which can be time consuming), but how do you unit test a query?
On the other extreme, I've seen cases where something separately queries two tables and does a join operation in code, which would be much easier/safer just by doing a sql join. So there's a tradeoff. There's a hard to define sweet spot somewhere in the middle, IMHO, where you take advantage of relational semantics/joins, and utilize various sql features like group bys and whatnot, but where you haven't shoved all of your logic into the database layer.
[1] You can create CTEs (with foo as....) in postgres, but they may have downsides wrt performance (at least, last time I'd read up on it they posed an optimization boundary). You can also create views, but similar issues may arise.
"It looks like this <code block/method/module> will cause XYZ to happen. Is that what you intended?"
Simple example (Python):
def foo(x):
print(x.lower())
"If x is None this will raise an exception. Is that an
expected outcome, and/or can we assume x is always non-None?"One thing I like about this approach is that it first describes your understanding of a particular code section, which helps illuminate any assumptions we might hold (and which might be incorrect). This is a good practice generally, IMHO, with or without the followup question. A guy I used to work with would make non-judgemental comments that just summarized his understanding of what was happening in the code, and it was helpful because when the comment was off, it illustrated a way in which the code was perhaps unclear or incorrect.
"Second, we can now pass in different implementations of our dependencies when executing in test3. This is very good, but let me rephrase that in more general terms: the values associated with certain names are now dependent on the environment in which we are executing. This should sound very familiar, dependency injection is just a more controlled form of dynamic scoping."
This seems slightly off to me, unless I'm misremembering my java. In the OrderService example, the value of the variable `bank` cannot change because it's a reference: it'll always refer to the same object. However, if the instantiated `BankService` is mutable, then the internals of that object could change in various ways. Hence, in practice, the dangers of this pattern only seem problematic if dependencies have mutable state.
Back when I was doing java, we used spring beans everywhere for this sort of thing and iirc, they had no mutable state. In Python, I use a similar pattern a lot, where I have classes that are in practice 'immutable once initialized' -- though of course, in Python you could always mess around with the internals at runtime -- which are segregated from classes or objects that have mutable state. (Similar to the structure described here: https://medium.com/unbabel/refactoring-a-python-codebase-usi...)
Of course, I get that you can't in practice know how stateful everything in your dependency graph is. But I think the real problem here (if there is one) isn't explicit DI in the form of dependency passing, but (unexpected/hidden) object mutability.
https://dilbertblog.typepad.com/the_dilbert_blog/2007/07/car...
Decent advice, imho.
[1] Note, this does not constitute an endorsement of Scott's political views.
1. Stopped trying to categorize cards and just create them all in a single deck (with tags in lieu of separate decks).
2. Whenever I find myself googling something or looking it up on StackOverflow (that's not wildly specific or related to domains I don't really care about), I add a new card for it.
3. Review cards for a few minutes in the morning.
The above has worked rather well. I mostly focus on topics I'm interested in diving deeper into (python, for example) and has helped fill various lacunae in my knowledge.
For broader sorts of knowledge acquisition, I use Boostnote to keep track of interesting notes/articles, but I have to confess I'm not that great at keeping it updated/relevant.
https://en.wikipedia.org/wiki/Wikipedia:Chesterton%27s_fence
https://www.youtube.com/watch?v=OWyMA_bT7UI
It used the windows version of Dragon Dictate that had a python interface which was then hooked up to emacs, iirc. I attempted to replicate his system at some point but never got it working -- seems that the libraries he referenced weren't well maintained and/or running windows through a VM on my mac introduced additional issues.
Assuming there are at least some out there who feel the same, it occurred to me once to try to publish/monetize these agony free routes but I realized it'd pretty much instantly lead to the Waze problem.
1. Start by questioning a long held assumption ("Do we need modules? Why? Is it because we stored functions in files?" Etc...)
2. Consider alternatives / what ifs. ("What if we used a database instead of files?")
3. Think through implications of the alternatives ("How would we identify functions?")
IMHO, this is a powerful technique for generating new ideas. Step 4, not explicitly performed here, might be to take ideas generated from steps 2 & 3 and apply them to other domains. ("Ok, perhaps a database of functions wouldn't be great for code, but perhaps it (or something similar) would be great for some other domain...")