HNHacker News
TopNewBestAskShowJobs

rented_mule

1,755 karma · joined July 4, 2021

submissionscomments
rented_mule··on Ask HN: Have coders made stuff complicated on purpose?
Sadly, it ends up going pretty deep, too.

6 or 8 years ago, I was a second level manager looking to find the next managers in my group. There was an engineer who was great at mentoring and was exceptionally practical. What do I mean practical? I repeatedly saw her going to her PM and saying things like, "If we can skip this one feature, we can get the other dozen done in a week instead of a month and there will be less than half the code to maintain." The PM was more than happy to do this because he had a lot more features in the pipeline that he wanted to get to, and giving up that one meant getting a bunch more.

That engineer initially turned down becoming a manager (a couple of years later, she became a great manager!). Her reason was that she felt she was the least capable engineer on the team, and she wouldn't know how to coach the better engineers. This confused me, so I asked why she felt she was the worst. She literally said, "[name redacted] can stay late and write a 1,000 lines of code in an evening. I haven't written a thousand lines of code in the last quarter." I had to explain that it's entirely possible that that person is a better programmer, but not a better engineer - they are not the same thing. And, that prolific programmer hadn't shipped a feature in months.

Features, at least good ones, are an asset. Lines of code are a liability. You need the features to be valuable enough to pay for the ongoing cost of the lines of code. This engineer understood this intuitively, without realizing it consciously. The prolific programmer did not get it at all.

So, even as I was trying to build a culture of better engineering (i.e., finding better trade-offs, including limiting complexity), these broad ideas about complexity==good were in the way, deep inside people's heads. Being able to foresee and head-off complexity is an incredibly valuable skill. Sometimes the complexity is worth it, but so often it's not.

rented_mule··on Spending 3 months investigating a 7-year old bug and fixing it in 1 line of code
The worst I experienced in this direction was also on a consumer device about 15 years ago. Performance was degraded and we couldn't explain it. A team of 5 of us was assembled to figure it out.

We spent over three months on it before finding a root cause. It was over two months before we could even understand how to measure it - we were seeing parts of the automated overnight test suite run taking longer, but every night it would be different tests that were slow. A key finding was that almost everything was slow on some boots of the device and fast on other boots of the device, and there was a reboot before each test was run. Doing some manual testing showed it being close to a 50% chance of a boot leading to slowness. Now what?

I eventually got frustrated and took the brute force / mindless approach... binary search over commits. Unfortunately, that wasn't easy because our build was 45-60 minutes, and then there was a heavily manual installation process that took 10-20 minutes, followed by several reboots to see if anything was slow. And there were several thousand commits since the last known good build (the previously shipped version of the device). The build/install/testing process was not easily automated, and we were not on git, otherwise using git-bisect would have been nice. Instead, I spent weeks doing the binary search manually.

That yielded the offending commit. The problem was that it was a massive commit (tens of thousands of lines of code) from a group in another part of the company. It was a snapshot of all of their development over the course of a couple of years. The commit message, and the authors, stated that the commit was a no-op with everything behind a disabled feature flag.

So now it was onto code level binary search. Keep deleting about half of the code in the commit, in this case by chunks that are intended to be inactive. After eventually deleting all the inactive code, there were still a few dozen lines of changes in a Linux subsystem that did window compositing. Those lines of code were all quite interdependent, so it was hard to delete much and keep things functional, so now on to walking through code. At least I could use my brain again!

Using the clue that the problem was happening about half the time and given that this code was in C, I started looking for uninitialized booleans. Sure enough, there was one called something like `enable_transparency`. Disabled code was setting it to `true`, but nothing was setting it to `false` when their system was disabled. Before their commit, there was no variable - `false` was being passed into the initializer call directly. Adding `= false` to the declaration was the fix.

So, well over a year of engineering hours spent to figure out the issue. The upside is that some people on the team didn't know how to proceed, so they spent their time speeding up random things that were slow. So the device ended up being noticeably faster when we were done. But it was pretty stressful as we were closing in on our launch date with little visibility into whether we'd figure it out or not.

rented_mule··on Wireless power transfer system achieves 270-kilowatt charge
Don't get too close with your pacemaker.
rented_mule··on YouTube's next move might make it virtually impossible to block ads
That would work against them because you exclude the segments served up the least, and those modified versions would only be served up once. That approach could work better for YouTube if you did it to the segments of the actual video rather than the ads. But then you could get into things like audio fingerprinting of the segments - only show fingerprints that were also shown to YouTube Premium members. I'm no expert at this stuff, but people who build blockers will brainstorm every time they have to sit through a YouTube ad.
rented_mule··on YouTube's next move might make it virtually impossible to block ads
If it's done that way, then something like SponsorBlock could upload a hash for every segment that is played by everyone with the extension. After a few hundred views, personalization will make clear which segments are the ads (because they are targeted, each ad segment will appear in far less of the segment playlists). Modify the segment playlist on the fly to only include the segments sent to everyone.

If SponsorBlock is still used by YouTube Premium users (to avoid content creator ads), this is even easier, as those users will have the "golden" set of segments.

rented_mule··on I am sick of LeetCode-style interviews
I'm retired now, but I used a variant of this that worked well for me. In my experience, much of what we do as engineers is learn new tools and application domains and apply what we learn to solve problems. So I wanted to test how good a candidate is at leaning and whether they appear to enjoy it (good and enjoy often correlate).

We would work together at the whiteboard, with me teaching them the basics of our application domain and our internal distributed computation framework. Then there were a series of 5 increasingly difficult iterations we'd work through. First, do a simple calculation. Now there's a need for a more difficult computation. And an even harder one. Then explain that it needs to run over billions of log lines, and it's I/O bound, so let's make it faster. Finally, a problem that requires them to probe me deeply about the nature of the data and only needs a hand-wavy answer (calculate a median in a single pass with limited memory - not too bad once you extract from me that the samples are all integers between 0 and 500). To the candidate, this looked like iteration driven by evolving requirements, not 5 interview questions.

I mostly let them drive, but I jump in now and then to keep things on pace. And I would try to answer any question the same way I would as a teammate / mentor.

They don't need to get every problem right. The increasing difficulty is designed so that almost everyone gets something right and something wrong (< 1% gets all the way through the last one). That's an opportunity for me to give supportive feedback to see how they respond. The best responses to feedback yielded something like "oh cool, I can also use that over here to simplify the code". The worst was someone insisting that you can flip the order of addition and division without changing the answer (i.e., order of operations doesn't matter), me saying "I think it does matter, try doing 1/2 + 1/4 in different orders", and them saying "no I have a math degree, I know what I'm talking about."

What they do need to do is be able to solve problems collaboratively, extract learning from an eager mentor, and extrapolate from those learnings to solve problems. The best candidates start finding trade-offs across the application domain / tech tooling boundary that I haven't even brought up.

One of the downsides is that it takes a while to calibrate as an interviewer, and that calibration is not entirely transferrable across interviewers, as candidates will respond differently to different interviewers. I did this interview thousands of times (my record was 14 times in one day, and frequently 20+ times a week - we were scaling an org from ~50 engineers to over 1,000). I never stopped learning about signals I was seeing, but I was mostly calibrated after a few dozen candidates. Leet code calibration is much more transferrable, but what use is that if it's not measuring what matters?

One thing that I think leet code interviews get very wrong is that interviews involve two sides trying to evaluate each other. Wearing people down with leet code is not a good pitch for your work environment. Many candidates told me at the end of my interview that they were shocked at how much fun/educational an interview could be, and many hires said that this interview style was the reason they accepted our offer over more lucrative offers, because they felt they would learn more and have more fun with us.

rented_mule··on Lucid Boss Says Tesla Has Lost Its "Sense of Purpose"
I agree, and want to reiterate from slightly different angles.

It drives me crazy to hear people say, "Is that made in China? Because I won't buy cheap Chinese crap!" Are they happy to buy "crap" from elsewhere? The implied causation between Chinese and "crap" is just not there. There has historically been a correlation, but so many people have made the leap to causation in their heads.

The vast majority of Apple's products have been made in China over the last couple of decades. It's easy to question some of Apple's design trade-offs (e.g., gluing batteries in to make things a fraction of a mm thinner), but those tradeoffs are part of the "designed in Cupertino" that is stamped on their products. The fit and finish is much harder to question, and that is done by Chinese hands. It's excellent because Apple, and Apple's customers, are willing to pay for that.

Chinese manufacturers have proven they are excellent at building to whatever price / quality tradeoff their customers demand. For goods sold in the US, it's US companies and/or US consumers setting the price / quality tradeoff.

To the extent there is a correlation, it comes from China being able to build to the cheaper, and therefore crappier, standards that consumers want. When China loses that edge (as Japan, South Korea, and Taiwan did before them), they will move further upmarket and others will step in to fill that market need (Vietnam? India? Various African countries?). This cycle keeps repeating... one of the funnest parts of watching "Back to the Future" today is that the line "all the best stuff is made in Japan" went from sounding so unlikely in the 1980s to prescient in the 21st century. And that line makes no sense to young people, whose experience is that the "best stuff" has always been made in Japan.

rented_mule··on Deniable Encryption
VeraCrypt's approach to achieve plausible deniability for disk encryption:

https://veracrypt.eu/en/Hidden%20Volume.html

rented_mule··on US opens probe into 130k Ford vehicles over hands-free tech
For me, it's more nuanced than this. Most mistakes that are made by human drivers around me make intuitive sense to me based on how they are driving. They are often somewhat predictable. In those cases, I can partially mitigate those ahead of time (not always - I've been broadsided where I didn't see the other car until the last moment). The wide adoption of cell phones required a significant update to my mental model, but you can spot things like lane drift, uneven speed, etc. that tell you to give space to someone because they are probably distracted. Both chemically-impaired and sleep-deprived drivers tend to give even more clues to stay clear.

I find predictions informed by observation much more difficult with automated driving systems. In many cases, such cars will appear to be driving perfectly, right up until a severe mistake is made. Instead of having observable clues, it requires an understanding of what sensor suite and what version of software is in which vehicle in order to understand the tendencies of that vehicle. Something we can't possibly keep up with, especially given that the developers of the machine learning models being deployed can't deeply characterize each deployment's failure modes.

Given all that, I feel more able to defend myself against human mistakes than automated mistakes.

rented_mule··on Stop Doing Cloud
I'm not who you're responding to, but...

> Go can serve about 500M users per month on Xeon E3 v2 CPUs. > Why, except for HA do you need more?

Because not everything is a simple CRUD website. The last place I worked had a website that appeared to only be moderately more complex than such a site, and had well under 500M users per month. But parts of the application involved complicated processing of many dozens of TBs per day of newly collected data to optimize the content being served (some stream processing, some batch). We had multiple clusters of 1,000+ machines coming and going for an hour or two at a time (often overlapping).

rented_mule··on Procrastination can be caused by anxiety
My natural tendency as a software engineer is to leave everything tidy at the end of the day (i.e., find a "good stopping point"). That meant no clear loose ends on what I'd been working on. But I often had trouble getting started the next day for reasons quite related to the article. Many years into my career, I figured out to leave a simple issue that I had already thought through the solution to, and leave myself a note that pointed me in the right direction the next day. Getting that initial success the next day was often enough to get me going again.
rented_mule··on Faces.js, a JavaScript library for generating vector-based cartoon faces
Someone I know has done some work to parameterize facial expressions: https://www.redblobgames.com/x/1845-face-generator/

He has a bunch of other interesting things like this, too: https://www.redblobgames.com/

rented_mule··on Can Xerox's PARC find new life with SRI?
Yeah, I'm a bit jaded here. You're right that they made a ton of money, but left a lot of great work to fester, even in laser printing. My perspective comes working in Xerox engineering on high volume printers in the early 1990s. PARC folks repeatedly came through with great ideas about how to make the printers faster and cheaper (at the same time).

My favorite demo they gave us was replacing the PDP-11 inside the printer, which was becoming increasingly expensive, with a Sun SPARCstation. They had done all the work to do a full demo on the prototype printer we were working on. Everything was much faster and it provided a lot more features. Plus development printer software could be developed in C or other languages, not just assembly. They were effectively told "that looks nice" and sent back to Palo Alto. That was hard to watch.

rented_mule··on Can Xerox's PARC find new life with SRI?
I would love to see the glory days of pure research groups like PARC and Bell Labs return. But, so many business folks are aware of how Xerox was embarrassed by PARC that it's unlikely that approach will be tried again anytime soon. When I speak of embarrassment here, it's not about the incredible innovation that came out of PARC, but about Xerox management's handling of that innovation.

Billions and billions (trillions?) of dollars in value was created (e.g., Adobe, 3Com, SGI→NVIDIA, Mac / Windows, laser printers, ethernet, etc.) based on their research, but ~0% of that value went to Xerox. From a business point of view, it was a disaster. And business folks are primarily the ones deciding what's funded at this level. If I'm an executive who's not sure if I, or my organization, will be able to tell the great ideas from the duds, why spend money that might expose that?

rented_mule··on -2000 Lines of Code (2007)
As your examples indicate, one reason it's a bad production metric is that it's a reasonable measure of the burden carried by the maintainers of the codebase. The production metric point of view classifies code as an asset. It's closer to the truth to view the functionality as an asset, and the code itself as a liability.
rented_mule··on Thoughts on the Future of Software Development
This kind of optimization is what I did for the last few years of my career, so I might be biased / limited in my thinking about what AI is capable of. But a lot of this area is still being figured out by humans, and there are a lot of tradeoffs between the math/software/business sides that limits what we can do. I'm not sure many business decision makers would give free rein to AI (they don't give it to engineers today). And I don't think we're close to AI ensuring a principled approach to the application of mathematical concepts.

When these optimization systems (I'm referring to mathematical optimization here) are unleashed, they will crush many metrics that are not a part of their objective function and/or constraints. Want to optimize this quarter's revenue and don't have time to put in a constraint around user happiness? Revenue might be awesome this quarter, but gone in a year because the users are gone.

The system I worked on kept our company in business through the pandemic by automatically adapting to frequently changing market conditions. But we had to quickly add constraints (within hours of the first US stay-at-home orders) to prevent gouging our customers. We had gouging prevention in before, but it suddenly changed in both shape and magnitude - increasing prices significantly in certain areas and making them free in others.

AI is trained on the past, but there was no precedent for such a system in a pandemic. Or in this decade's wars, or under new regulations, etc. What we call AI today does not use reason. So it's left to humans to figure out how to adapt in new situations. But if AI is creating a black-box optimization system, the human operators will not know what to do or how to do it. And if the system isn't constructed in a mathematically sound way, it won't even be possible to constrain it without significant negative implications.

Gains from such systems are also heavily resistant to measurement, which we need to do if we want to know if they are breaking our business. This is because such systems typically involve feedback loops that invalidate the assumption of independence between cohorts in A/B tests. That means advanced experiment designs must be found that are often custom for every use case. So, maybe in addition to thinking more like product managers, engineers will need to be thinking more like data scientists.

This is all just in the area where I have some expertise. I imagine there are many other such areas. Some of which we haven't even found yet because we've been stuck doing the drudgery that AI can actually help with. [cue the song Code Monkey]

rented_mule··on Thoughts on the Future of Software Development
I'm not sure that there is an endpoint, only a continuation of the transitions we've always been making.

What we've seen as we transitioned to higher and higher level languages (e.g., machine code → macro assembly → C → Java → Python) on unimaginably more powerful machines (and clusters of machines) is that we took on more complex applications and got much more work done faster. The complexity we manage shifts from the language and optimizing for machine constraints (speed, memory, etc.) to the application domain and optimizing for broader constraints (profit, user happiness, etc.).

I think LLMs also revive hope that natural languages (e.g., English) are the future of software development (COBOL's dream finally be realized!). But a core problem with that has always been that natural languages are too ambiguous. To the extent we're just writing prompts and the models are the implementers, I suspect we'll come up with more precise "prompt languages". At that point, it's just the next generation of even higher level languages.

So, I think you're right that we'll spend more of our time thinking like product managers. But also more of our time thinking about higher level, hard, technical problems (e.g., how do we use math to build a system that dynamically optimizes itself for whatever metric we care about?). I don't think these are new trends, but continuing (maybe accelerating?) ones.

rented_mule··on Deleting Software I Wrote Upon Leaving Employment of a Company
I saw something that was sort of the inverse of this. A teammate quit with zero notice in a 3 AM email. In that email, he indicated he'd made the decision to open source all the code that he had written in his last year or so with the company. It was in a public github repo in his personal account. The company had a lot of open source, but this was never expected to be part of that.

We were able to get it taken down and no other action was taken against him, but it was an interesting move on his part. His motivation didn't seem to be bad blood - he continued pleasant interactions with several of us, including those in his management chain, long after leaving.

rented_mule··on CA train project needs another $100B to complete route from San Fran to LA
How about we add Michigan, Indiana, Kentucky, Tennessee, Arkansas, Mississippi, Louisiana, and Texas to the California example? The Interstate 69 project through those states has been going on for decades, and it is nowhere near completion. But I-69 doesn't get the same scrutiny. Maybe because it's about cars rather than trains?
rented_mule··on Amazon Cancels Fees for Customers Moving to Rival Cloud Services
According to a 2014 US Supreme Court decision...

"While it is certainly true that a central objective of for-profit corporations is to make money, modern corporate law does not require for-profit corporations to pursue profit at the expense of everything else, and many do not do so."[1]

Public or not, people involved in a business do have fiduciary duties to their investors. But those duties are not specifically about profits. They are about "obedience, information, loyalty, and care"[2].

----

[1] https://caselaw.findlaw.com/court/us-supreme-court/13-354.ht...

[2] https://online.hbs.edu/blog/post/fiduciary-duty-to-investors

rented_mule··on What if AGI is not coming?
I'd love to see more algorithmic efficiency in ML. But hasn't ML been going in the other direction for at least a decade now? It seems to me that it's been going aggressively towards brute force algorithms. Specifically, figuring out how to do as much matrix multiplication as possible.
rented_mule··on Florida Passes Sweeping Bill to Keep Young People Off Social Media
> (a) May not retain personal identifying information used to verify age once the age of an account holder or a person seeking an account has been verified.

That's good for the people being verified. But it means the companies can't retain proof that they did, in fact, verify age. So how do companies accused of this defend themselves, especially if they're in the right? That's a bit scary in a state where the governor used his power to retaliate against the largest employer in the state for expressing their view of a law that affects their employees.

rented_mule··on Documentation for the JSON Lines text file format
No, this is often used for logging. In my last job, we were generating on the order of a million lines like this per hour in the log I worked with most often (many millions per hour if you count all the logs in the company). Each line of that log was between a few hundred bytes and ~100 KB. It would take a lot of work to support that in a database. That log was processed by many Spark jobs (some in streaming fashion, others in batch). "Small" summaries were added to a columnar database each night. The main table in that database contained hundreds of columns and billions of rows for more interactive analysis.
rented_mule··on Institutions try to preserve the problem to which they are the solution
You are replying to a post that references exceptionally expensive efforts. The inflation adjusted cost of the Apollo Program was hundreds of billions of dollars[1]. By what measure is that cheap? Maybe I'm missing intended sarcasm?

[1] https://www.planetary.org/space-policy/cost-of-apollo

rented_mule··on The Golden Age of Cordless Power Tools
Not who you're replying to, but I also have a brushless 36V Makita chainsaw. I buck and split 2-3 full cords of firewood every year for myself. The biggest logs I've cut with it are ~30 inches in diameter. I only have a 16" bar, so that means coming at it from all around. The Makita is stunningly good for my use case, much better than the gas saw I used to have.

I have two sets of batteries, and they'll get me through more cutting than I want to do in a day. Occasionally weather will cause me to need to do more than I want in a day, and one set of batteries will charge in just enough time for me to get through the other set and take a little break, so I can go all day long. In my case, the only time I'm not cutting near grid power is when I'm clearing a log that fell across a trail I'm riding in my UTV, but I'm not going to keep riding down a trail that has enough downed trees to consume a set of batteries.

There's no place to rent chainsaws within an hour's drive of where I live (I live in a rural, forested area). There are certainly more powerful gas-powered saws, but I don't need the power. If I suddenly feel like getting some cutting done, I much prefer not having to worry about whether I have gas and premix (I still have to worry about bar/chain oil, but a gallon lasts me 2-3 years). The nearest gas station is a 30-40 minute round trip. I love not having to worry about disturbing my wife or the neighbor when their windows are open. It was fun last summer when my neighbor came out of his house to be surprised that I'd bucked 3 logs, each 30 feet long and 18" in diameter, while he slept with his windows open. Saying there are "much better gas powered" saws implies a single dimension of comparison. That's not the case - there are just different trade-offs.

rented_mule··on Ask HN: Anyone left software to study non-STEM subjects?
Not me, but a friend... She was nearly 10 years into a career, most of it at a FAANG and the rest FAANG-adjacent. All as an ML engineer. Her spouse is also a software engineer.

She left her engineering job ~18 months ago to study music on the way to a music career (she had studied a lot of music during her earlier school years and has been composing for years). The music program involved everything from the physics of sound to composition, production, engineering, performance, etc. It was a deeply intensive one year program that she has now completed.

She keeps getting offers to do software around music (effects, workflow management, etc.) as there aren't enough well qualified software/ML engineers around that deeply understand the composition / production / engineering process. But she wants to be more on the music side. She has taken one such gig to pick up a mentor who composed the scores for several prominent video games, an area she's interested in.

She still has more than a year of runway, so she's taking 2024 to generate a portfolio of her own music and sort of build a brand (my words, not hers).

So, she's finished the school part, but hasn't completed a career transition. It's exceedingly unlikely that she'll ever replace her tech comp, but that was never the goal. It's been great for her mental health. Part of that is that she's been very actively working on her mental health through all of this. This new journey has required its own kind of resiliency because the path is far from clear and requires a lot of reflection and a lot of strategy planning about how to proceed. A supportive spouse has been a huge plus.

rented_mule··on Apple has not fixed the macOS audio left/right balance bug for nearly 10 years
I've been happily using an app available for free on the App Store to work around it for years now:

https://apps.apple.com/us/app/balance-lock/id1019371109

rented_mule··on Thoughts on tech employment
What's to stop Apple, for example, from setting up companies all over the world that employ their people and own their IP local to the employees? Then Apple US buys / licenses products for US distribution from those companies.

Massive companies have far more incentive to "engineer" their way around such laws than lawmakers have to find something without loopholes. Especially given how lobbying works in the US.

rented_mule··on Ask HN: What are some homeless shelter innovations?
The Delancey Street Foundation focuses on adjacent / overlapping problems - recovery from substance abuse, convicted criminals re-entering society, and the homeless situations that often come with those situations. They have been remarkably successful and financially self-sustaining. I lived next to their development in San Francisco for 5 years and regularly patronized their retail businesses. For the first year or so, I had no idea about the foundation behind the businesses (or that their members were the employees) until I read something posted on the wall at one of them.

http://delanceystreetfoundation.org/

https://en.wikipedia.org/wiki/Delancey_Street_Foundation

rented_mule··on Only 7k Miles? EV Owners Not Happy About Tires Not Lasting Long
I think you're absolutely right, assuming the driver is paying attention. Too many don't. I encounter a lot of people bouncing around in a 10-15 mph range (sometimes 20-30 mph) on the highway while they are looking at their phone. This happens in all kinds of vehicles, but the coasting vs. regenerative braking impact on tires emerges in these situations.
← PreviousPage 8 of 12Next →