HNHacker News
TopNewBestAskShowJobs

mikehollinger

1,155 karma · joined August 5, 2015

I lead teams that design and build high performance hardware and software, most-recently in applied AI for Sustainability, and before that, in the deep learning and hardware accelerator spaces.

Find me at https://www.mikehollinger.com

submissionscomments
mikehollinger··on X.com is Twitter, but what are [a-z].com?
> People just google “bob builder” anyway.

Today they do. When a domain was $200 to register in the 90’s, people treated URLs like phone numbers were also treated at the time - to be written down, memorized and then typed precisely in (with slashes!) to find whatever Bob the builder was offering.

It’s odd to me tbh that phone numbers were solved with contact lists and address books, along with the occasional “new phone, who dis?”

mikehollinger··on Purple Llama: Towards open trust and safety in generative AI
> Output sanitization makes sense, though.

Part of my job is to see how tech will behave in the hands of real users.

For fun I needed to randomly assign 27 people into 12 teams. I asked a few different chat models to do this vs doing it myself in a spreadsheet, just to see, because this is the kind of thing that I am certain people are doing with various chatbots. I had a comma-separated list of names, and needed it broken up into teams.

Model 1: Took the list I gave and assigned "randomly..." by simply taking the names in order that I gave them (which happened to be alphabetically by first name. Got the names right tho. And this is technically correct but... not.

Model 2: Randomly assigned names - and made up 2 people along the way. I got 27 names tho, and scarily - if I hadn't reviewed it would've assigned two fake people to some teams. Imagine that was in a much larger data set.

Model 3: Gave me valid responses, but a hate/abuse detector that's part of the output flow flagged my name and several others as potential harmful content.

That the models behaved the way they did is interesting. The "purple team" sort of approach might find stuff like this. I'm particularly interested in learning why my name is potentially harmful content by one of them.

Incidentally I just did it in a spreadsheet and moved on. ;-)

mikehollinger··on Microsoft open-sources ThreadX
I really liked ThreadX when I was working with it a while ago. Calling it an "operating system" is charitable in the sense that it's an interrupt service routine and a bunch of functions to handle 'creating threads' and managing memory allocations.

It was super light weight and quite nice to get stuff done. You call tx_thread_create or some such and then tx_malloc and so on and then they have tx queues and semaphores and mutexes and so on.

Neat to see it's still living on.

In our case we were doing real-time power/fan/thermal control loops so we needed control over when certain actions were happening and needed to be able to guarantee that we could sense a problem and then (within X ticks) issue commands to respond, or make the system safe to prevent damage.

mikehollinger··on How to build trust as a new engineering manager
> But you weren't trustworthy.

Yup. I preface a statement about once a week by saying “there’s a difference between ‘I can’t say’ and ‘I don’t know.’ In this case, …”

It’s been useful. It helps to communicate without breaking bonds of confidentiality or trust.

It also helps people from reading too much into an off the cuff “I dunno” which is literally true. I do not in fact have knowledge of what you’re asking about. Maybe I should or shouldn’t but I can tell you I don’t know.

mikehollinger··on GPU Survival Toolkit for the AI age
Agreed. This isn’t actually that useful of a guide in the first place.

Tbh the most basic question is: “are you innovating inside the AI box or outside the AI box?”

If inside - this guide doesn’t really share anything practical. Like if you’re going to be tinkering with a core algorithm and trying to optimize it, understanding BLAS and cuBLAS or whatever AMD / Apple / Google equivalent, then understanding what pandas, torch, numpy and a variety of other tools are doing for you, then being able to wield these effectively makes more sense.

If outside the box - understanding how to spot the signs of inefficient use of resource - whether that’s network, storage, accelerator, cpu, or memory, and then reasoning through how to reduce that bottleneck.

Like - I’m certain we will see this in the near future, but off the top of my head the innocent but incorrect things people do: 1. Sending single requests, instead of batching 2. Using a synchronous programming model when asynchronous is probably better 3. Sending data across a compute boundary unnecessarily 4. Sending too much data 5. Assuming all accelerators are the same. That T4 gpu is cheaper than an H100 for a reason. 6. Ignoring bandwidth limitations 7. Ignoring access patterns

mikehollinger··on The C4 model for visualising software architecture (2017)
I like C4. I'd previously discounted the "context" diagrams that inevitably show "User -{do their job}-> System-of-Record" as useless. Now I see the point.

I'm working through a new piece of system design for a complex system using C4 with our extended teams.

In drawing the top level this time around though I realized that b/c we have a semi-disconnected mobile app (aside from a web UI) we need to degrade certain behaviors gracefully if we don't have certain results yet, either b/c we're offline, or the backend is just slow for some reason.

That rippled down into the next layer of the system as a job queue, which then rippled into the UX, because we can now say definitively which parts of the system may take an unbounded amount of time to complete a request.

I'm sure we'd have gotten to this conclusion eventually but the analysis put the issue front and center.

All because I had to draw a picture with "Remote-User --{Mobile Stuff}--> Backend."

Also whichever tool you use is up to you. I just use draw.io for example with different tabs for each layer. As we elaborate on this at some point we'll break the lower layers out to their own files.

mikehollinger··on Localtunnel – Easily share a web service on your local development machine
> Run a service in your machine in your desk, pass your IP to a friend and that was all you needed. Of course, there were security implications and IP are a scarce resource now, but I liked the way it was then.

When I was a 1337 h4x0r as a teen I ran a port scan on the IPs from my ISP (Road Runner), and found a VNC iirc that was hosting some windows system that was a quake 2 or quake 3 server.

I saw players. I changed gravity to -100, waited, then +1000 or something like that. I was amused. They learned a lesson.

mikehollinger··on Absurd Success
> Is that actually true for SSDs?

Not precisely. The logical view of a page living at some address of flash is not the reality. Pages get moved around the physical device as writes happen. The drive itself maintains a map of what addresses are used for what purpose, their health and so on. It’s a sparse storage scheme.

There’s even maintenance ops and garbage collection that happens occasionally or on command (like a TRIM).

In reality a “write” to a non-full drive is: 1. Figure out which page the data goes to. 2. Figure out if there’s data there or not. Read / modify / write if needed. 3. Figure out where to write the data. 4. Write the data. It might not go back where it started. In fact it probably won’t because of wear leveling.

You’re right that the controller does a far more complex set of steps for performance. That’s why an empty / new drive performs better for a while (page cache aside) then literally slows down compared to a “full” drive that’s old, with no spare pages.

Source: I was chief engineer for a cache-coherent memory mapped flash accelerator. We let a user map the drive very very efficiently in user space Linux, but eventually caved to the “easier” programming model of just being another hard drive after a while.

mikehollinger··on Emerging architectures for LLM applications
How prescient is the "Hidden Technical Debt" [1] paper from ~8 yrs ago compared to this? See the top of pg4 for a figure that I've personally found to be useful in explaining all the stuff necessary to put together a reasonable app using ML/DL stuff (up until today, anyway).

I see all the same bits called out:

- Data collection

- Machine /resource management

- Serving

- Monitoring

- Analysis

- Process mgt

- Data verification

There's some new concepts that aren't quite captured in the original paper like the "playground" though.

I've kind of been expecting a follow-up that shows an update to that original paper.

[1] https://proceedings.neurips.cc/paper_files/paper/2015/file/8...

mikehollinger··on An Excel error led Austria's SPÖ to announce the wrong candidate as the winner
> Is this specific to Excel?

No. And yet they still have situations where excel itself will differ from ieee 754. See https://learn.microsoft.com/en-us/office/troubleshoot/excel/...

mikehollinger··on An Excel error led Austria's SPÖ to announce the wrong candidate as the winner
> Excel still doesn't have any way of doing unit tests or confirming that the content is valid and as it should.

I make this point when chatting with new team members that say "let's just use X," without really understanding how X works, and I use Excel as the analogy. Most people know how to use an equation field. Most people don't consider writing a spreadsheet programming, but it is.

And just like other types of programs, you can have really insidious bugs, and you need to consider how you'll address those, or address changes in the underlying or internal components of the system.

A famous one is that floating point math in excel can give you different answers for "is this the same as that" depending on how you write the equation, for example.

mikehollinger··on Apple customers say it’s hard to get money out of Goldman Sachs savings accounts
JFYI: you can skip the Apple middle man and just open a Marcus.com account. It’s Goldman Sachs on the back end. Works great. No drama. Decent app. Reasonable website.

Same rates as far as I can see.

mikehollinger··on Why You Should Give Up on Your Dreams
> In the true Murican spirit you can compensate natural abilities with "hard work" -- for a time.

Yup. I’ve done this and seen it happen in others. When I started my career, I could stay up late or pull an all nighter to focus on and clutch out some project, make a deadline, or whatever.

The first time I didn’t make it, was an eye opener.

15 years later I fell into a semi-related trap. Instead of getting help, I was pouring more and more of my time (without all nighters) into a project that was continually “almost there, except for this one thing...” and got negative but helpful feedback after a meeting where we had to “reset the plan.”

Any one of those earlier problems could’ve been addressed if I hadn’t tried to shield the team from them, even if it meant changing the requirements or budget. Instead, we got to the end, and had nothing to show but a bunch of partially-working pieces.

mikehollinger··on “Tinder for Canceling Meetings”
Trello probably has more lawyers, more security professionals, and more QA than “Meeting Swipe Corporation” has employees?
mikehollinger··on Career advice no one gave me: Give a lot of notice when you quit
> 6-8 weeks is kinda weird though, unless you‘re extremely senior and on critical path for a lot of things, or you‘re using up accrued PTO.

If you're a leader in a team, definitely give more notice. It's the professional thing to do. Something that the post -doesn't- say is that you should have a transition plan written down before you tell your boss, just in case you get cut off.

Of course they can still summarily kick you out the door, but it's a chance for you as a leader to do right by the team.

mikehollinger··on Copyright Registration Guidance: Works containing material generated by AI
The issue to me here is that code is a literary work. Literary works are copywritable. Except that AI generated works without human authorship aren’t per this.

Interestingly this came up when copilot was announced and one comment or another was “this will be an interesting bit of case law.”

Well now we have to see who wants to go first and establish precedent for the next half-century. :-)

mikehollinger··on Copyright Registration Guidance: Works containing material generated by AI
I noticed this as well.

The implication to generated code is pretty astounding tho.

What about all those (c) foo 2023 prologs you see all over the place? Will there be arguments that specific lines weren’t written by a human so therefore the entirety the claim isn’t valid? Or will there be refactors to push generated code out to a well known path or construct (akin to a header / implementation or some such)?

And on top of that, what if your “secret sauce” comes back to some form of generated construct (looking at you, Whoop, with your “strain” and “recovery” scores.

Further still - can a “machine-readable medium” that was written by a machine be patented?

mikehollinger··on Forget Milk and Eggs: Supermarkets Are Having a Fire Sale on Data About You
Entertainingly I -used- to do this at Walgreens, except now they started asking to verify the zip code associated with the account. :-)
mikehollinger··on Help, Bing Won’t Stop Declaring Its Love for Me
I’d be curious to see if the engine’s ability to search / crawl content is now forming a feedback loop, where articles like this are now influencing the system’s outputs. In theory it’s stateless, except that it’s not — it’s subject to whatever web results come up when someone asks about “jail breaking” or “Sydney.”
mikehollinger··on Ask HN: Best LastPass alternatives for corporate password management
1Password has been pleasantly useful and non-enterprisey but still useful to teams and individuals.
mikehollinger··on An instruction oddity in the ppc64 (PowerPC 64-bit) architecture
> What is a hardware thread? I thought threads were a software abstraction?

This is a great question! Threads are a programming abstraction, but not just in software. Processors present an "architected state" to the programmer. Internally, the machine itself has a much more complex set of logic managing that presented state. The feature we're talking about here is generally called Simultaneous Multi-threading or SMT.[1] POWER was the first architecture to support this in commercial devices back in 2004.[2]

What's important to understand is that the processor core itself is made up of multiple units, each of which have their own internal state, task management, and methods of communication with each other. They don't just run a linear "fetch/decode/execute/store" cycle, even in a pipeline. If you were to do that, you'd end up with an extreme under-utilization of the silicon (or put a different way, someone else would figure out how to get better performance than you).

With SMT enabled, multiple real programming threads can execute at the exact time, while the core "figures out" how to share resources between them, increasing the effective throughput of the device at the expense of silicon.

With the newest POWER cores you can get up to SMT-8, and (if you want) even tune cores down to SMT-4, SMT-2, or just turn the feature off and dedicate all resources to a single thread while the operating system is running.

All of that is presented to the programmer as by way of the architected registers and behaviors in the processor. In reality there's a huge number of other registers and machinery running behind the scenes that isn't visible to user-space, the operating system, or in some cases even the hypervisor (if present).

Source: I worked in firmware for a couple of generations of these devices, was lead for a simulator team, and was functional owner for the launch vehicle for POWER7.

[1] https://en.wikipedia.org/wiki/Simultaneous_multithreading

[2] https://en.wikipedia.org/wiki/POWER5

mikehollinger··on Examples of floating point problems
Love it. I actually use Excel which even power users take for granted to highlight that people really need to understand the underlying system, or the system needs to have guard rails to prevent people from stubbing their toes. Microsoft even had to write a page explaining what might happen [1] with floating point wierdness.

[1] https://docs.microsoft.com/en-us/office/troubleshoot/excel/f...

mikehollinger··on Ask HN: Reading material on how to be a better software engineer?
Honestly pretty well.

One of our colleagues took aggressive advantage of the ping pong tables at the office (in the before times) every afternoon after lunch. I’d noticed as had some others. I had a short talk with that person about the fact that in a shared office environment he was essentially showing up post-workout for the balance of the afternoon. I actually don’t recall what changed - but we stopped noticing the issue. I think they just started playing toward the end of the day then just went home after.

mikehollinger··on Ask HN: Reading material on how to be a better software engineer?
“The Effective Manager” by the people behind Manager Tools (a great podcast) is a really, really good view of this.

Re: Communication, this book starts there and builds upon that foundation. It’s targeted at line managers, but anyone in a leadership position can and will benefit. For example, I don’t do one-on-one’s with my team. I do “Project Management 1:1’s” or “Weekly project status updates.” I don’t always give feedback the way they prescribe (because the team doesn’t formally report to me). I do, however, go out of my way to say “Just to let you know, when you do X, Y happens.” And from the book, learned that there’s a 10:1 ratio of positive to negative feedback to strive for.

Side note: Manager Tools has a great podcast series that is arranged into a “map of the universe” [1] which shows a huge list of topics, and how to discuss everything from good or bad performance, dealing with various types of conflict, to even how to structure emails or how to address a personal scent issue (which I’ve actually used).

[1] https://www.manager-tools.com/map-of-the-universe

mikehollinger··on I wish my web server were in the corner of my room
My first real job was to work with another intern to write a “tool to keep track of who’s been trained on what” since they were doing it with a spreadsheet. This was in 2001.

We wrote it on the LAMP stack which gave us the full suite of whatever you could find on a Linux CD at the time.

Fancy graphs? No problem.

Send reminder emails? Sure boss! We dutifully started hacking and testing and hacking to get that function in.

To test, we decided to send emails to jackfrost, santa, and so on from our own sendmail server to the corporate mail server. It worked fine because we weren’t spamming it, we were just sending a few messages every now and then as we debugged.

Turns out there’s really a Jack Frost that worked for us.

He was not pleased.

We were amused.

I think we apologized, and I forget how we figured out he was a real person.

mikehollinger··on Human Motion Diffusion Model
> Could make for a very interesting control scheme for an adventure game.

Zork and QWOP's lovechild? :-)

mikehollinger··on Are You Ready for the Internet? (1994) [video]
Oh absolutely! I guess I'm confusing my own point - which is / was that in some places cell towers are showing up -before- wireline connections.
mikehollinger··on Are You Ready for the Internet? (1994) [video]
1. This brought me back. When she clicked into the "mall" and the navigation was an imagemap with hotlinks [1] going off to a flower shop, I immediately thought of a website that some guy payed me to build back in the 90's when I was in high school. He bought SellMemphis.com, SellAtlanta.com, etc and wanted to create a directory for local businesses. Also - good on you, Philips Flowers, for keeping 1800florals.com in what must've been intense pressure from 1800flowers.com!

2. We still can't buy items directly in-stream. However, product placement is huge, and not just for consumer goods. Music featured in Stranger Things ends up trending on Spotify. Apple creates a playlist (in its own platform) for "Defiant Jazz," featured in Severance, also on its own platform).

3. Regulation (or lack of regulation) played out in an interesting way. We now have an interesting problem with digital infrastructure in the US anyway, where copper cables (still) only get you so far, wireless is the new expectation, and countries that -had- no comms infrastructure are by default more modern now than large tracts of the US, because they skipped past the "telephone poll" phase straight to cell towers.

[1] https://web.archive.org/web/19990208003609/http://branch.com...

mikehollinger··on Ask HN: Which books do you consider real gems in your field of work/study?
Engineer turned engineering leader here.

"Good Strategy / Bad Strategy" by Richard Rumelt - An awesome book on strategy, which explains very plainly how to construct a reasonable strategy, and see signs of bad strategy. It (among other things) dissects NVIDIA's rise in the late 2000's, and predicts (more or less) the next ten years of where the company went.

"The Effective Manager" by Mark Horstman - All the things that no one says or tells about management and communication.

"Team Topologies" by Skelton and Pais - A really good view of organizational design patterns and anti-patterns for software teams rooted in the premise of Conway's Law.

mikehollinger··on Ask HN: How do startups/lean teams successfully handle mat/pat leave?
Oh! I am just (next week) wrapping up leave! First - congratulations! My husband and I just had our first kiddo earlier this year and went through a similar "what do we do" sort of moment. We managed our time and workload by going to a "one week on, one week off" schedule, alternating between the two of us. Oh aside from all this - definitely pay money for Huckleberry (or an app like it). It will help you just manage your life. :-)

A lot of this is borrowed from Manager Tools "How To Go On Vacation" podcast. [1]

Here's what worked and what didn't for me:

What worked:

A. Name a clear "number one." [2] In my case I named two since I have a tech leadership role. I called out one of our lead architects as my "number one," and my peer director as an HR focal.

B. Make a list of everything on your plate, and your team's plate. For each item, write down a small summary of it, and then figure out who owns it. Tell that person they now own that work item. If you don't have a clear owner, the "number one" you chose becomes the owner.

C. I gave my team ~6 months (after we told family), a three month reminder, a one month reminder, and then "it's go time" when our baby showed up three weeks early. :-)

D. I took the list of projects and made a company-wide (not public) shared read only doc that had "if you're looking for X, contact ___," and put at the top and bottom "if you need anything that's not on this list, talk to ___" (your number one).

E. My husband and I both took off the first two weeks, then started the "on/off" schedule. I set two different out-of-office notifications when I'm out. The external one said I was out for paternity leave, promised baby pictures, and had the two contacts (my number one and director). The internal one said the same as the external, but also had a link to the "coverage doc" that explained project-by-project who to reach out to. It also said in blunt instructions: "No news is good news. Please make a decision without me. If you can't, or need help, contact (number one). If you need air cover, contact (director). If we will do irreparable harm to the business or you have a business continuity or ethics concern, text me. I won't be checking email or slack."

F. Turn off notifications / uninstall email / slack etc on your phone. Enjoy those first two weeks. :-)

G Book a "catch up" meeting ahead of time with your number one and any key contacts for when you return.

J. Book 1:1's with key contacts for every week you return from leave. DEFINITELY DO THIS. YOU WILL BE OUT OF TOUCH. THIS IS A WAY TO LESSEN THE PROBLEM AND MAINTAIN RELATIONSHIPS.

Some warnings:

M. After a few weeks we "got the hang of it," and caved on the "I won't check in" rule and started lurking on slack. Once people figured it out they started slacking me, which started a feedback loop. Don't start the feedback loop.

N. A few times I tried to work a day on my "off" weeks because it felt like "I got this!" or something urgent came up. It wasn't a good use of time or a good idea. I was out of sync with the team, and about 1/3 of the time behind b/c I hadn't scrolled through -all- of the slack channels to see that issue X had already been addressed.

O. You're pushing work down. That means that folks are going to have to push work off of their plates themselves. They should not turn around and delegate something you are delegating to them. You should expect them to take a task from you, and then stop doing one or more other tasks by either delegating those "smaller" things to someone else, or dropping them to the floor ("Mike's on paternity leave, I'm covering for him, sorry I can't do your TPS report" is a fair answer). Emphasize that delegation model.

[1] https://www.manager-tools.com/2015/05/how-go-vacation-part-1

[2] You should have a lieutenant / number one anyway just for succession planning / hit-by-a-bus reasons.

← PreviousPage 2 of 10Next →