subhobroto

OK, then what's my plan when coyotes and mountain lions attack my child and I on our regular walks on rural property? As we build more housing and cities close in, these wild animals are being run out of their natural habitat.

Is the answer "dont be on rural property!" or are there real practical solutions?

> but I think bombs and bioweapons and etc are very bad examples for you here

Are there better examples?

Also, I for one don't undermine the drive and tenacity of an evil person and to what extent they are willing to cause harm.

> Anyone that knows even a little bit about guns knows that this is utter nonsense

Most people in California who vote on these matters have not held a BB gun, let alone a semi automatic.

They have 0 idea that you just cannot buy actual guns from a grocery store in California anymore!

They think you can just buy a gun at Walmart like you can buy a can of Coke. I was able to pull up clips made in 2023 and 2025 that were literally claiming that. Hasn't been true since atleast 2009, likely even earlier.

A few years ago a local Walmart was clearing our their air gun and rifle selection after there had been a shooting on the east coast that was all over the news. Since ammo have become really expensive, I bought out the whole shelf of air rifles so I could continue to target practice with a focus on my breathing.

People called the cops on me. Multiple people verbally abused me as a gun nut and recorded me buying them on their phones. I had air guns - *children* *toys*. They thought it was the real deal!

The local sherrif's department received nearly a 100 calls that hour when we spoke. When I asked them why they even bothered to turn up because they know no Walmart in a 300 mile radius have ever sold a rifle in the last 20 years as was described to them over the phone, they just shrugged and said "politics".

- can you build bombs to blow up an apartment complex full of 1000s of people?

- can you poison the water supply of an apartment complex full of 1000s of people?

- can you drop a harmful substance using a $50 drone onto an open area where of 1000s of people have congregated?

You cannot defend yourself from a hungry coyote or surprised mountain lion with a dollar bill but you can certainly protect yourself or your child from one with a gun

> if you can make a gun, you can certainly make ammunition

theoretically true but having re-sleeved ammunition, the chances of injury is tremendously different. That said, a lot of people in California are having to resort to re-sleeving ammunition, not out of choice but because for all practical purposes, California has made buying ammunition impossible.

While you can crawl and bite your way through getting a horribly castrated gun in California, the real struggle begins buying affordable ammunition.

For regular people to own a gun that you can actually use in California, (not LEOs or certain other people), you either needed to have inherited them or bought them from the cartels. Otherwise you own something of limited use that insanely expensive to operate.

I don't understand the problem solving mindset that thinks banning guns would solve the problem of a person intent on causing harm.

In the U.K., where I feel guns are only showpieces (do even cops have them?), stabbing is a known problem.

In India, where ammo is way more expensive than machetes and knives, people are literally murdered with them.

The only argument I can understand, when it comes to banning guns, is that it reduces the blast radius that an evil person can have.

So what's next, lock down the air, radio, roads, internet, water, food supply chains because these are all attack vectors?

If that's the proposal, what's my plan when coyotes and mountain lions attack my child and I on our regular walks on rural property?

> There really is no cost difference between writing shitty code and "perfect" code now; its just a matter of how good you are at describing "perfect" to the LLM.

The consequent is at odds with the antecedent. It's a performative contradiction (if the output were truly "free", the skill of the operator would be a zero-value variable - yet, by requiring skill, you acknowledge a cost) as I prove below

> The cost of "perfect" is only perhaps a few fractions of a cent higher than shitty.

Is your cost model accounting for the cost of specification, of review and additional cycles required if review fails or the specification itself needs to be adjusted?

> If we simply tell the agent these rules ahead of time, we get "perfect", consistent code each time

No, in the simplest case, your cost of perfection is simply moving up the chain of abstraction from implementation (coding) to design and specification. In reality it also splits and moves a part of that cost downstream to verification.

This isn't some special, magical insight I have, I'm reiterating Tesler's Law right back to you.

I also encourage you to read software history - for decades it has been trivial to split out perfectly working CRUD from an ER and UML diagram, no LLM necessary. The insight is understanding why we continue to hire cheap human labor to spit out CRUD instead of using those tools.

The cost of software is, and always has been, in the figuring out the intent, not the generation of syntax.

I wish pg was more active on HN - I expect this is one of the reasons why he wanted founders to have and share the painpoints of their (potential) customers. Figuring out the intent is expensive. Mistake the intent and the best case scenario is a pivot.

> Instead, the right train of thought is: "what would perfect code look like?" and then meticulously describe to the LLM what "perfect" is to shape every line that gets generated.

I don't think there's perfect code.

Code is automation - it automates human effort and humans themselves have error, hence not perfect.

So as long as code meets or exceeds the human output, it's "good enough" and meets expectations. That's what a typical customer cares about.

A customer will happily choose a tent made of tarp and plastic sticks that's available at their budget, right now when it's raining outside, over an architectural marvel that will be available sometime in the future at some unknown pricepoint.

Put another way, I don't think if you built CharlieAI CharlieGPT today, where the only differentiating factor over ChatGPT was that CharlieGPT was written using perfect code, you would have any meaningful edge.

I am yet to see any evidence where everything else being equal, one company had an edge over another simply due to superior code.

Infact, I have overwhelming evidence of companies that had better code succumb and vanish against companies that had very little, if any code, because those dollars were instead invested in better customer discovery, segmentation and analytics ("what should we build?", "if we did one thing that would give our customers an unfair advantage, what would be that thing?")

Software history is full of perfect OS, editors, frameworks, protocols that is lost over time because a provably inferior option won marketshare.

You are using a software controlled SMPS to power your device right now. You have 0 idea what the quality of that code is. All you care about is whether that SMPS drains your battery prematurely and heats up your device unnecessarily. It's extremely unlikely that such an efficient, low overhead control system was written using well abstracted modules. It's more likely that control system is full of gotos and repeated violations of DRY that would make a perfectionist shudder and cry.

Michael, Thank You for reading and responding back.

We disagree on the diagnosis and path forward, not the symptoms. Let me explain:

> I think the current system gives H1B employers way too much leverage over H1B employees and degrades the job market for everyone

Correct and it's by design. The overhead of H1B needs to be in the black when it hits the bottomline - the H1B isn't a charity auction to take brilliant engineers from developing countries and move them into the U.S. - it's to ensure the companies turn a profit on it.

> I think the H1B system is a net negative for the US economy, and it disproportionately hurts small businesses

Correct again on both accounts. While it is a brilliant solution 1% of the time, it's misused 99% of the time.

> It sounds like your argument is that H1B doesn't matter because the companies using it aren't really innovating and so they'll naturally be outcompeted by smaller businesses who are too small to take advantage of the H1B system. Is that correct?

Correct again

> you're saying you advocate free market solutions and that's why we shouldn't mess with the H1B system. The H1B system is the opposite of a free market solution. It's extra regulation that we'd be better off without

Correct again

Where I disagree with you is when you said we need to add more regulation to the existing H1B system. To me, and this is not something I'm hearing for the first time, it sounds like a band aid on top of a bunch of band aids 12 feet deep.

The H1B system was wonderful when it was initially implemented. The U.S. was undergoing massive technological shifts leading to tremendous supply shocks - just like how we are struggling to purchase GPUs, RAM and SSD today. As much as we disagree, history has proven repeatedly this shock will pass (unless the market is distorted by new regulation).

However the H1B has long since distorted into a geo-arbitrage, QoL and CoL hack. Even the current administration's $100k fee is a bandaid. Just the fully loaded cost of a 4-6 year domestic education is more than that, so the domestic supply is being destroyed.

Ideally what should happen is we should decommission the H1B completely, no ifs and buts, and have a "cool down" period where we notice what impact it actually has on the domestic demand. I do appreciate that this isn't accounting for the case where there are only a 1000 people worldwide who know how to train an LLM from scratch, or, 1000 toptier cardiologists worldwide that we would like to attract - I'm very sure we will figure something out for them but I argue we need to discover why our domestic supply is lacking in the first place instead of continuing to rely on band aids.

The U.S. today seems to rely on stents to save it from heart attacks. We should probably take a strong, hard look at the diet and lifestyle choices instead of continuing to rely on stents as a savior. It was necessary when we had an emergency, but a sustained reliance on emergency intervention points to underlying structural issues.

> One point I didn't see on there is lack of clarity of the objective of the JIRA Epics and Stories and verify that those underlie actual customer painpoints.

Interestingly, the author addresses this in a different post: https://yusufaytas.com/what-good-execution-looks-like/

I quote:

> Clarity of Direction

> If people don’t know where they’re going, they create their own destinations. It’s as simple as that. That’s when you get alignment issues, rework, and endless questions

> ... You can start simple. Here’s what matters. Here’s what doesn’t. Here’s where we are going. Here’s why. When you miss these, inevitably, your execution cripples down.

> nobody will ever have to maintain it, so it doesn't matter

I'm curious about software that's actively used but nobody maintains it. If it's a personal anecdote, that's fine as well

Great points about Over Engineering. No one ever seems to ever acknowledge how very successful Craiglist continues to be with a relatively simple stack because they really understand their customers.

One point I didn't see on there is lack of clarity of the objective of the JIRA Epics and Stories and verify that those underlie actual customer painpoints.

It seems like for every 10000 hyper scalable, cross region redundant clientside SPA MVPs that are blasted over Facebook or Google or PH Ads, there's one concierge MVP where the founders talk to you, find out pain points, validate them manually by hand and then grow organically.

In certain cases, it's even possible that no one in the engineering team had the confidence to say "Do we really need to build any of this at all to see if our startup has teeth?"

What would be interesting is to speak to the technical teams of companies like Quibi and ask them questions like:

- How are we testing whether this actually solves a real customer need?

- Who validated the need for Turnstyle Technology? How? When you launched, did you track how many were leaving the platform if it was turned off for them?

- Are there specific hurdles to serving short-form content that make a company built around it valuable?

- Are there ways we could help the leadership validate their market hypothesis by writing the least amount of code?

I imagine if any engineer was being interviewed to be hired at Quibi and asked them questions like these, their interview would be prematurely terminated.

It's widely accepted that both interviewing and working at a company especially at startup, requires suspension of disbelief, but I sometimes wonder if some of us should just say "No, we shouldn't do any of these JIRA tickets. There's a much simpler way of getting this done. Sal, Jane, Lin, come with me and let's talk to the customers and find out if this would even move the needle for them".

If everything around is smoke and scary shadows, then who knows whether that rustling in the shadows is a bear or a gopher? Over Engineer assuming the worst. Then we would be ready for a sudden million customers filing their taxes on a Friday evening in June.

The peace and fulfillment of knowing what the customer really is willing to pay for and building towards that is sublime. It's so addictive that there's absolutely no going back

My response is the opposite of ad hominem. It's me noticing and acknowledging that a large group of people comment on energy policy who understand neither energy nor policy but feel confident to both vote and comment on energy policy because they have been brainwashed.

> a dollar spent subsidizing utility solar will go much, much, further to decarbonizing the grid than a dollar spent subsidizing rooftop residential solar

That's completely, trivially provably wrong.

For one, rooftop residential allows decentralization and redundancy. Customers can have their house continue to run while the grid as a whole is down.

Utility solar absolutely does not - solar here is just another source of energy into the centralized grid. From a physics and math perspective, that source of energy could very well have been a coal plant. It doesn't have the same decentralized and redundancy benefit as a 10kWh PV array and a 30kWh battery on a home where the home no longer even needs the grid anymore.

Any rational consumer would appreciate having power than not - We cannot argue physics and math or consumer demands - that's just insanity.

A person who repeatedly argues against this basic math doesn't have the best interest of the consumer in mind. Further, when there's a group of people who repeatedly argues the same illogical swill using the same phrasing as the others, it's a cult.

I could go on and on but I have learned that arguing with brainwashed people takes a lot of time and effort and most of the time, in the end, it ends in a stalemate. Most of the time, it reenforces them they are right and the other side is ignorant and "doesnt get it". No thank you.

On this very thread a person who actually designs and implements grid projects agreed that the only objective utility solar serves are those of the utilities themselves.

The reason why this person is provably correct is because you could come to the same conclusion from first principles as well. To most people, the above statement is a "duh!" moment.

They gave solid, clear, objective examples of how utilities have other much higher priorities than ensuring consumers have access to the most affordable, reliable, resilient power.

They further gave evidence and insight into how the way utilities fund their infrastructure projects actively make them opposed to what's good for the consumer, because decentralization and redundancy would collapse the value of existing collateral.

Now that's a high quality, high value, high merit discussion.

When someone comes and lectures me about how my dollars should go towards initiatives that absolutely do not help me, rather makes me even more dependent on a 3rd party that doesn't have me as their first priority, I dont care about what they have to say especially when I have options available where my dollars directly go towards initiatives that absolutely do help me.

The most generous interpretation I have of their motives is that they are brainwashed.

I would encourage beginning from first principles, gaining clarity into what the final objective is - to provide affordable, redundant, reliable, resilient power to the consumer or to ensure the utility has smoother operations and a tighter grip over the consumer and ensure the consumer is completely dependent on the utility?

If the brainwashing isn't complete and there's a tiny chance of a breakthrough, maybe changing domains would help?

What's better for a local economy that has willing, capable backyard farmers:

- allow them to sell their product that meets all regulatory requirements, to other willing consumers at the same market clearing price or

- destroy them completely and remove any incentives for them to grow, in favor of having a massive, centralized farm?

> Thanks for the kind words!

You deserve and earned them!

> There's so much frictional cost to hiring someone on an H1B visa (legal fees, admin overhead) that it's not practical if you're only hiring 1-2 employees

Very true but as you saw in my email, I have extremely experienced friends back in India who I have been able to hire as contractors without issue. No H1B - just plain old Slack, email and Forgejo. The playbook for asynchronous work is well tested and debugged by now. 2019 was a blessing.

I will concede, this doesn't work for every company - a hardware or biotech company definitely would appreciate people all being together in the same physical lab, in which case I hear you!

> the government designed it to be employer-agnostic

... but the government cannot be employer-agnostic, Michael.

The government is not an impartial, unbiased mainframe running in a DC somewhere. It's a group of people accepting and pushing policy who can be influenced, just like I am influencing your today, and you, me.

As an SMB and bootstrapped founder, you then have to choose between spending your time and efforts on being at the influencing table vs making actual design and business decisions at your startup the moment you yield influence to this group.

The bigger business simply doesn't have to make that decision. So don't help tip the scales further against yourself and SMBs like you.

That's one of my points that I was hoping to discuss in our email - that involving the government adds further overhead, resistance and expense into the system, so we should exhaust all other options before we even consider it. I personally have never seen an option that needed government intervention that couldn't be solved by the free market. I don't work in healthcare or education or finance - maybe those do require government intervention - I am entirely unfamiliar about those domains and not talking about them.

The other interpretation of being employer-agnostic is that the H1B isn't tied to a "sponsoring company" and doesn't require any of the transfer shenanigans. Sure, but the issue it isn't that way is because it's a rare "dual intention" visa in that, you are a non-immigrant who can become a citizen through the H1B. This was a feature added to the H1B to entice top quality talent. The problem with making the H1B employer-agnostic is that now you can I can start a perfectly legal, fantastic lifestyle businesses hiring H1Bs, petitioning for their greencards and immediately letting them go. As long as they can figure out a way to eat and sleep, they can now become citizens. So for it to be employer-agnostic, we need to remove the "dual intention" - the very carrot employers use (if you tough it out through all those JIRA tickets, you'll get to be a citizen!)

> I just mean on a large scale, the companies that use H1B visas will generally outcompete the ones that don't.

This is where I continue to push back. I was hoping to discuss over email but do you feel you could have built TinyPilot at either MS or Google, not as a side project but as an official product offering? I don't want to get too tied up into the specific features that TinyPilot offers - I'm using it as a proxy for a very useful, innovative product that provably solves real customer problems.

At the large scales where H1B makes sense, you as a major decision maker at the company wouldn't allow a worker with a risky status like the H1B be responsible for high impact, meaningful pieces of work. Actually, forget H1Bs - at the large scales where H1B makes sense, you would simply not entrust a single individual, H1B or not, with high impact, meaningful pieces of work.

If we disagree on this take, please say so - I am here to learn and listen.

The original intention of the H1B was to handle temporary supply shocks in knowledge work while the U.S. slowly fixed those supply constraints on its own.

If avocado toasts became an overnight sensation, the H1B was a way to provide breathing room to local avocado farms so the demand could sustain or grow (and not collapse) while they came up to speed to meet that sudden demand.

The H1B wasn't designed to be a way to absolutely wipe out local avocado farms because it's cheaper to just import avocados from Mexico.

The H1B has completely diverged from that and going in the opposite direction where it's actively and negatively impacting domestic markets. Massive corporations in Asia have grown whose sole business model is exploiting this geographical arbitrage and nothing else.

What piece of critical, useful software that has had a mention on HN can you name or recall that has come out of these many mutibillion dollar outsourcing giants?

A $60k/yr salary as a resident doctor is fantastic if you did most of your education in Asia but if you attended medical school anywhere in the U.S. and didn't have a 100% scholarship, you're starting your life off in crippling debt.

During COVID, there was an explosion of domestic coding bootcamps to address the supply constraints - this is precisely the kind of domestic corrections we, as the U.S. need to encourage and develop local talent, get them educated and motivated about tech, but these bootcamps require an investment that in Asia covers education, boarding and lodging without any scholarship. There's just no competition when it comes to cost. We in the U.S. have an extremely high quality of living and our CoL reflects that. As I wrote in my email, things we take for granted here - running water (not potable, just water that you could water your plants with), 24x7 electricity and internet - these are still unavailable where I was born, so of course, the CoL is cheaper. Way cheaper.

One might say, "OK then, free markets for the win" - that itself is a separate debate on its own.

> Ultimately, companies who use H1B visas will outcompete companies who don't because the H1B system gives them cheaper labor costs. The solution has to come at the regulatory level.

I have utmost respect of your work, a customer of your fantastic product and have been meaning to reach out to you for a while (infact I learned about Cory from your blog in 2021) but I had to push back hard on this.

TinyPilot didn't happen in India nor China. I can argue it would have been cheaper to build it at any one of those countries but you know much better about it than I.

Labor costs only matter when you're selling an absolute commodity that has no edge than price.

Of all the people I would have expected to say that the solution has to come at the regulatory level given the experience, success you've had, with your transparency in how your company was doing, I am utterly surprised it was you.

I am more than happy to continue, reached out - I just wish our initial email would have been way more pleasant!

> Unfortunately, the Chinese were right

About what? Are you familiar with how the life of a Chinese salaryman is going about for the last one year while us in the West are trembling in our shoes about how open weight Chinese models are threatening SOTA frontiers?

> Large companies dictating economic companies is anti-innovation

Yes but what's the solution? To pass even more regulation against the large companies and make them behave?

> in Europe where you can see a clear gap between countries that are built on non-zero sum systems which are thriving, everything just works

Some concrete examples?

> They are focused on giant transmission line projects and hydro

Is that because of the scale they need to achieve to support the investment?

> until I demonstrated that the project would never feed back to the grid

Financial greed aside, and I mentioned this previously, feedback at a large scale isn't free especially if the impedance of the grid cannot be predicted - the frequency or voltage or both would spike. Monitoring for these conditions are expensive and addressing them is even more expensive - the cheapest solution is you do a shutdown until things stabilize but this is kinda catch-22 because that itself might have its own cascading effects.

Let me ask you this - if you were to update a local neighborhood (like a block or two of 1000 homes) distribution station, that all have their own solar and battery, where the homes could independently power themselves for a day - what changes or upgrades would you make to ensure they can share load for that one day when the larger grid is suffering an outage?

Now would that cost and complexity be lower or higher if instead, nothing was changed at grid scale at all but each of the individual 1000 homes doubled their own capacity (let's say 30kWh a day if you're OK with that)?

> but making that change would require a massive restructuring of electricity and natural gas utilities. Such a restructuring would revalue the debt that backs the existing infrastructure. This would be a great thing for the average person but not for the people currently in charge of regulating what types of systems are allowed

fair. I appreciated the insight in your original message - datapoints from industry insiders like yourself do help people like I gain some understanding of why we have to go in alone on this and not wait

> Public utilities and pension funds hate green energy, they are a major factor in west's pathetic performance when it comes to solar adoption vs China

No this statement is absolutely wrong. Here's why:

> west's pathetic performance when it comes to solar adoption vs China

China is dominating energy because the CCP doesn't care what their citizens think. They need energy and they are doing everything they can do to get it. They will put you behind bars at best or kill your family and demolish your house if it gets in the middle of a power line trench. For China, energy isn't a "nice to have" - they realize it's essential and they won't stop until they get there.

China is the person out in the mountains being chased by a hungry bear while we in the west is the person sitting in their air conditioned room debating whether to drive or take an Uber to have a drink with buddies.

News came out last week that you can buy a Chinese hypersonic missle for $100k - you can't even build a little two car garage where I am for double that price.

> Public utilities and pension funds hate green energy

Pension funds don't care whether energy is green or orange. What they hate are the horrible returns affected by all the stealing and grifting that happens in the name of "green energy".

Public utilities (atleast in the jurisdictions that I am aware of) love any infrastructure work - they are guaranteed a 10% ROI by the government on any approved infrastructure work they do. If you could work with them to build infrastructure to cremate just newborn kids and get it approved by the CPUC, they will happily start work on it tomorrow. The reason why they hate green energy is because after they've made their 10% ROI, they are now stuck with a power source that costs them more than their non-green sources and that hurts their razor thin margins.

However, as the customer - I don't care either about what public utilities and pension funds hate or don't.

What I do care about is having affordable and reliable power and I absolutely can get that with my own solar panels and batteries. The fact that it's green is a happy sideffect for most.

The reason why every home in the U.S. isn't overflowing with solar panels and batteries is because of regulation and government shenanigans making retail costs really high. Average people in Pakistan, South Africa and Lebanon certainly power their whole homes with solar panels and batteries but their governments don't have nonsense taffifs and fees on Chinese solar equipment.

> It turns out that a big problem is that whenever we install local generation it costs utilities a ton of money

So a question:

- Lets hypothesize that distributed, decentralized systems cost way more than centralized systems

- If you agree with that hypothesis, can we next hypothesize that building a distributed, decentralized system that can support power on one block and can allow it to continue to stay on while the "central feeder line" (please tell me the proper word for this made up word is) to all the blocks is down, because that one block has a local distributed, decentralized power source, is of value to the community?

In the past, commercial factories were the only places that could afford this kind of redundancy but it feels to me, thanks to crashing prices of solar and batteries (I could never have imagined 12kWh brand new LFP could be purchased for $2k), this level of redundancy is now very much realistic at the consumer, residential level. It just doesn't work locally today because the utility poles lack the smarts to do the isolated switching and safe islanding. For example: one unsettled question today is if a lot of customers on one such island are on solar and the grid is down, how do we safely supply power within nominal specs to the whole of the island - but this isn't a physical unknown, we know how to solve it. It just is lacking implementation.

> These costs, which include debt service, maintenance, upgrades etc amount to 5-7 cents/kWh. Whenever you generate your own energy you cost the utility 5-7cents/kWh that they have to pay regardless of your usage

Capitalism has repeatedly proven its ability to cut costs down while improving QoS. I realize you really believe in the numbers you have been provided - that it costs a utility 5-7cents/kWh that they have to pay regardless of my usage, but before SpaceX, it used to cost multiple millions of dollars and years of planning and design to launch one rocket.

> Besides direct subsidies, wealthier home owners have often been paid the retail rate for the electricity they sell to the grid. This causes higher electricity bills for those in apartments and those who can't afford to put panels on their roof

I don't think you thought this up yourself, so I won't blame you for it, as this exact, word for word swill is mindlessly repeated by a lot of people, so thats ample evidence of brainwashing going on.

The subsidies and retail rate (both of which have been murdered by now thanks to swill like this) incentives were not a sneaky reverse welfare program snuck in by the wealthy.

They were infrastructure incentives for people who could afford to make those infrastructure investments.

Investments have always required incentives and a positive ROI. You don't put money into your 401k, Roth or HSA because you expect to lose money in 20 years.

The goal of solar subsidies was never some sneaky wealth redistribution with unforseen sideeffects but rather to rally support from the private industry and wealthy homes to spearhead rapid decarbonization, energy independence, and grid decentralization.

A single mother treading water, barely being able to afford groceries isn't your persona for actually making rapid decarbonization, energy independence, and grid decentralization happen - however, the wealthy that you so despise of, certainly put a 10kWh (sometimes more) PV array on their 3000 sqft rooftop and actually feed power to the grid that was reeling under tremendous growing strain.

People hanging portable solar panels from the balconies of their apartments barely make power to run their kitchen fridge so that's out as well.

Mom and pop landlords and corporate run apartments aren't going to put solar for their tenants because they are not legally allowed to sell power above utility rates while they don't enjoy the 10% guaranteed ROI that utilities get (which is where utilities actually make their money), so that's out too.

This makes me sad - We could have had a future where the grid was fully decentralized, where our single mother neighbor would never had to worry about the lights getting turned off even when there was a downed power line or wildfire or a snowstorm turning down power lines half a mile away, where she could plug in her EV into my shed instead of having to drive miles away to a crowded charging station.

We are numbers people here - so here's a numbers perspective:

If I had taken the same money I had to spend on a "grid compliant" installation (so I could connect all of this to the grid) and put it into the SNP500 instead, I would never have had to worry just about a power bill (as bad it is - $0.60/kWh) but also my inflation adjusted grocery bills for the rest of my life.

If you reached out and notified him, Thank you. I hope he has time to revisit - had a few more followups. Cheers!

Dr. Hipp,

> I dare say you will be hard pressed to find a dataset of significant size that doesn't have at least one invalid entry somewhere

I agree. In my experience, and you've forgotten more than I have learned, the mark of a good data engineer is how they account for invalid entries or whether they get `/dev/null`ed.

> Checking the datatype is not the same as validating. There is lots of data out there that is invalid, and yet still has the correct type. In fact, that is the common case. Increasingly strict type rules will not fix that.

I am having a hard time letting go of this opportunity to learn from you, so in case you have time and read this again - When you say "There is lots of data out there that is invalid, and yet still has the correct type", I read "type" as "shape" or "memory layout" and "invalid" as "semantically wrong".

So, is a good example of this a value of `-1` for a person's age? The database sees a perfectly valid integer (the correct shape), but the business logic knows a person cannot be negative one years old (semantically invalid).

In that case, to be explicit, `0` or `14` is a "valid type" for an age (usually integer), but completely invalid data if it's sitting in an invoicing application for an adult-only business?

Again, thank you for your time and attention, these interactions are very valuable.

PS: I'm reminded of a friend complaining that their perfectly valid email would keep getting rejected by a certain bank. It was likely a regex they were using to validate email was incomplete and would refuse their perfectly valid email address

First off, I am so glad the famous "HN conjure" actually worked! My "if Dr. Hipp was reading this thread" was tongue in cheek because on HN it was extremely likely that's precisely what would happen. Thank you for chiming in, Dr. Hipp - this is why I love HN!

So, in case you missed it, you're responding to Dr. Hipp himself :)

> I don't understand what that has to do with rigid typing.

Now I would like to learn a bit from Dr. Hipp himself, so here's my take on it:

Scripting languages (like my fav, Python) have duck or dynamic typing (a variation of what I believe Dr. Hipp, you specifically call manifest typing). Dr. Hipp's take is that the datatype of a value is associated with the value itself, not with the container that holds it (the "column"). (I must say I chose the word "container" here to jive with Dr. Hipp's manifest. Curious whether he chose that word for typing for the same reason! )

- In Python, everything is fundamentally a `PyObject`.

- In SQLite, every piece of data is (or was?) stored internally as a `sqlite3_value` struct.

As a result, a stack that uses Python and SQLite is extremely dynamic and if implemented correctly, is agnostic of a strict type - it doesn't actually care. The only time it blows up is if the consumer has a bug and fails to account for it.

Hence, because this possibility exists, and that no objective research has proven strict typing improves reliability in scripting environments, it's entirely possible our love for strict types is just mental gymnastics that could also have been addressed, equally well, without strict typing.

I can reattempt the "HN conjure" on Wes McKinney and see if this was a similar reason he had to compromise on dynamic typing (NumPy enforces static typing) to Pandas 1.x df because, as both of them are likely to say, real datasets of significant size rarely have all "valid" data. This allows Pandas to handle invalid and missing fields precisely because of this design (even if it affects performance)

A good dynamic design should work with both ("valid" and "invalid") present. For example: layer additional "views" on top of the "real life" database that enforce your business rules while you still get to keep all the real world, messy data.

OTOH, if you dont like that design but must absolutely need strict types, use Rust/C++/PostgreSQL/Arrow, etc. They are built from the ground up on strict types.

With this in mind, if you still want to delve into the "Lack of rigid typing has two issues" portion, I am very happy to engage (and hope Dr. Hipp addresses it so I learn and improve!)

The real world is noisy, has surprises in store for us and as much as engineers like us would like to say we understand it, we don't! So instead of being so cocksure about things, we should instead be humble, acknowledge our ignorance and build resilient, well engineered software.

Again, Dr. Hipp, Thank you for chiming in and I would be much obliged to learn more from you.

> Why is it a problem that it allows data that does not match the column type? SQLite is intended for embedded databases

I'm afraid people forget that SQLite is (or was?) designed to be a superior `open()` replacement.

It's great that modern SQLite has all these nice features, but if Dr. Hipp was reading this thread, I would assume he would be having very mixed feelings about the ways people mention using SQLite here.

> I understand why they wouldn’t, but STRICT should be the default.

No wait, what do you mean?

As I mentioned at https://news.ycombinator.com/item?id=47619982 - your application layer should be validating the data on its way in and out. I mention the two reasons I use for DB fall back

> caught a bug where I was accidentally inserting the wrong type and it just silently worked in regular mode

Typically one would design their "DTO"s to catch such errors right in the application layer, way before it even made it into the DB.

Different people call this serialization-deserialization layer different names (DTO being the most ubiquitous I think) but in general, one programs the serialization-deserialization layer to catch structural issues (age is "O" instead of 0).

The DB is then delegated to catch unchanging ground truths and domain consistency issues (people who have a DOB into the future are signing up or humans who don't have any email or address at all when the business needs it).

In your case, it's great the DB caught gaps in the application layer but one would handle it way before it even made it there.

The way I think about DB types are:

1. unexpected gaps/bugs: "What did I miss in my application layer?"

2. expected and unchanging constraints: "What are some unchanging ground truth constraints in my business?", "What are the absolute physical limits of my data?" - while I check for a negative age in my DTOs to provide fast errors to the user, I put these constraints in the DB because it's an unchanging rule of reality.

Crucially, by keeping volatile business rules out of the database and restricting it only to these ground truths, I avoid being dragged down by constant DB migrations in a fast-evolving business

> All of the listed features except for strict tables and generated columns have been in SQLite for 10+ years, and those two are certainly not new

Correct.

As I mentioned in my GP, these features have been around a long, long, long time and in the current age of AI that would happily tell you these features exist if you remotely even hint at it, I would assume one would really have to go out of their way to be ignorant of them.

It doesn't hurt to have an article like this reiterate all that information though, I just would have loved the same level of effort put into something that's not as easily available yet.

A gap exists, and the article doesn't mention at all, about actually configuring solutions like litestream, rqlite, dqlite (not the same!) to build HA, cross region replicated SQLite using purely OSS software provided you architect your software around it. Now that would be a really good "Modern SQLite: Features You Didn't Know It Had" article!

Another interesting discussion point is how far self hosted PostgreSQL and pgBackRest can get you to a near-zero data loss high RPO, RTO setup. It just blows my mind that we can self host all this on consumer grade hardware without having to touch a DC or "the cloud" at all.

> Does anyone know good resources for this to learn and draw inspiration from?

Is there a reason why something more custom built, like ParadeDB Community edition won't meet your needs?

I understand you're speaking about SQLite, while ParadeDB is PostgreSQL but as you know, it's non-trivial to get good search quality, so I'm trying to understand your situation and needs.

None of these are news to the HN community. Write-ahead logging and concurrency PRAGMAs have been a given for a decade now. IIRC, FTS5 doesn't often come baked in and you have to compile the SQLite amalgamation to get it. If you do need better typing, you should really use PostgreSQL.

However, I will concede, and the article doesn't mention at all, far less are aware that you can build HA, cross region replicated SQLite using purely OSS software provided you architect your software around it. Now that would be a really good `Modern SQLite: Features You Didn't Know It Had` article!

Another interesting discussion point is how far self hosted PostgreSQL and pgBackRest can get you to a near-zero data loss high RPO, RTO setup. Its simply amazing we can self host all this.