prions

I'll write a charitable response to your flippant comment.

Living in a city has more noise than outside of cities, but that doesn't mean that people deserve to be exposed to noise levels that are harmful to their health. This also ignores people who are born in the city, and can't leave it for various reasons.

Various laws on noise levels have been laws in NYC for decades at this point because it's a public health and quality of life problem.

It will end when people like the author drop their victim mentality and start putting their own ideas into practice. It seems that this person is content with yelling on the sidelines about how good and efficient things used to be.

Meanwhile, the shitty tech is winning. According to this author, bad code and bad tools and bad frameworks are reigning supreme over real engineering. Why?

"The situation is really bad for the industry." Also, why?

People with the mentality like the author don't actually want to build things. They just want to sit and complain. Their sense of righteousness and victim mentality gives them more pleasure and validation than actually engaging with the "modern" tech world.

Some other articles by the same:

- "Using a framework can make you stupid!"

- "So-called modern web developers are the culprits"

- "One sure way to determine if you are stupid"

- "SQLite the only database you will ever need in most cases"

- "No, your website is not a web app even if you call it so"

Just taking a look at his youtube channel. Why is he so hot on culture war issues now? It seems that right to repair has taken a back seat to his polemics about covid lockdowns, vaccine skepticism, and homeless people.

So you'd rather they manipulate the data and potentially mislead users? Sounds like a slippery slope to me \s

Look at some objective measurements segmented by race, such as earnings or life expectancy, and revisit whether you think race has "suddenly" mattered

Youre underestimating the value of a college degree. Research like [1] shows that degree holders have significantly more lifetime earnings than high school. You also miss how the democratization of education has benefited our society and helped lift millions out of subsistence lifestyles and class lock-in.

"This world needs a lot more new plumbers than new liberal arts majors"

This is a completely subjective and politically charged statement. I think the world needs more informed and educated critical thinkers. Its curious how the 90s led to a collapse in liberal arts departments and enrollments, and 20 years later we're now in an era of disinformation at massive scale.

1. https://www.ssa.gov/policy/docs/research-summaries/education...

Because life under Epics vision of undercutting and delaying releases to Steam is so much more egalitarian and noble

People are rooting for Sweeney due to his alpha nerd rage righteous more than how “good”Epic is to their customers

(amateur painter) Generally, shadows have a lower chroma and value than the color of the object.

If I'm painting a purple sphere, then the cast shadow will be a much darker value of purple.

It's really easy to take a both-sides stance here, but the last presidency (ending in that same president causing an insurrection attempt at our capital) shows that one side is clearly more dedicated to authoritarianism than the other - socially, politically, and legally.

People really seem to be hung up on equivocating a sitting president attempting to overturn an election with people on twitter saying mean things.

Thanks!

Senior Data Engineer. Interested in DE/ML Eng jobs but open to exploring other work.

  Location: New York City
  Remote: Yes, but prefer companies with a local presence
  Willing to relocate: No
  Technologies: Python, Airflow, Flink, Java, SQL, K8s, Spark, Hadoop, Kafka
  Résumé/CV: Work in progress! Just beginning my search
  Email: In my profile

This is really cool! I'd love to seed the GAN with my own artwork and generate new pieces in my style

You are not immune to propaganda

I'm not discounting it. I just care more about point-source pollution than carbon footprints abstracted across a global supply chain.

The footprints can be reduced at scale over time, whereas these cars will be continuously producing more pollution as they age.

On the downside, the engines in cars will pollute more as they age. This will make reducing pollution more difficult as people stick to their older and less efficient cars.

> Power corrupts, and absolute power corrupts absolutely

You could definitely say this about their self contained power grid.

If Texas was so interested in spreading out power, then surely they would be more willing to trust their elected, representational Federal government instead of having a Texas specific version of everything?

> Texas generally fears a tyrannical government more than they fear a random shooting

Why do you think this is the case?

As someone whose been in an awfully designed office at a "thrifty" company, I can't imagine how returning to that office would be a gain for me.

I was put in a row of desks literally next to the open kitchen/conversation area. My day had constant distractions and I had no control over how I wanted my work environment to be.

Working from home, I'm incredibly comfortable AND productive. The noise level, lighting, etc are all how I choose them. I dont have this tension around other people getting to decide for me how I want to work.

If you work in a FAANG type of office environment, the above probably doesn't apply to you. Your office is professionally designed and cost many millions of dollars. You are given high quality equipment to make you comfortable. Your office has amenities that make being there easier.

SQL proficiency is important but I wouldn't say it supersedes programming experience. To me, Data Engineering is a specialization of software engineering, and not something like an analyst who writes SQL all day.

As DE has evolved, the role has transitioned away from traditional low code ETL tools towards code heavy tools. Airflow, Dagster, DBT, to name a few.

I work on a small DE team. We don't have the human power to grind out SQL queries for analysts and other teams. Our solutions are platforms and tools we build on top of more fundamental tools that allows other people to get the data themselves. Think tables-as-a-service.

Would be a shame if the Parler CEO literally refused to moderate his platform even when warned of the consequences multiple times.

Nooo! Amazon HAS to let me use their business! The first amendment says so!!!!!

What rights as a US citizen has Trump lost here? The right to shitpost on twitter?

Free speech extremists seriously think a private company should be forced to keep someone who uses their platform to incite deadly riots.

Trump's free speech here has consequences. 5 people have died.

> That depends on what Republican wins. Not all Republicans are Donald Trump-style authoritarians.

The last week of falling behind Trump in that massive election fraud occurred (without any evidence) shows that the GOP is the party of Trump whether they like it or not.

Thinking anything else is delusional.

When people talk about raising kids right, it always pans back to growing up in a rural area away from most people. Besides the dark skies/wilderness why is growing up in a rural area better than in a city for children?

Given that the majority of people live in a city/metropolitan area, I'll assert that my subjective opinion that growing up in a city would be better for a child than your subjective opinion that rural areas are better.

> This is why I'm frustrated with the idea that we need to enact change and policies at the federal level. The federal government should just exist to protect our liberties and organize national defense. Everything else should be organized more locally at the community level where voters actually have some skin in the game.

You'd think that after months of "locally led" covid responses in the US that this naive take would somehow become less popular.

Whether small-government enthusiasts like it or not, the US is highly interconnected on almost every level. There are certain things that absolutely require federal responses. Things like pollution and viruses absolutely need to be handled from a nation wide perspective. How would healthcare work as a piecemeal implementation across local lines? Its almost as if "letting localities decide" is a nice way of ensuring something will fail without forcing it outright.

Your argument is implying that the federal government is somehow ineffective, but given the federally led improvements across the last century - from roads to environmental regulations, public health and even the internet just shows that you're either ignorant of intentionally dishonest.

How is anything about his hamfisted (publicly admitted) attempt to disenfranchise voters even close to cunning?

Maybe it’s because his party and half of America is also on board with it?

I really disagree here. Monolithic dags are a bigger pain to manage.

Breaking them out into smaller dags makes retrying/backfilling/etc a lot more straightforward. It also lets you reuse those pieces more easily.

We have compute DAGs that are the upstream dependency for many other dags. Originally, this dag was monolithic and loaded data into one table. But because the dag is split into computation and loading we can easily add more downstream dags without changing how the first one operated.

> - Don't use it for tasks that don't require idempotency (eg. a job that uses a bookmark).

You can totally design your tasks to be idempotent - but its up to you to make them that way. The scheduler or executor doesn't have any context into your job.

This is why I encourage people to use a unified base operator and then pass their own docker containers to it. Aka like how https://medium.com/bluecore-engineering/were-all-using-airfl... outlines it.

> - Don't use it for latency-sensitive jobs (this one should be obvious).

IIRC this is being addressed in Airflow 2.0

> - Don't use sensors or cross-DAG dependencies.

This is a little extreme. I've never ran into issues with cross dag dependencies or sensors. They make managing my DAGs way easier because wee can separate computation dags from loading dags.

context: I built/manage my company's Airflow platform. Everything is managed on k8s.

Not everyone has:

- long commutes

- crappy coworkers or offices

- a space large enough to comfortably work remotely (or the money to afford a bigger space)

- a partner, kids, or both, which makes extended social isolation more livable

- a rich network for career growth and opportunities

God forbid some people want to live in big cities and don't make their choices solely based on reducing costs and bottom line expenses (ironic since every other day people here rail against big corporation bean counters). Example: Facebook's latest internal polling - the majority of people want to be in the office sometimes.

People suddenly waking up and realizing the office is a huge scam is the current du jour opinion here. But time and time again the HN demographic only speaks to itself.

And of course people will reply with that it expands choice, but that doesn't stop those from cheering that companies going full remote like its a universal good thing for everyone.

And on top of that, my observations are anecdotal. No need to point that out.

edit: Going full remote is a huge cost savings to companies. A cost that is now hoisted onto employees. So unless employees are receiving some equivalent compensation for blowing out my utility bills and refitting my office, be careful who you're cheering with.

Its not trivial to create and manage Data Pipelines if you care about scale, serving a wide range of inputs and outputs, or making this data easy to surface and spread throughout your org (i.e. making it actually useful to regular people).

"Static ETL" like running the same database load every day at 1:00am isn't a super challenging problem. Doing it across many tables with complex transformations and multiple steps easily can be. You really have to consider reliability, processing speed, failure methods and other problems that dont really arise until you hit a certain scale.

There's also the issue of what people want out of a Pipeline that's changing. If you want people to be ""data driven"", then that means they need easy access to potentially all of your company's data on an ad hoc basis. So now your boring ETL 1 am pipeline isnt really serving any of these new usecases.

How do you create flexible pipelines that can be created from any dataset on an ad hoc basis? This is where tools like Airflow or Prefect come in. Creating a platform that can create these types of Pipelines is a real problem.

And before you even ask yourself _how_ to process this data, you need to also ask _where_? If you want to do what I outlined above - making your data more accessible and easy to use - then you probably need to rework how you're storing your data. But Data Lakes (and others) are a whole topic in and of itself.

Really similar to the pipelines that I engineer/manage at my current company. Although we have our Airflow on kubernetes.

One optimization though is separating your loading tasks from compute tasks. This makes the pipeline more resilient and makes backfilling/reprocessing less of a headache.