baron_harkonnen

Where are you getting that I'm assuming that "personal working style is the best and only one for every person"? I am recommending what worked for me. It's a bit different than the question the OP ask's but, given a similar situation, it's what has worked.

I suspected you my be projecting a bit of your own insecurities here.

Unfortunately HN has become an increasing hostile place, and I think rather extreme reactionary response to a simple recommendation is becoming the norm.

In a weird way, thanks for finally waking me up to the realization that is community is not what it was and not one where I should be spending time anymore.

Call it what you want but that advice comes from experience. I worked remote for many years before pandemic and had plenty of issues with isolation.

Sometimes in life socialization comes easy and requires no work, but that's not always the case. This is a common problem people face once they leave high school/college. The truth is that real social relationships, the ones that keeps us more fulfilled take work.

I realized that to not feel lonely all the time when working at home I had to put some effort into making sure I reached out to friends, scheduled lunches and calls (which can happen during that 6 hour period, that's one of the benefits of WFH), make sure I went to local meetups etc. The end result was that from that effort I made some fantastic friendships that were much more fulfilling.

Even in the OP's current work environment a little more deliberate effort can go a long a way. I'm sure other coworkers feel the same way, if they pair programmed in the office why not reach out and schedule some pairing sessions? I've know many remote places that do this. Schedule virtual coffees during the day where two people can just chat, again something I've seen at many remote places.

Yes, for decades people could get a basic level of socialization for no effort, but like most no effort things that socialization goes as easy as it comes.

Having been in a similar situation to OP, that was the lesson I learned so that's the advice I'm giving, which I hardly think is tone deaf.

I would recommend looking into decoupling your social needs from your job.

Even in the time before wide spread WFH this was an issue, during good times it feels great to go into work and get all your social needs met but when layoffs come or companies collapse suddenly that great work friend everyone loves gets let go and in a few more weeks they effectively don't exist anymore.

It's great if you can meet people at work and create a real friendship (I've certainly done that). Now that you're remote you can put more time into keeping up with those people. Schedule lunches, video calls etc.

Get to know your neighbors better, join local interest groups, schedule video chats with friends you haven't chatted with in a while that live far away, and make sure to get lunch with local friends whenever you can.

It will be a bit of transition but ultimately you'll have a much richer social life and honestly enjoy work more as well since you have more outlets in the day that have nothing to do with your 9-5.

A variation of this comment shows up in every FB related post, and it seems completely absurd to me that anyone holds this view seriously.

The idea that interacting on Facebook is somehow "keeping up with friends" is itself perhaps the greatest victory of Facebook's own marketing. The idea that 'liking' a picture and leaving a public comment is a sincere human interaction is ridiculous. It's social noise used to replace interpersonal connections and is being pitched that this is somehow better.

If FB suddenly disappeared people would likely go back to texting each other images and calling each other more often, in private where they could more openly admit their struggles and non-public views. People would stop pretending that they had more than 30 "friends". Daily likes and emojis would be replaced with actual video calls, less frequent but longer, more intimate conversations.

This isn't speculation as it's how everyone I know who isn't active on facebook communicates. When I call my distant friends and family the interactions are entirely different than the public facing, image maintaining, completely non-intimate communication that happens on any "social" media.

You're not keeping up with what's happening when you interaction on facebook, because what's really happening is struggles and concerns that you don't necessarily want to share in public with everyone you know. Real human connections involve being vulnerable around someone you trust, which is fundamentally in opposition to the foundations of how something like facebook works.

> chain rule is basic calculus that precedes understanding hessians.

It doesn't have to be that way. The hessian is an abstract idea and the chain rule and more specifically backpropagation are methods of computing the results for an abstract idea. When I want the hessian I want a matrix of second order partial derivatives, I'm not interested in how those are computed.

For a more concrete example, would you say that using the quantile function for the normal distribution requires you to be able to implement it from scratch?

There are many, very smart, very knowledgeable people that correctly use the normal quantile function (inverse CDF) every day for essential quantitative computation that have absolutely no idea how to implement the inverse error function (an essential part of the normal quantile). Would you say that you don't really know statistics if you can't do this? That a beginner must understand the implementation details of the inverse error function before making any claims about normal quantiles? I myself would absolutely need to pull up a copy of Numerical Recipes to do this. It would be, in my opinion, ludicrous to say that anyone wanting to write statistical code should understand and be able to implement the normal quantile function. Maybe in 1970 that was true, but we have software to abstract that out for us.

The same is becoming true of backprop. I can simply call jax.grad on my implementation of loss of the forward pass of the NN I'm interested in and get the gradient of that function, the same way I can call scipy.stats.norm.ppf to get that quantile for a normal. All that is important is that you understand what the quantile function of the normal distribution means for you to use it correctly, and again I suspect there are many practicing statisticians that don't know how to implement this.

And to give you a bit of context, my view on this has developed from working with many people who can pass a calculus exam and perform the necessarily steps to compute a derivative, but yet have almost no intuition about what a derivative means and how to use it and reason about it. Calculus historically focused on computation over intuition because that was what was needed to do practical work with calculus. Today the computation can take second place to the intuition because we have powerful tools that can take care of all the computation for you.

> at most you can follow recipes cook book style.

Here I disagree with you pretty strongly. Once someone is comfortable with differentiable programming it's much more obvious how to build and optimize any type of model.

People should be more concerned about when to use derivatives, gradients, hessians, Laplace approximation etc rather than worry about the implementation details of these tools.

Abstraction can also aid depth of understanding. I know plenty of people who can implement backprop, but then don't understand how to estimate parameter uncertainty from the Hessian. The latter is much more important for general model building.

> to read this as though it is very obvious and commonly accepted fact

I'm not entirely sure what you're referring to by "this" but assuming you mean my comment, I think what I'm saying is very much up for debate and not an "obvious and commonly accepted fact". Karpathy has a very reasonably argument that directly disagrees with what I'm suggesting [0]. Of course he also agrees that in practice nobody will every use backprop directly.

Whether it's JAX, TF, PyTorch, etc the chain rule will be applied for you. I'm arguing that I think it's helpful to not have to worry about the details of how your derivative is being computed, and rather build an intuition about using derivatives as an abstraction. To be fair I think Karpathy is correct for people who are going to be learning to explicitly be experts in Neural Networks.

My point is more that given how powerful our tools today are for computing derivatives (I think JAX/Autograd have improved since Karpathy wrote that article), it's better to teach programmers to learn think of derivatives, gradients, hessians etc as high level abstractions. Worrying less about how to compute them and more about how to use them. In this way thinking about modeling doesn't need to be restricted to strictly NNs, but rather use NNs and example and then demonstrate to the student that they are free to build any model by defining how the model predicts, scoring the prediction and using the tools of calculus to answer other common questions you might have.

edit: a good analogy is logic programming and backtracking/unification. The entire point of logic programming is to abstract away backtracking. Sure experts in Prolog do need to understand backtracking, but it's more helpful to get beginners understanding how Prolog behaves than understand the details of backtracking.

[0] https://karpathy.medium.com/yes-you-should-understand-backpr...

Given the current state of automatic differentiation I'm not so sure it's even necessary or particularly useful to focus on backpropagation any more.

While backprop has major historic significance, in the end it's essentially just a pure calculation which no longer needs to be done by hand.

Don't get me wrong, I still believe that understanding the gradient is hugely important, and conceptually it will always be essential to understand that one is optimizing a neural network by taking the derivative of the loss function, but backprop is not necessary nor is it particularly useful for modern neural networks (nobody is computing gradients by hand for transformers).

IMHO a better approach is to focus on a tool like JAX where taking a derivative is abstracted away cleanly enough, but at the same time you remain fully aware of all the calculus that is being done.

Especially for programmers, it's better to look at Neural Networks as just a specific application of Differentiable Programing. This makes them both easier to understand and also enables the learner to open a much broader class of problems they can solve with the same tools.

I'm a bit surprised that nobody has mentioned how much we have failed as technologists that a story is being shared by png images of text... brave new world indeed.

Just because this is an older problem doesn't mean it's not a problem.

Yes, people that loved fixing up cars would have complained about "kids these days" not knowing how to fix a car. I was certainly one of those kids.

However seeing this happen with a field I know makes me realize those people complaining about people not knowing how to take care of their cars were right.

And while this is true of "any new tech", it's a relatively recent phenomenon as tech has only been advancing this rapidly in last 100 years or so.

It is a bad thing to be increasingly alienated from the tools you rely on for your day-to-day existence. Of course it's in part because cars and computers have both been increasingly designed by their manufactures to be difficult to take apart and understand. Tools that you don't understand increasingly control you rather than the other way around and I personally think this is a trend worth resisting.

It's good to point out that this is an issue and even better to encourage more people to be curious, and learn how things work. Another example that I'm surprised of is people's homes. I know a shocking number of home owners that cannot fix a single thing in their own home without calling a "professional" (and it's increasingly surprising how many professionals also don't understand what they're doing!) This was extremely evident during the freezing temperatures in Texas where many people didn't know how to shut off the water to their homes.

One of the best parts of the "hacker" mentality is to encourage people to not be scared of their tools and the things they own. While things have gotten more complicated, you can do a large amount of repair and modification on your own for almost everything. It honestly feels very liberating (not to mention saving you a lot of money) to snake a clog 12 ft deep in your drain, replace your car's serpentine belt, restore old hardwood floors, repair broken refrigerators from parts, etc.

I don't think that's as relevant as the current demand for VCs to unload their investments as fast as possible before the next major economic catastrophe.

It boggles my mind that we're set to nearly double the record number of IPOs this year, and last year was the highest number since 1999, yet in general no one finds this at all interesting or alarming.

> It's not the wackos we should be worrying about.

It's really amazing to me how easily the "left" was able to be tricked in the same death of critical thinking as the "right".

The "stick to the libs!" angle is far more responsible for the rise in support of Trump leading up to the election. People on the "right" were manipulated for decades into reducing their political beliefs to defending themselves from a fictitious adversary (this is why I put quotes around these terms). If you listen to any radical Trump supporter you'll quickly see that a large part of their logic is based on a deeply held belief that roughly half the country is mind washed, irrational liberals that seek to destroy their way of life.

This rewriting of people skeptical of the vaccines as "wackos" serves the same purpose for the "left" and mainstream progressives have gobbled it up without hesitation. They now see roughly half the country as a bunch mind washed, irrational "wackos" that are a threat to the foundations of our society.

Both the "left" and "right" (terms which honestly don't make any political sense any more, evidence by exactly this irrational support for corporate suppression of voices on the "left" and it's dissent on the "right") are currently structured so that any real, meaningful political discourse about the future of the country is dissolved into two insane groups of people throwing rocks at each other.

If you find yourself defined by either of these major narratives, then you are being played.

Not to mention that industry itself only accounts for 6% of Arizona's water usage [0] so we're talking about one single structure increasing the industrial usage by ~10%.

[0]

As resources start collapsing, more and more major world powers will begin to keep their own resources for themselves.

China didn't say they were targeting a net reduction in building coal, only that they weren't going to be building coal for anyone else.

This is likely due to the projected increase in demand for coal in China and a realistic assessment about how resource constrained they are.

And to be clear, while collapse of fossil fuels, something people have been concerned about for decades, will be better for the planet, it will still mean a rapid decline in our standard of living and essentially the collapse of modern industrial society.

> We just got through our wood and lumber shortage

Prices have started to drop but I would hardly say we "got through" the shortage. Just last week I was talking to the owner of a lumber yard asking when cedar would be readily available again, and he was skeptical that it ever would be in steady supply again. He mentioned looking to get out of the industry since he wasn't too sure it had much of a future left for him.

Hopefully he's wrong, but I'll remain skeptical of a return of the lumber industry to what it was.

> you replace trusting your ISP with trusting a different group of unknown people with similar motivations

I've always seen this argument but it's never made sense to me.

For starters I absolutely don't trust my ISP. I know they are collecting, storing, likely selling my data and that they are 100% going to comply with any government requests from my government (I don't even trust that they would only respond to legal requests).

Years ago I used to use AirVPN. They claimed:

> AirVPN started as a project of a very small group of activists, hacktivists, hackers in 2010, with the invaluable (and totally free) help of two fantastic lawyers and a financing from a company interested in the project and operated by the very same people.

Maybe they're lying but at least there's some chance they actually care about privacy.

But even if they don't care about privacy at all and are lying, at the very least they are based in Italy and have their servers spread throughout Europe. Additionally you can pay via crypto (which gives you more anonymous payment options than your ISP). Simply being in another country then the one I live in makes it much harder for my government to arbitrarily request my data.

Yes if I want to do highly illegal activity that is going to get my government interested in me I absolutely don't think that would be enough. But if I want privacy from routine surveillance this seems like a fantastically better option that 100% giving up.

You're just reiterating my point. Even if this where true, it doesn't add any information. For a Catholic it simply affirms their Catholic worldview, however for someone with a secular worldview it is also perfectly inline with their existing views.

As such the statement "Surprisingly" make no sense, since this information is not "surprising" to anyone, even if it were true (which is it very likely not).

Let's suppose this was even true, what would it matter?

There's this weird assumption in popular culture that somehow the moment near death contains more truth than other moments, as if right before passing we have some "ah!" moment. But the opposite seems far more reasonable: as we get closer to death our reasonings will become increasingly distorted.

For example, a common narrative is "on their death bed they had wished they had focused less on career and more on family." Of course that's what you'd think on your death bed! You're likely scared, feeling alone and wish there was someone with you. You also don't have to worry about rent, gaining the respect of your peers, what you're going to do with your free time, retirement etc.

The idea that the moment of death brings some sort of grand understanding only makes sense if you presuppose there is some grand sense behind it all. Otherwise it's just the final moment of a long process of physical, emotional and mental decline.

> In practice, these huge models are, in laymans terms, fucking awesome and work really well

A similarly surprising result from an adjacent community, Bayesian Statistics, is that in the case of hierarchical models, increasing your number of parameters can paradoxically reduce overfitting.

The scale of parameters in Bayesian model's is no where near that of these deep neural nets, but nonetheless this is a similarly shocking result since typically adding parameters is penalized when model building.

It's a bit more explainable in Bayesian stats since what you're using some parameters for is limiting the impact of more granular parameters (i.e. you're learning a prior probability distribution for the other parameters, which prevents extreme overfitting in cases with less information).

I wouldn't be too surprised if eventually we realized there was a similar cause preventing overfitting is overparameterized ml models.

Rather than implying a limit to the usefulness of data, I find it speaks more to the folly of substituting data for critical thinking.

You can reduce a fever by treating the underlying infection, soaking in ice water, or taking acetaminophen. No good doctor would judge a patient's health solely because a single metric, namely body temperature, was able to be moved into acceptable ranges. That doesn't mean temperature isn't extremely valuable data, and essential to decision making, but that it cannot be a substitute for understanding and solving the real problem.

I once knew of a SaaS company that had perpetually growing MRR (Monthly Recurring Revenue), great right? Except, churn was also growing. An increase in MRR was achieved by upselling a perpetually shrinking group of core customers. The core KPI of this company was MRR, and, unsurprisingly, this company does not exist anymore. Again, here is a case where we can see all that other data (churn, upselling) is very useful, as is the KPI. But the key to success or failure here is whether or not you want to really expend the effort to understand the problem or just chase a KPI.

KPIs are seductive because they make managing team's performance seem much easier: just get this number higher and you're doing good, get it lower and you're doing bad. But that's like playing a game of chess where each piece is controlled by a different person, and that person is judged solely on how many times they can get the king in check.

> most people on hn will not experience much issues

If the AMOC collapses as we are already starting to see [0] then at least the European portion of the HN community will be deeply impacted.

If the American South West continues it's path of aridification [1] then plenty of HN member will be impacted as well.

[0]. https://science.howstuffworks.com/environmental/earth/oceano...

[1]. https://www.pnas.org/content/117/22/11856

> much more than that at times in the last few thousand years.

When was that?

If you look at this estimate of temperature on Planet Earth[0][1] you can clearly see the last time the planet saw > +2C was the Pliocene.

Even if you look more closely at just the Holocene you don't see > +2C anywhere[2]

[0] https://upload.wikimedia.org/wikipedia/commons/5/5f/All_pala...

[1] https://en.wikipedia.org/wiki/Geologic_temperature_record

[2] https://en.wikipedia.org/wiki/Global_temperature_record#/med...

I once mentioned Goodhart's Law to a data scientist at a company and they immediately rejected it based on the unironic assertion that

"that would mean that KPIs shouldn't be the sole measure of our performance that that doesn't make sense!"

My experience in the field has been that an astounding number of products have been destroyed and users harmed by failing to heed Goodhart's Law.

> China would never get rid of such a powerful way to annoy everyone else.

I've mentioned this in another comment, but you're severely underestimating the real relationship between North Korea and China. For the government, maybe it is only a strategic relationship, but if you talk to anyone who grew up in China there is a real feeling of kinship with the people that live in NK.

For a long time there has been a lot of cultural exchange between the nations and many Chinese near the border are indistinguishable culturally from North Koreans.

An attack on North Korea, even if a justifiable retaliation, would feel like an attack on the Chinese people for a great many everyday people in China.

The closest analog to the US would be Canada. Most American's, especially in the North, feel that Canadians are very culturally similar. Many people in far Northern states have friends and family in Canada. Overall the bond is much closer than with Mexico, which has more culture, ethnic and linguistic differences.

Imagine if Canada were ruled by a despot, who then irrationally launched a nuclear strike on another country. Even if American's were to feel that that was wrong, we would also still likely strongly defend Canadian people from aggressive retaliation. And this analogy is far from perfect, the sense I've gotten in conversations with Chinese about North Korea is that they're much closer.

NK has a very complex relationship with China and I'm fairly sure they would not be "wiped out" immediately.

Even in the extreme case of North Korea nuking Japan, it would still lead to an extremely complex international situation. The US retaliating for Japan with a nuclear strike or any similar situation of extreme force would be a far too risky of a move.

Not only is NK important to China strategically as a buffer between themselves and the US friendly South Korea, but the Chinese people feel a kinship with the people of NK, not entirely dissimilar to the way the US views Canada culturally. Aggressive retaliation would unquestionably make China an aggressive and active enemy of the US (or whoever decided it was their job to retaliate).

North Korea's proximity to China, not just physically but culturally as well, makes the issue far more complex than if a rogue nation where make similar threats.

> i just got used to working in vectors and arrays for computational tasks.

Have you worked with R? R, like matlab, natively supports vector based operations. In fact, all values in R are vectors. Many of the problems with Pandas ultimately boil down to the fact that you have to replicate this experience without truly being in a vector based language.

Pandas indexing system is overly complex and I've never personally benefited from that. To start with there are __getitem__, loc and iloc approaches to accessing values. If your library constantly has to warn users that "you might being something wrong, read the docs!" that should be a warning sign that you don't have the correct level of abstraction. R has a much more sane api and assumptions about when you want to access a value by reference (which is almost always) and by value.

Then when doing basic operations like "group by" you end up excessively elaborate indexes that are in my experience useless and always need to be manually squashed to something coherent.

It's a common joke for me that whenever even a seasoned Pandas user cries out "gaarrr! why isn't this working!?" I just reply "have you tried reset_index?"... this works in a frighteningly large number of cases.

Of the many companies I've done data science with I can only think of a few, rare exceptions where R wasn't used as much as if not more than Python.

If you're mostly dealing with Neural Nets you won't see much R, but for anything really statistical in nature R is a much better tool than Python. For anything that ends up in a report R is much better than Python (a lot of very valuable data science work ends up being a report to non-technical people).

> breaks down on data manipulation

This is very outdated. The tidyverse eco-system has bumped R back into being first in class for data manipulation now. This becomes less true as you get further and further from having your data in a matrix/df (I can't imagine doing Spark queries in R), but if you already have a basic data frame, manipulation from there is very easy.

Even for things that end up in production, whether you're in R or Python, whatever your first pass is should always be a prototype and will have to be reworked before you get close to moving it to production.

I used to very strongly agree with you re: matplotlib, but I've recently switched from using almost exclusively ggplot2 to almost exlusively Matplotlib and my realization is that they are very different tools serving very different purposes.

ggplot2 is obviously fantastic and makes beautiful plots, and very easily at that. However it is definitely a "convention over configuration" tool. For 99% of the typical plot you might want to create, ggplot is going to be easier and look nicer.

However matplotlib lib really shines when you want to make very custom plots. If you have a plot in your mind that you want to see on paper, matplotlib will be the better tool for helping you create exactly what you are looking for.

For certain projects I've done, where I want to do a bunch of non-standard visualizations, especially ones that tend to be fairly dense, I prefer matplotlib. For day to day analytics ggplot2 is so much better it's ridiculous. The real issue is that Python doesn't really offer anything in the same league as ggplot2 for "convention over configuration" type plotting.

Fully agree on Pandas. R's native data frame + tidyverse is world's easier. Pandas' overly complex indexing system is a persistent source of annoyance no matter how much I use that library.

> Maybe I'm the outlier

A good sanity check of this is to use Dunbar's number (usually considered at ~150)[0] and see how many of your friends and friends of friends have a number of social media "friends" that reaches or exceeds that number.

In case you are unfamiliar: Dunbar's number is a proposed cognitive limit to the number of real social relationships we can have based on brain size.

The total number of real, stable social relationships you can have is physiologically limited, so having 200 facebook "friends" that truly are your friends is impossible.

Because we do have connections with people outside of social media (hopefully!) even 150 friends is nearly impossible unless you are friends with literally every social relationship you have.

So if you want to see, I would first establish that you are really in the group you think you are. If you are only connected with true, irl friends you likely don't have much more than 30 "friends". Then sample the graph of that network a bit and see where you are in the distribution of "friends".

0.https://en.wikipedia.org/wiki/Dunbar%27s_number