opportune

Something has to give in your situation because simultaneously pursing both a PhD and full time job really limits your free time out of the gate.

I don’t think it’s reasonable to even try to pursue side hustles in your situation, it makes no sense and I’d be surprised if you’ve even been able to give any attention to this at all - just stop telling yourself you need to do it/forgive yourself for not doing this even if you think you should.

One thing that helps me squeeze in exercise is working out by 1. commuting via bike/walk to work 2. going on a run at lunch time when I wfh 3. lifting weights during regular work hours around 2-3pm (easier than you think in most office jobs. not always possible though)

Also the second generation of Flume/Spark different vs MapReduce/Hadoop has to be understood in the context of what other assumptions changed at the same time. At Google, GFS was replaced with Colossus (can’t share specifics but this was also accompanied by a change in “data/machine topology” and associated networking changes away from uniform less specialized servers) which made it so “move code to data” became less important. Similarly Spark was originally meant to run on HDFS but became a lot more popular once it started being able to use things like s3 as its storage layer and public cloud VMs for compute (which was a similar transition to GFS->Colossus).

In terms of usability the other two main innovations were to make it easier to program a workflow that chained MapReduce operations (without an intermediate, expensive, blocks-until-all-nodes-done disk write step, nor a jankass orchestration engine) and subsequently to declaratively specify the desired output (eg SQL) without requiring the user to specify the implementation.

They’ve since added more stuff like streaming, ML, whatever, but the biggest change from 1st to 2nd gen is really in the data topology.

The problem with using technology X “because Google does it” or “this is the best open source version of what Google uses, so let’s use it because Google does it” is that companies neglect that Google does not just use the technology out of the box they

1. created the software for their own needs 2. maintain a developer team to improve it and address requirements/pain points/integration 3. have an internal pool of experts in the form of the developer team and “customers”/early adopters 4. most likely have other proprietary systems like Borg or Colossus which integrate with the software very well, which OSS like Hadoop may not (another example: OSS Bazel vs Blaze+Forge+Piper+Monorepo structure).

Something like HDFS was hugely painful for many teams because they had no idea how it worked or how to debug it, had no idea how to fix it or extend it, and didn’t have any good tooling to understand why something was slow. All they could do was try to configure it, integrate with it, and find answers for their problems online. That’s because HDFS was “free” but a team capable of properly maintaining, supporting/operations, and developing HDFS was extremely expensive.

When you work in a very large and complex codebase you encounter a few things that this author doesn’t seem to consider or thinks are very minor:

1. Refactoring something introduces non-negligible risk. Consider a class with many fields and multiple mutexes it uses to control concurrent access to those fields. Even just consolidating those mutexes introduces the hard-to-conclusively-find-in-testing risks of introducing a deadlocks and livelocks. And that’s like the base case of refactoring the class: anything involving splitting the class up, moving data fields up or down the stack, changing the way member functions (which acquire locks) call each together is even more complicated and risky. It is just not worth refactoring this thing unless you have a very very good reason.

2. A function or object often has a many-to-many relationship in what it touches: it is called or accessed from multiple places and it calls and accesses many things. Non-trivial improvements to abstractions typically involve changes at both ends: which may be “as simple” as updating all the call sites to take a new argument or handle a different kind of error (hopefully all your call sites are structured so error handling is compatible with their abstraction!) or as complex as completely refactoring multiple levels up and down the stack to reflect better-abstracted semantics.

No you shouldn’t lazily copy-paste around such problems when they are straightforward enough. But it can so so much less work (and again, less risk of breaking things) to use composition + wrappers, or inheritance, or to copy some little chunk to code than to do things the “right” way.

3. Let’s face it, your cool new abstraction sounds right in your head, but in a complex system it may just be playing abstraction whackamole once all the bugs and edge cases you’re not initially considering get addressed. It may be impossible to fully understand the entire system from beginning to end, without which it’s hard to be confident you’re actually improving things before embarking on your epic partial rewrite, or at the very least know you’re not changing semantics around some arbitrarily-drawn box. But if you’re not even changing the semantics, see point 1.

You’re right and wrong. MapReduce is two things: a pattern for massively parallel computation, and the name of the initial implementation of the pattern at Google.

While the initial implementation at Google quickly got replaced with better things, the MapReduce pattern is everywhere in the data space, and almost taken for granted now. Hadoop is basically the same: a shitty (I think HDFS is still pretty good, just not the compute part) initial implementation of the pattern that was quickly iterated and improved upon.

Also, a big reason people stopped having to think about eg rack-local operations is that most people operating on huge amounts of data now aren’t doing it on traditional generic servers, they’re using something like s3 on VMs in Public Cloud datacenters if they’re doing something relatively “low level” or more likely just using something like Snowflake, Spark/Databricks (pretty close to OG mapreduce…), etc.

They don’t get all the stress of owning a business per-se because they most likely vote on a CEO/a council to handle that. Just like the Walmart heirs aren’t stressed about owning a business because they vote on a CEO to do that part, or how you as a voting citizen don’t get all the stress of running a country.

It is true though that an employee will be much more personally invested in that kind of voting than a typical investor given that’s they essentially can’t diversify like an investor can, without being so rich that they make more from investments than working anyway. And that they may choose to give themselves the option of direct democracy to varying degrees instead of representative democracy. In a standard white collar corporate environment that could kick up workplace politics to insane levels.

This seems reasonable. From what I am able to pick up, they thought they had found some clever loophole to ICO by selling NFTs. Despite the C&D some of their founders key benefits are still up: https://founderskey.io/images/fk/legendary-key.png

Basically 1. until the platform is implemented the key is basically an IOU. However, it can be sold and transferred for a profit AIUI so it’s not like a crowdfunded game where you get some similar benefits as those can’t be sold for a profit. 2. all your NFT corresponds to is a literal key representing the benefits Impact Theory will make available to you in the future so per the Howey Test you are purchasing a contract (not a specific piece of art or cosmetic) yielding profits from the sole efforts of a third party (Impact Theory).

Honestly, they probably could have gotten away with it if they had been a little less lazy! I bet if they tried to raise money by selling some kind of character art (that currently exists like a normal NFT, not a custom avatar they’ll give you one day) that they’d eventually also display in their final product they’d have been fine.

The interesting takeaway though is that it seems like you can’t do kickstarter-style crowdfunding through transferable NFT ICOs, at least in this form in which you got nothing but a key.

No mention of ESGs? Want to know why corporations participate in this idiotic charade, as well as other weirdly ubiquitous theatrics? It’s because paying lip service to this kind of stuff is a requirement for certain funds to be able to hold their stock: https://en.m.wikipedia.org/wiki/Environmental,_social,_and_c...

So corporations have a big financial incentive to do the bare minimum to satisfy ESG requirements, and carbon offsets are an unregulated sham. No surprise, the market finds a way and delivers impossibly cheap “offsets” so that such environmental stewards as Chevron, and soon Phillips Morris, can be considered to be environmentally friendly and socially benevolent enough to be included in ESG funds.

I imagine it would take me that long the first time having not set it up before, but that even if you started over from scratch it’d take you much less time.

I made another comment to this effect, but I think testing and CI (not necessarily CD though) have such quick payoff times in terms of productivity that you almost always want to set them up ASAP.

I’m not a huge fan of this because it’s kind of painting all “original (technical) problems” with a broad brush, while also being a rewording of the typical YAGNI/establish PMF advice lines. I suspect the author is coming from the perspective of primarily working on “applications software” like consumer sites or solving some novel/poorly addressed business need with software.

A lot of these things aren’t binary yes/no decisions. They range from “don’t even do the thing at all” to “develop a custom solution completely in house”, sure, but with a huge spectrum of choices in between: buy off the shelf, get something basic but inflexible working, get something more advanced and flexible with more setup working, establish a process-to-be-automated that engineers perform manually.

Especially on the <easy setup, kinda shitty> to <hard setup, powerful/flexible> gradient you really need to consider your needs and options’ risk/reward. Some things like tests and CI have such quick payoff times you almost always want something that does a decent job of things. Moreover, when selling technical products half the time your competitive advantage is actually being able to efficiently iterate and improve on the product faster than everybody else because competitors are bogged down in tech debt and manual operations.

IME it’s not just the quiet and calmness. There’s few children and families (mostly elderly folks), what few people there are mostly keep to themselves and won’t initiate conversation with you, and abandoned places are left to degrade without being vandalized or littered with trash. There are roads and shrines that are still maintained but most likely empty when you visit. There’s nothing, and very little even “regular abandoned stuff” like broken glass or boarded up windows, to make you feel on-edge or unsafe.

I think the word “ennui” was almost invented to describe rural Japan. There is a mesmerizing, eerily graceful sense of decline.

In a lot of other rural places on Earth, even if they are in poverty, there is usually a decent amount of activity even if some buildings are abandoned. There are families (often very large ones with many kids), drunk hooligans, outgoing salt of the earth types.

Rural Japan in contrast is quiet, mostly elderly, and more culturally “refined” at least on its face. And if you go wandering around, even as a very conspicuous foreigner, because of the language barrier and general cultural introversion, the few people you do see will mostly politely ignore you or not initiate conversation. Similarly while buildings may be abandoned and overgrown with plants there is usually very little trash (not counting what was simply left there, think more like plastic bags and empty beer bottles from teens hanging out) and no graffiti.

Other places that may be obviously in decline have evidence of a future and continued habitation, even if bleak, from young people vandalizing abandoned places, families going about their day, and confused middle aged folks trying to ask why you are even visiting their place and if you’re lost. In rural Japan, especially if you don’t speak Japanese, it’s more like you are a ghost wandering around a place resigned to evaporate, whose remaining inhabitants stoically continue on their peaceful day to day.

Yes but not only does it cost much less (financially, debatable with time) but it’s individualized. Not many private schools offer truly individualized education

Historically schools were usually a way to do one of two things: provide a strong education (mostly in the humanities) to the upper class, and teach basic literary skills so that kids could read religious scriptures and then later, post Enlightenment, adequately participate in government and society (including basic arithmetic and non-religious literary skills). The first public schools in New England were motivated by the pre-Enlightenment attitude and public schools became more widespread with the post-Enlightenment attitude.

The idea of public schools as a way to enforce national unity (~WW1 when it became a priority in the US to de-italicize ethnic identities and loyalties) and teach kids skills that would set them up for nationally important jobs like those in science and technology (Cold War) was a much later phenomenon.

Except they have much more flexibility, up to individualization, in terms of curriculum content and pacing, teachers from potentially more varied backgrounds/main jobs besides teaching, and the ability to eject unruly and problematic kids that impact other kids’ abilities to focus and learn?

I went to regular school from K-12 and I also felt like my social skills were significantly stunted for a long time. Which is not to say that homeschooling can’t negatively impact your social skills, but that plenty of kids experience stunted social skills, social anxiety, awkwardness, introversion even with regular schooling.

Also as an adult I gotta say, in most ways it’s traditionally schooled kids who are “stunted” (immature, have mannerisms/beliefs/habits incompatible with the adult world) until they hit about 20-26 more so than homeschool kids. I’ve met homeschooled kids who probably considered themselves stunted, and in some ways (being sheltered) they were, but they were also significantly more “adult-like” and mature in other ways earlier than their peers. This is also true for a lot of traditionally schooled kids who are “late bloomers” IME.

Can’t both statements be true?

If I had to guess, most of the original homeschoolers did it because they had strong, fringe religious beliefs. Then in your generations more secular parents started homeschooling but it was still a pretty highly self-selective group (a stay at home parent who didn’t mind staying at home most of the day teaching their kids, a belief they could do better at educating their kids even without the resources we have on the internet now to make it easier or the ability to use the internet to organize with other homeschooling parents, anecdotally a lot of either super-weirdo parents or hippy granola parents (more in unschooling)).

Now with the internet and WFH homeschooling is possible for many more parents than just those that don’t work fulltime and want to teach their kids full time, and it’s easier to create curricula, and easier to network and schedule things with other parents.

I do think there is probably a general problem of some pathologically bad parenting traits self-selecting much more than “normal” parents to homeschool: parents with control freak/micromanagement issues they subject their kids to, socially anxious parents who want to spend all their time at home with their family, parents with fringe/paranoid/extreme beliefs, having some kind of weird attachment or smothering issues they subject their kids to. Which isn’t a problem with homeschooling per se as much as it is selection bias in who chooses to homeschool.

Wow, all I can think is how exceptionally lucky these kids are to be taught these concepts/exposed to these problems at such a young age by a willing and enthusiastic teacher. Most people professionally employed as K-12 math teachers wouldn’t even be able to teach this kind of curriculum well. I looked at the author’s blog history to find out if they were doing this for some huge amount of money and found a blog that suggests not only were they doing it for free (skimmed it, but didn’t see any reference to pay) but they had to try to convince other parents to let their kids participate! https://buttondown.email/j2kun/archive/a-foray-into-math-cir...

In my opinion, there is a relatively unknown (to those outside mathematics) huge “privilege” gap in mathematics education that makes it so those that only follow a cookie standard or accelerated curriculum are relatively unprepared for careers in mathematics compared to those tutored (or taught in special magnet programs, or by their mathematician parents) in these kinds of non-standard-curriculum concepts from a young age. Mostly, the problem is that the standard curriculum is almost purely rote-computational until you become a college ~Sophomore and it abruptly changes to being open ended and proof-based (which is the world most pro mathematicians live in) requiring skills in creatively applying logic. So students with this kind of exposure from a young age have a much easier transition to that while also scooping up all the math-career-builders like early papers and contest wins on the way.

Those other parents probably don’t know this but OP is providing an immensely valuable service that is hard to find in some areas and which some parents would pay a huge amount of money for.

You may be interested in an Astrophysics course. I took one in college taught by an Astronomy professor and was surprised most of the content was focused on what we could determine about stars, galaxies and such based on what we could measure from them. In retrospect that seems obvious but I guess I had assumed the content would be like really heavy theories of stellar formation or gravity or something.

In my course we basically progressed from the traditional OG methods of measurement to increasingly sophisticated methods. It’s amazing how much you can learn about stars and galaxies just from combining models of black body radiation, spectral lines, and red shift with the wavelengths of the light they emit.

You are conflating two concepts, sin as religious concept and sin as “morally transgressive” concept. Because much of what you describe regarding guilt, OCD, depression arise from morally transgressive acts (eg regret for treating loved ones poorly) that nobody denies the existence of; what people increasingly deny is the spiritual concept of sin, which depending on the religion often includes things that aren’t unambiguously morally transgressive (all cultures consider stealing and unprovoked murder wrong, but breaking kosher is only a sin in Judaism because it’s a rule they think their god commanded of them).

I do think it’s true that many cultures have some kind of ritual by which you can absolve yourself of sin (be it religious or areligious), though not necessarily sacrifice. For example in Christianity the act of confession files the role. In the secular world this role is mostly filled via therapy, but a lot of other people choose to fill it with things like volunteering (someone who did a lot of bad things during their addiction may get involved in addiction outreach or speaking). I suppose given the time or financial cost of these activities they do represent a sacrifice in some way though.

Anyway, I agree that everybody has some concept of “moral transgression” which you as a religious person might call sin. But I don’t think it’s denied in contemporary culture at all - to the contrary, with things like cancel culture and such, we are much less willing to allow many sins to go unpunished - nor do I think “sacrifice” is necessary to mentally recover from the guilt it induces.

Yeah, the fact it’s an 8 million year old fossil really makes the tie to homo sapiens’ origins ludicrous. The earliest evidence for Australopithecus (where humans started to diverge from other apes like Chimps) was 4 million years old.

And there are already 12.5 million year old Ponginae (Orangutan relatives) fossils from India. Since all Ponginae are Asian but none are in human ancestry, that’d imply even earlier radiation of early African apes - which have just as much a claim to possibly being early human ancestors if they were pre-Ponginae - into Asia than this would.

Finding early-early apes in Turkey is cool but it in no way challenges the out of Africa hypothesis unless you’re really grasping at straws wanting that hypothesis to be false. I don’t even know why that’s part of the story. It’s just as cool and much less speculative to frame it as chimp-like apes living in Turkey/Asia.

Gigantopithecus as well as many hominins besides Homo sapiens, did evolve and radiate across Asia?

Homo Erectus/Heidelburgensis/Neanderthals/Denisovans were all widely distributed outside of Africa. Neanderthals and Denisovans in particular have no records of ever having their range include Africa (Neaderthals may have lived in parts of North Africa but they’re firmly Eurasian). Homo Floresensis only ever lived in Flores.

Gigantopithecus, a giant non-hominin Ape, only has evidence of living in China. It’s thought to be related to Orangutans. You probably are dismissive of Orangutans because they are only a single ape (actually multiple species) outside of Africa in the present day but the Ponginae subfamily including Orangutans and Gigantopithecus had many other species, they are just now all extinct, but all were Asian: https://en.m.wikipedia.org/wiki/Ponginae

That said, I do think there is no evidence that Homo Sapiens “evolved outside of Africa” and I suspect most people who do push this are doing so for either nationalistic reasons (we found early hominin fossils in our country, we must be where humans evolved and are the “original” humans!) or racism (not wanting to be descended from “Africans”). But it’s definitely possible there was gene flow from Homo Erectus outside of Africa back into Africa in a way that made it into the human genome, and many people have non-African human ancestry from Neanderthals and Denisovans

If extraterrestrial life were silicon-based, it may instead of cells have ceramic and metallic structures (like filaments) that grow via crystallization. I think that is what the parent is getting at - cells are part of our carbon/water based biology but maybe not all possible forms of life

I know the Apple Watch can be very inaccurate in measuring energy expenditure, but I weigh about 175-180lb and my ~6k runs almost always register at 450-600kcal.

Check out the actual paper: https://n.neurology.org/content/neurology/early/2023/08/21/W...

There is significant overlap in the distributions of prodromal and control Parkinson’s patients even for the measure with the strongest signal. While some values for that measure seem to be almost conclusive markets of Parkinson’s, there are ranges in which the overlap is enough to make it not useful as a sole measure without having a high false positive rate.

Seems promising at catching a decent proportion of Parkinson’s super early, but I don’t think by itself this could yield both a good false positive and false negative rate. Keep in mind this is an ebullient press release summarizing a more conservative publication

That’s very fair, as a kind of meditation-skeptic I probably too aggressively dismissed and mischaracterized it.

The Tao Te Ching itself isn’t prescriptive at all with meditation and the only time it really comes up is in a reference to a breathing exercise, which you could just as well interpret as a one-off for the excerpt rather than formal or ideologically sanctioned meditation. In later early Taoist texts meditation (particularly breathing exercises) was promoted as a way of cultivating various beneficial forces internally, rather than as a linear path to salvation or anything. But then even later they start getting into the Taoism stuff I don’t care for like meditation as a way of becoming immortal lol. Personally I think it’s possible to read the classical texts and come away thinking the Tao doesn’t necessarily have anything to do with, positively or negatively, with meditation.

I think of OG Taoism as a kind of deconstructivist reaction to Chinese culture at the time - highly political, rigidly Confucian - and defining itself more by where it disagreed with the contemporary schools of thought than by what it promoted. While it’s not contradictory to say meditation is compatible with Taoism, any kind of strong rules or expectations about meditation would contradict with early Taoism just on the basis of specifying some kind of rigid understanding of meditation or prescriptive rules about such a complex thing (because that would be very Confucian).

What would you recommend as an introduction to Daoism, even if academic?

I am not an expert on the subject but IMO Taoism has a very similar phenomenon to Buddhism (and indeed all religions or popular philosophies really) in that the “classical”, original, core teachings are pretty different from the organized later movements under the same name. So the context - philosophical, historical, sociological, spiritual - in which you approach the subject and whether you’re doing so on the basis of the original thing or its more organized movements might make it so people interested in it under different contexts both think the other is ignorant or incorrect.

I haven’t read the Tao of Pooh myself but I want to point this out because I think it’s possible to understand Taoism (the classical philosophy) and concepts like Wu-wei without necessarily knowing anything about Neidan and or “Taoist Magic”

Yeah, I think the Siberia-Alaska oversea connection had been established by then as well. So had the Kilwa Sultanate (connecting much of Africa).

But I think to claim the whole world, except islands, was connected we’d have to include Australia too. And before the 17/1800s there isn’t a lot of evidence of contact between Australia and Indonesia. According to this article, maybe contact started around 1500 or so: https://amp.abc.net.au/article/9320452. Though in the same article they mention some 1000 year old Kilwan coins found off the coast of Australia so you may actually be right that we were fully connected even in 1000!

I’ve had a similar experience with Taoism. It is a kind of tough nut to crack coming from Western cultures what with our extensive theologies, holy books, and prescriptivist religions. I highly recommend “The Way of Chuang Tzu” by Thomas Merton.

Perhaps this betrays some fundamental ignorance on my part, but I think understanding and internalizing the Taoist mindset makes meditation a little less relevant or necessary. Taoism IMO is the sublime wisdom of not attempting to be wise (usually manifesting as inane and unnuanced rules, or clever-sounding quotes) and not neuorotically attempting to conform to practices or ways of thinking forced on us by culture, tradition, ideology, etc. Meditation and mindfulness help incrementally in that pursuit but they are like climbing rungs of a ladder next to an elevator that Zhuangzi built for us.