HNHacker News
TopNewBestAskShowJobs

astrophysician

521 karma · joined May 3, 2019

astrophysicianhn@gmail.com
submissionscomments
astrophysician··on Covid vaccine makers commit to not seek approval until complete Phase III trials
Ah I gotcha yea, I just misread the article. That makes more sense and seems much more reasonable.
astrophysician··on Covid vaccine makers commit to not seek approval until complete Phase III trials
Question: is it not a standard procedure to stop a Phase III trial early when the statistics warrant it? Isn't this a pretty cut-and-dried statistical question? Naively I would think that there's nothing controversial about ending a trial early, isn't that the ethical thing to do?

I guess the political maneuvering angle before the election complicates all of that, but if the Trump administration is really willing to actually fabricate Phase III trial data, then why trust anything regardless of whether or not the Phase III trial "completes" or not?

astrophysician··on How satellite ‘megaconstellations’ will photobomb astronomy images
Not just radio astronomy but any other type of research astronomy. Astrophotography has different goals. If you want a good picture, this isn't a huge deal. If you're measuring pixel-level brightnesses using state of the art equipment, the story is completely different. The "fear mongering" you state is not fear mongering, it's an illustration of just how much of a research image is ruined by these tracks.
astrophysician··on How satellite ‘megaconstellations’ will photobomb astronomy images
...wow.

(1) astronomers are pretty darn aware of everything going on, and it's really shocking as an astronomer to see these sorts of claims. There have been close collaborations with Space X to try to mitigate some issues, but these are partial mitigations and the most important ones are only planned, not implemented. The "successful" mitigations I think you're referring to is their "DarkSat" coating, which is not actually a successful mitigation. It is still ~2x as bright as the ~maximum level of acceptable brightness for e.g. LSST.

(2) Space-X did not release any details about brightness characteristics, and they themselves claim that they were "surprised" by the brightness. To people that work in space rather routinely, that is staggeringly incompetent.

(3) > why do most not know Starlink was never visible except at dawn or dusk

Near earth object (NEO) detection only happens during these times. NEO includes "asteroids that threaten human life" so I would hope you agree that this is pretty important. Also, depending on latitude and time of year, "dawn or dusk" can constitute your entire observing window. So this doesn't really make things better.

(4) all of these mitigations should be done before companies can start doing ~10,000 satellite launches. The only reason this isn't done is outdated legislation that never conceived of this being possible.

astrophysician··on Backblaze Hard Drive Stats Q2 2020
Hey Andy, I've got a question: I went to analyze your raw data dumps and saw a couple of things that I wanted to understand:

* A hard drive may go missing in the data from one day to the next without appearing as "failed" -- does this mean you've taken these hard drives offline for some reason?

* A hard drive may appear as failed one day, and then appear the next day with a failed=0 flag. Does this mean that these drives have been serviced and returned to the front lines?

astrophysician··on Amazon satellites add to astronomers' worries about night sky
Not technically; I've got a PhD in astrophysics but no longer work in the field. You can contact me at astrophysicianhn@gmail.com.
astrophysician··on Backblaze Hard Drive Stats Q2 2020
I really enjoy reading the Backblaze analysis on this every year, it's such a valuable and interesting data set. I do have one suggestion: it would be great to go one step further and add confidence intervals for the AFR estimates. E.g. if you see 0 drive failures, you don't really expect an AFR of 0 (that is not the maximum likelihood estimate), and the range of AFR's you expect for each drive decreases as a function of the number of drive days (e.g. if one drive has 1 day of use, we know basically nothing about AFR so confidence interval would be ~0-100% (not really, but still quite large), or it would be smaller if you wanted to add a prior on AFR).

It would also be interesting to see the time-dependence (i.e. does AFR really look U-shaped over the lifetime of a drive?). That would require a dataset with every drive used, along with (1) number of active drive days, and (2) a flag to indicate if the drive has failed and of course (3) which kind of drive it is. Does Backblaze offer this level of granularity?

EDIT: They offer the raw data dumps!

https://www.backblaze.com/b2/hard-drive-test-data.html#downl...

Backblaze, god bless you.

astrophysician··on Psychedelic drugs reduce depressive symptoms by helping people accept emotions
I disagree -- "studies" like this give people the potentially misleading impression that psychedelics are helpful for depression and other mental health issues, so people who otherwise would have not used them may be more likely to, and we have no idea how this shakes out in an unbiased sample.
astrophysician··on Amazon satellites add to astronomers' worries about night sky
Sure! I think your questions are very reasonable, we'll have to see how it all shakes out.
astrophysician··on Amazon satellites add to astronomers' worries about night sky
> Is it a cost thing or a capability thing? i.e. is it possible for Starlink to have what they want without interfering with terrestrial astronomy or are we really choosing one or the other? Quite sympathetic to astronomers if it is cost thing, but undecided (maybe slightly leaning Starlink) if it is capability thing.

I think it's a bit of both; right now SpaceX is trying to get the brightness of their starlink sats down to ~7th mag (they're at 5th mag now, they have an experimental coating that should bring this down to ~6th mag), and it sounds like if they can do this the problems will be far more manageable (for LSST specifically this amounts to a few extra months of observing time, however the impact even with these mitigations is strongly dependent on the specific science case).

However, this work should be done before approval. What happens when SpaceX realizes that getting things down to 7th mag is too difficult and costly and time-intensive? Without regulations in place, I would expect they shrug their shoulders, do some PR to frame the problem as "us vs those pesky astronomers" and launch what they want to launch. We have virtually no leverage. But from the standpoint of people that operate in space pretty regularly, what we're asking is pretty basic. Astronomy of course isn't going to be thrilled with the amount of satellites growing by a factor of 10-100x, but we live in the real world. But something that monumental needs to be done thoughtfully and carefully and with respect to the billions of dollars of R&D already paid for that will be impacted.

> Assuming that within 20 years we can have sufficiently cheap space launch systems that there are sufficient geosynchronous telescopes for amateur use would that solve this problem?

Again, space delivery costs are only one factor. Not to say this wouldn't enable new kinds of instruments or make some new things possible, but it would not solve the problem, no. There are too many things that can't realistically be done from space.

astrophysician··on Amazon satellites add to astronomers' worries about night sky
You make a fair point -- I think maybe it is not totally fair to blame the companies here -- they are in a competitive environment and really the best we can expect is for them to follow the letter of the law, which it seems they mostly did. There is a suit against the FCC that argues they incorrectly claimed that their constellation would not have an "environmental impact" (which would have required an environmental impact study) but not sure how well that will hold up in court; it looks like that's best legal avenue we have at the moment to fight this.

The problem is really that there needs to be better oversight and more regulation. The laws as they stand do not address the problems that these constellations produce, because they were written in a time when starlink-like operations were scarcely imaginable. SpaceX and amazon, etc., should definitely be allowed to do cool and serious projects like this, but the oversight agencies basically saying "hey do whatever you want" is a recipe for serious accidents like collisions, as well as a needlessly enormous impact on astronomy research. If there were, say, albedo requirements or something, everyone would have to plan for that, but that is not in place today.

One part that irks me even in the absence of regulation is that SpaceX provided no forewarning of how bright their Starlink satellites would be, which astronomers were shocked by. Space-X claimed they were "surprised" by their brightness which is either a lie or belies serious incompetence.

astrophysician··on Amazon satellites add to astronomers' worries about night sky
Still a lot.

* NEO's (asteroids) are really only detectable at twilight so this would severely hamper efforts here, and I hope most people would understand how important NEO detection is from a practical, human-survival standpoint.

* Depending on your latitude/day of the year, "twilight" can constitute the majority of the observable time...

* e.g. for LSST 30% of all images will have a StarLink trail. If Space-X can somehow manage to get their satellites down to 7th mag (15% of their current brightness, 50% of the brightness of their "DarkSat" version), then LSST can mitigate a lot of impact; it will come down to extra cost + observation time for many science cases, however a lot of science cases are still SOL even with these mitigations (precision cosmology, NEO detection, etc.).

The point I'm trying to make isn't that astronomy is "doomed" (though some subfields will be) it's that we should not be in a situation where we are scrambling to fix problems like this (i.e. Space X is now helping mitigate things voluntarily, thank god, and only after a huge outcry). There should be regulations that ensure everyone is on the same page and that prevent issues like this and ensure at least a bare minimum set of requirements are met.

Those regulations are not there only because when these laws were written, nobody even dreamed of a situation like StarLink or Kuiper. I don't think Space X or amazon are "evil" or anything, they're doing great work, but they have zero incentive (except maybe PR) not to trash billions of dollars in astronomy research, and that is neither good nor necessary.

astrophysician··on Amazon satellites add to astronomers' worries about night sky
This is really a worry.

* Many experiments and a lot of science cannot be done from space, even if delivery costs were much cheaper. E.g. LSST (0.5 billion in US funding alone, + lots of other funding sources), or wide-field astronomy important for near earth object detection and monitoring (e.g. ASASSN). These things are too complex and too large to realistically do in space.

* Ground-based astronomy is inherently cheaper (no space costs, no extra engineering effort, and importantly, the ability to fix things and upgrade equipment), and therefore has a much lower barrier to entry. You can do truly meaningful research from the ground for ~$1 mill + a team of 4 or 5 core people, whereas space requires 10-100x this + an entire fleet of people with all of the associated bureaucratic nightmares + extra time required and engineering challenges. Space offers some benefits, but often times the tradeoff is either not worth the cost, or the experiment is not feasible to do in space.

* Regardless of whether or not you care about the impact on astronomy research (hey, astronomers care about humanitarian stuff too), the way SpaceX bowled through and just started launching their Starlink constellation was irresponsible, reckless, and actually quite ignorant (e.g. they were surprised by how bright their satellites were...this is basic stuff people). On the other hand, I'm not sure how much blame you can really place on a company being reckless when the law allows for such recklessness. There should absolutely be more regulation and oversight. It's in the governments' best interest because it's going to just destroy (needlessly) a lot of R&D already paid for.

Ninja edit: mentioned Space-X in here but this article is technically about Amazon's Kuiper constellation; however statement is the same (this isn't really a single company problem, this is a new-technology-without-appropriate-regulations-problem). Starlink is the largest constellation and the biggest problem (planned to have ~10x the satellites as the Kuiper constellation, they are lower orbit than anything else, and not sure how the albedo compares to Kuiper, but the brightness is > 99% of other satellites).

So: these constellations will multiply number of orbiting satellites by 10-100x, they are extremely bright (> 99% of all satellites), and they are very low orbit. So its a perfect storm of problems for astronomy.

final ninja edit: check out https://www.astro.princeton.edu/~gbakos/satellites/ to see one astronomer's page about this stuff, some actual images that illustrate how big the problem is with the extremely limited amount of starlink satellites currently in orbit, and some other links. OK that's all

astrophysician··on Stacked images of the comet, photobombed by Starlink satellites
> I'm not trying to really frame it as SpaceX. I think it's a bit more fundamental than that. There are benefits to lots of satellites providing services, there are benefits to decreased launch costs. SpaceX is only really in the discussion because they've leveraged the latter to achieve the former for their own benefit (and hopefully the benefit of others).

you're right that this isn't just spaceX, this is a much larger problem. Not of satellites in general, mind you, we've been successfully contending with those for many decades now (they do have a detrimental effect from an astronomical perspective, but we can deal with them for the most part or incorporate them when designing instruments/experiments). The difference is the brightness and number of satellites in a constellation like starlink. They are brighter than >99% of other satellites.

> Put another way, even if Starlink wasn't a thing, do we think there wouldn't be the same amount of more satellites in the sky for other reasons and for a hundred different companies 20 years from now? If this is an inevitable problem (and I think it is, unless we decide Space industry is just not worth it), then what's the solution for astronomy?

The solution is more oversight, disclosure, and planning. Space-X is obviously going to do everything to the letter of the law and no further, because they are a competitive business. What you do in circumstances like that is add regulation and oversight so everyone is subject to the same rules and standards.

https://www.lsst.org/content/lsst-statement-regarding-increa...

^ the above shows the results from some collaboration that, if Starlink follows through, could mitigate some of the effects in LSST (not completely). But this is the sort of thing that needs to be done before companies like Space-X start launching things.

> They announced publicly and filed with the FCC 4-5 years ago. The first criticism I'm seeing about telescopes or astronomy is in late 2018 and 2019, and is more worried that they won't be able to avoid collisions and we'll have kessler syndrome[2], but maybe you know of earlier criticism. Was there criticism from the astronomy community prior to this? It seems like if they thought it would be a problem, they should have spoken up. If they didn't, why didn't they? The number of satellites is increasing, has this been an entirely non-issue for that community?

Actually, yea the collisions are a problem for a variety of reasons, but I'm not even talking about that, I just mean the problems arising from the combination of very bright and very numerous constellation satellites. But that is another good point, and another good argument for international regulation to prevent catastrophes like that.

Back to your point: Space-X actually didn't release any relevant technical info that astronomers could use, so the brightness was quite a nauseating shock. Space-X was actually surprised by the brightness themselves (which is astounding to me). Here's a quote

"However, even at fifth magnitude the satellites are bright enough to pose a problem for professional astronomers who require long exposures on large telescopes to observe faint celestial objects. Seitzer said that, before Starlink, only about 200 objects in Earth orbit were that bright, but by the end of 2020 that could increase by a factor of nine because of Starlink. “So, life is going to get really interesting,” he said."

...

"While astronomers may have been aware that SpaceX was planning to launch Starlink satellites, they didn’t expect them to be that bright. “What caught everyone, principally, by surprise was the sheer brightness of the ‘string of pearls’ going across the sky,” said Jeff Hall, director of the Lowell Observatory in Arizona and chair of an AAS committee that handles light pollution and space debris issues." -- [https://spacenews.com/starlink-vs-the-astronomers/](https://...

The point is that the current legal frameworks in place are from a different era and are not prepared to deal with Starlink-like constellations, so astronomers don't really have much they can do at this point (there's a suit against the FCC right now, but who knows if or when that will succeed). There needs to be a collaborative solution and better oversight because completely destroying billions of dollars of R&D may sound like draining the swamp but all you are really doing is needlessly destroying billions of dollars in R&D. There's no reason why there can't be an oversight committee that enforces certain limits on constellation manufacturers (e.g. albedo limits, etc).

From https://docushare.lsstcorp.org/docushare/dsweb/Get/Document-...:

"There is a long history of international regulations for radio interference via the International Telecommunications Union (ITU) going back to the 1930s. However, there are no regulations in place for light pollution from space as there are for the radio spectrum....While the willingness of SpaceX to help is a sign of hope, without federal and international requirements for space light pollution, it is far from guaranteed that others would follow suit."

> Whatever it is, we should just do it now, and maybe try to get some more money out of SpaceX for it, because it needs to be done anyway. This is a "the world is changing because of technology, I wish it wouldn't" type of problem, and we've seen how those play out. You're much better off if you work with the flow than against it. That is, coal miners can complain all they want and try to get intervention, but those that see the writing on the wall and jump into those retraining programs sooner than later will be much better off. That doesn't mean they shouldn't petition the government for more subsidies to help with that though...

There fundamentally no real solution, only partial mitigations, and those are only possible with cooperation. Things like transient detections (useful for broad swaths of astronomy like cosmology, but also for near-earth object detection (i.e. asteroids) that has some very important practical benefits) will suffer greatly. Astronomers are simply asking to "work with the flow" -- we're doing absolutely everything we can to do that.

Astronomers are trying to get the word out now that this is a huge problem that needs a solution, otherwise we will lose an insane amount of pricey (and priceless) research. Right now astronomical organizations are playing extremely nicely with Space-X, not because we like them (I think a lot of people are rightly furious, some of these issues were due to pure incompetence and recklessness on Space-X's part), but because we have no other good option available to us. Luckily in this case, Space-X is relatively cooperative and supposedly "on track" to make some changes to partially mitigate some of these issues, but fundamentally some instruments are just SOL. And it will get much, much worse without interventions by governments.

This is very different than a "coal-miners" situation — we have no alternative. We either do the research, or we don't. Not to mention, there are currently billions of dollars of research being affected right now. That's already been payed for by you, the taxpayer.

> Okay, but why? I can accept this is the case, and even likely for some things (I suspected not everything could be done exactly the same in space), but without details how am I or anyone else to weigh whether they think that specific task is more or less important than the alternative?

Well, multiple things: ground based observatories have the ability to upgrade/maintain instrumentation (very very important), more sensitive/larger apertures that aren't really possible to put into space (these make it possible to do, e.g. LSST), extreme environmental conditions in space mean the addition of expensive complex engineering challenges, etc. Some things are possible to do from space, but mean 10-100x multiplier on the cost and/or development time of an already expensive project, regardless of delivery costs. Some things, like 8.3 meter mirrors that involve large machines to carefully assemble in place are just never going to be feasible to do in space.

> In that case, maybe what we need are some legal frameworks to make sure this is assessed ahead of time (not that we can enforce anything the space agencies or companies in other countries).

yea thats exactly what we need!

astrophysician··on Stacked images of the comet, photobombed by Starlink satellites
To be honest, I understand where you are coming from, and share your opinion that many issues are more important than astronomy, including humanitarian issues. I think actually most astronomers would agree with that (the ones that I know, anyway).

But the problem here is with the way that this debate is being framed. There is a false dichotomy that has been repeated here which I sum up as basically "Space-X is trying to solve humanitarian problem A, and astronomers are getting in the way! Deal with it astronomers, you are not more important than humanitarian issue A." but that's not at all what's actually happening. This is a monumentally impactful project undertaken by Space-X in secret, with enormous ramifications on science that -- despite the simplistic claims here -- are not something that can be worked around (LSST and other projects cannot be done in space, regardless of the cost of space delivery). Instead of responsibly investigating the impact, consulting with scientific agencies to either mitigate them, or prepare for the impact (as they are doing now after a huge outcry, but there is little that can be done now), they have instead opted to completely blindside entire branches of science, ruining decades of work around the globe and impacting tens or hundreds of billions of dollars in R&D. Even their press releases now say things like they are "learning how astronomical detectors work" -- that is extremely disturbing. That should basically never happen. There are responsible ways to do what Space-X has done, and they have shirked those responsibilities.

astrophysician··on Stacked images of the comet, photobombed by Starlink satellites
Yep. Lots of very clearly uninformed opinions here. Can confirm there are many advantages and disadvantages of space-based vs ground-based telescopes. Saying "well astronomers should just use space telescopes those are better" is quite ridiculous and reveals a deep ignorance for how astronomy is actually done (as though an entire field of science has not maybe had that thought?).

Extreme expense and complexity of space based telescopes is one large downside (saying "well space x will make that cheaper" um, maybe? but that will not happen for quite some time, and you realize you are parroting a PR department known for being hyperbolic? While ignoring the actual, real impact these satellites are currently having on billions of dollars of astronomy research, which happens on 10-20 year timescales?).

There are numerous benefits conferred by ground based telescopes that cannot be accomplished in space. Ground based telescopes allow you to have much larger collecting area, allow you the option to maintain + upgrade instrumentation (not possible in space, and if someone mentions "hey they replaced the hubble mirror, didn't they?" yes, at the cost of 250 million dollars), not to mention the barrier to entry for building ground based instruments is much lower and allows smaller projects to have a large impact without requiring a ridiculous amount of funding. See, e.g. https://www.astro.princeton.edu/~gbakos/satellites/ <- one astronomer's page detailing the impact these satellites currently have on his work; this is just one example. Lots of links, e.g. https://www.forbes.com/sites/startswithabang/2019/11/27/this... that will confirm the bare basics of what I'm saying here.

People thinking "oh well ML should fix that" may also be completely unaware that: no it cannot, unless you just want a pretty picture, because astrophysics is done on pixel-level brightnesses and once you saturate a pixel, there is no more information there, not to mention that even non-saturated pixels will have an insane amount of noise now, which degrades the SNR of otherwise good observations. Astronomy is done with some of the most sensitive optical/IR/radio equipment that exists (a large motivation to fund astronomy is for the innovation here that is relevant to other applications).

People saying "oh but space x is going to add some anti-reflectivity coating so it'll all be alright, Elon will fix it." Ohhh boy. Yikes. Here's the incredible page addressing this on space-x: https://www.spacex.com/updates/starlink-update-04-28-2020/ <- it's incredible because it flat out admits to an astounding level of ignorance about the impact of this, and makes several promises that may seem nice to a non-astronomer but basically amount to "we'll do something but it will not really change anything at all for any astronomy research, but we totally understand now that someone has finally educated us on the bare basics of operating in space."

> For example, earlier this year we launched DarkSat, which is an experimental satellite where we darkened the phased array and parabolic antennas designed to tackle on-station brightness. This reduced the brightness of the satellite by about 55%,

To give you an idea of how completely meaningless this is to astronomy: these satellites currently have an average magnitude of 5.5, brighter than the brightest stars, and virtually all of astronomy tries to be sensitive to > 10th mag (i.e. 1.5% of the brightness of 5.5 mag, so we're looking for > 99% brightness reduction before this even begins to mean anything) + this coating will affect IR measurements because black coatings are black because they absorb light, and things that absorb light get hot and things that are hot emit IR radiation. LSST will have a 5-sigma single exposure sensitivity of at least 22 mag (0.000025% of the current satellite brightness, no that is not a typo, so we would need 99.99997% brightness reduction).

> The huge collecting area of a larger telescopes like Vera C. Rubin Observatory leads to a sensitivity that will render even the darkest satellites visible.They are so sensitive that it won't be possible to build a satellite that will not produce streaks, in a typical long integration. There is much that can be done to reduce the impact of satellite streaks, and that starts with an understanding of how astronomical sensors work.

So nice of them to think of that, um, now? The rest of that section is a goldmine, saying basically "here is why we really can't mitigate this for CCD detectors used by all of astronomy because, well, they are looking at dim things and bright things ruin a large portion of the observations" while also concluding with

> While it will not be possible to create satellites that are invisible to the most advanced optical equipment on Earth, by reducing the brightness of the satellites, we can make the existing strategies for dealing with similar issues, such as frame-stacking, dramatically more effective.

No, no, no, no, no and no. There is absolutely nothing you can do with 30,000 5-6th mag satellites in the sky at all times. They will be in your images. Frame stacking will help to a point, but not even close to what you would need (not even within a few orders of magnitude). We will lose a large amount of effective sensitivity and a large amount of science will be completely lost.

People still shrugging this off should remember: instruments like LSST cost the taxpayer half a billion dollars and took 20 years of development, countless and priceless time investment from a large fraction of astrophysicists and astronomers (many at the top of their field), and would allow us to do incredibly impactful science, both from a philosophical standpoint (being able to observe 20 billion galaxies + stars allows you to do cosmology and other astrophysics that is otherwise impossible, see the science book: https://arxiv.org/abs/0912.0201), and from a practical standpoint (like NEO's)

So: even if you don't think astronomy is important, you should still care about this. It's a colossal waste of money due to the complete arrogance and ignorance of one rich guy. Absolutely did not have to be this way.

astrophysician··on The Bitter Lesson (2019)
I think what he's basically saying is that priors (i.e. domain knowledge + custom, domain-inspired models) help when you're data limited or when your data is very biased, but once that's not the case (e.g. we have an infinite supply of voice samples), model capacity is usually all that matters.
astrophysician··on On moving from statistics to machine learning, the final stage of grief (2019)
Hi, sorry for the delayed response!

> Can you give some examples where frequentists hit these alleged flat-prior landmines? I am admittedly a Bayesian by training, not a frequentist, so perhaps it's just my ignorance showing, but I'm not aware of any such situations.

You're probably right: I myself am also a Bayesian by training as you can probably guess but went through the usual statistics education from a frequentist standpoint, and once I learned Bayesian statistics it was almost an epiphany and much more intuitive and understandable than the frequentist interpretation (but that's just me). In all honesty, I think good frequentist statisticians and good Bayesian statisticians have nothing to worry about, since both should know exactly what they are doing and saying as well as the limitations of their analysis.

I wouldn't put myself in either the "good frequentist" or "good Bayesian" categories, by the way, I am just an imperfect practitioner, but I think that's the case for most people. My argument against frequentist statistics for the masses is a practical one: I found myself getting into much more trouble and having much less insight into what I was doing when I had a frequentist background than I did when doing things from a Bayesian standpoint, and I see many imperfect frequentist statisticians like myself running into the same problems I used to (mostly ignoring priors when they shouldn't or thinking a flat prior is always uninformative, etc.), but I admit that is a wholly subjective experience. I never once thought about priors before learning Bayesian stats, and I find many people I meet with a frequentist background also forget the significance of priors because they also are imperfect practitioners.

> Frequentist statistics generally relies on performance guarantees (bounds on the false positive error rate for tests, in particular, and coverage for confidence intervals) which are derived under the lack-of-explicit-prior, so as far as I can tell they should be doing fine. I'd be interested in seeing examples where frequentist analyses fail because of the (implicit) flat prior.

Yea, I totally agree, I just find that statistics is important in many more contexts than just this. While you can do this sort of thing from a Bayesian perspective (using Jeffery's priors or whatever the situation calls for), in my experience frequentists have a tough time departing from this type of analysis once they start diving into areas where priors are important (unless they are also familiar with Bayesian stats!)

> The Gaussian distribution is a marvelous thing. The central limit theorem is, in my humble opinion, one of the most beautiful and surprising results in mathematics.

Agree with you, but CLM doesn't always help you. You may not always be interested in the statistics of averages in the limit of many samples. I agree when you are doing this, CLM is a godsend.

> Perhaps I'm just cynical, but I'd say probably not. A Bayesian decision process is still a decision process and still subject to all the problems that the frequentist decision process (null hypothesis significance testing) is subject to: inflated family-wise error rates, p-hacking (except with Bayes factors rather than p-values), publication bias, and so on. At best getting everyone to do Bayesian analyses might be roughly equivalent to getting everyone to use a lower default significance threshold, like 0.005 instead of 0.05 (which prominent statisticians have advocated for).

I disagree here. Discussing the full posterior forces you not to reduce the analysis to a simple number like a significance threshold, and to acknowledge the fact that there are actually a wide range of possibilities for different parameter values, and it's important to do this when your posterior isn't nice and unimodal, etc. I don't disagree that sometimes (well, many times) the significance threshold is all you really care about (e.g. "is this treatment effective, yes or no"), but that's still a subset of where statistics is used in the wild. E.g. try doing cosmology with just frequentist statistics (actually, do not do that, you may be physically attacked at conferences).

But again, I want to emphasize that doing Bayesian stats can also give you a false sense of confidence in your results, I don't mean to say Bayesians are right and frequentists are wrong or anything, I just mean to say that sometimes priors are important and sometimes they aren't, and I personally find that have an easier time understanding when to use different priors in a Bayesian framework than a frequentist one.

> Sadly there's a trap in Bayesian model selection (often called Bartlett's paradox, though it's essentially the same thing as Lindley's paradox) which can be difficult to spot. No names out of respect to the victim, but several years ago I saw a very experienced Bayesian statistician who has published papers about Lindley's paradox fall prey to this. Explicit priors didn't help him at all. He would not have fallen into it if he had used a frequentist model selection method, though there are other problems with that.

Like you say, there are problems with both approaches, but my point is that when the prior is explicit, we can all argue about its effects on the result or lack thereof. Explicit priors don't "help" you, but they force you to make your assumptions explicit and part of the discussion. If your only ever using flat priors, it's easy to forget that they're there

> And here we are again calling a statistical test that thousands of people do every day "contrived." You already know how I feel about that.

I don't mean to be flippant about it or dismissive, I mean exactly what I said:

contrived: "deliberately created rather than arising naturally or spontaneously."

What test is it in lindley's paradox are you referring to when you say thousands of people use everyday? Just the null rejection? Or is there another part of it you're referring to?

> Yes, it's a very simple example, because that helps illustrate what's happening. Lindley's paradox can happen in arbitrarily complex models, any time you're doing model selection.

My point isn't that it's simple, my point is that it's incredibly awkward and unrealistic and not representative of how a Bayesian statistician would answer the question "is p=0.5" which is a very strange question to begin with. The "prior" here treats it as equally likely that p=0.5 exactly and p != 0.5, which if that's your true assumption, fine, but my point is that this is a very bizarre and unrealistic assumption. Maybe it seems realistic to a frequentist but not to me at all. If someone was doing this analysis, I would expect to get a weird answer to a weird question.

> Preferring simpler models is not exactly what's going on in Lindley's paradox,

Exactly! I'm not sure what is going on in Lindley's paradox to be honest; I don't understand the controversy here: the question poses a very strange prior that seems designed to look perfectly reasonable to a frequentist but not to a Bayesian. But I suppose this is an important point about the way priors can fool you!

> at least not the way that most people talk about Bayes factors preferring simpler models (e.g. by reference to the kln(n) term in the Bayesian Information Criterion). The BIC is based on an asymptotic equivalence and drops a constant term.

I'm with you so far, and BIC is a good asymptotic result, but I'm talking about the full solution here (which is rarely practical*), that doesn't drop the constant term

> That constant term is actually what is primarily responsible for Lindley's paradox, and has only an indirect relationship to the complexity of the model.

I mean I think we're splitting hairs here? Maybe? My point was that Bayesian model selection won't make up for a strange prior, but given the right priors, Bayesian model selection just makes sense to me. But again, this is the important limitation of most Bayesian analyses: the prior can do strange things, especially the one used in the Lindley's paradox example in the Wikipedia page.

But honestly, if you think I'm missing some important part of Lindleys' paradox, please do elaborate, I have not heard of this before you mentioned it but I still am confused as to why this is considered something "deep" but I assume that just means I am missing something important.

astrophysician··on On moving from statistics to machine learning, the final stage of grief (2019)
> I wouldn't consider estimating, say, the mean length of a population of fish contrived (unbiased estimate: x-bar). Nor would I consider estimating the probability of an event based on observations of the event happening or not happening contrived (unbiased estimate: p-hat = #successes/#trials).

Sure, maybe not contrived; my point is that flat priors may work in many "typical" textbook stats problems, but they are one of many choices, and that choice is important to be explicit about and not sweep under the rug. Because if your entire life is measuring sample means, fine, you're never going to need to think about this very much and life will be nice. But when one fine day you decide to do something more complex, these are the land mines that you shouldn't really ignore.

> These kinds of simple estimation problems and the associated statistical tests account for probably 90% of statistical practice. Dismissing them as contrived is silly.

Whether it's 90% is totally dependent on the types of problems you do. I don't mean to dismiss them, you're right for many problems MLE is just fine. I meant to illustrate that "unbiased" comes with many caveats, and that in many real scenarios flat priors are not ok.

> More generally, MLE estimates are always (under regularity conditions) asymptotically unbiased even if not unbiased for a finite sample. This means that the amount of bias decreases to zero as the sample size increases, no matter what the parameterization is.

Is this not true of the MAP for most priors? Gaussian/Laplace priors will have this property too, since priors become asymptotically less important the more data you have. If your prior is zero over some of the support, you're out of luck but this doesn't strike me as a good argument for MLE > MAP or for using flat priors everywhere. When we have infinite data, sure, priors are irrelevant, but we live in the real world where data is not infinite.

> Finally, there is very often a natural parameterization for any given problem. If you're interested in the arithmetic mean of a population, there's no reason to use a log-scale parameterization.

Sure, agree that parametrization isn't a problem a lot of the time, but it is something important to be mindful of, and this points towards, again, not forgetting that you are always using a prior and that you should think about whether or not that prior makes sense.

> Why worry about bias in other parameterizations when you can just use the natural parameterization, where the estimator is unbiased? Again, I don't think such scenarios are contrived: a very large proportion of statistical analyses deal with simple measurements in Euclidian (or very nearly Euclidian--we can typically ignore, for example, relativistic effects) spaces: real world dimensions, time, etc.

Yea, I mean sure: for easy problems, parametrization is obvious. That's kind of tautological. But sometimes it's not obvious, or sometimes for computational reasons you need to work with a log(theta) instead of theta, etc. If you're a frequentist and you're thinking life is great because you don't need to worry about priors, you're wrong and sooner or later you will get into trouble; be it a parametrization issue or something else, priors are not just something you can completely ignore. It's like saying "I always drive without looking in my rearview mirror" -- ok, great, you will be fine a lot of the time, but eventually one day you will change lanes on the highway at the exact wrong time, and you will really regret your habit of not looking in your mirror.

> If you're a Bayesian and very concerned about parameterization effects you can also use a Jeffrey's prior, which is parameterization-invariant. Notably, for the mean of a Normal distribution, the Jeffrey's prior is... the flat prior!

Yep, totally agree, I have no problem with Jeffrey's priors (when they make sense), and that's all well and good. Just to clarify: I am not saying "don't use flat priors" -- flat priors are extremely reasonable and a good idea in many cases, my point is flat priors are still priors, and you are still making a statement by using them: "lets assume all possible values of theta are equally likely a priori". Sometimes we don't really believe that but it's useful to see the implications of making this assumption. But sometimes priors are extremely important (e.g. we want a time-dependent measurement of a poisson rate, like conversions per dollar of ad spend, and conversions are relatively rare: priors are your friend here, e.g. a GP prior = Cox process or something else, even if this prior is an operational assumption)

> Yes and no. The Bayesian and frequentist approaches answer different mathematical questions, but they are used by humans to answer the same human questions, such as "do these two populations have the same mean?"

Yes, agreed.

> Indeed this is one of the primary reasons Lindley's paradox arises: the Bayesian model comparison (using marginal likelihoods or Bayes factors) gets tricked by the diffuse prior, while the frequentist model comparison (using null hypothesis testing) does not.

Ah lord, but this is a terrible justification for using null hypothesis rejection: we're almost always choosing a very simplistic distribution (e.g. Gaussian) to do this, and reducing the question to "we reject H0 because its very unlikely" is part of the reason why there's a replication crisis in e.g. social sciences, because they're taught this simplistic picture without any of the necessary nuance ('here are the assumptions we make, and under these assumptions + H0, it is a little bit unlikely that we would have observed x'). That's a recipe for disaster. Is it not much better to discuss the full posterior, "degrees of belief" and to be explicit about all of our uncomfortable prior assumptions? I prefer Bayesian model selection over null hypothesis rejection 100% of the time, especially because "Bayesian model selection" is the only logical way to do model selection, the only caveat is that it depends on reasonable prior assumptions and these are the hard part (but again, at least it is explicit!).

Also, the Lindley's "paradox" example certainly seems contrived: we believe there's a 50% chance that p = 0.5 exactly?? I just don't understand that type of analysis. Come up with a prior, derive your posterior, decide the answer to your question yourself (what is the chance that p=0.5 exactly? well, it's exactly 0%. How much more likely is it that p=0.5036 vs p=0.5? That's a better question...). By contrived, I mean that it appears designed to exploit the fact that Bayesian stats will automatically prefer simpler models, especially one with 0 degrees of freedom that is relatively close to the right answer, but that's a Good Thing (TM).

Both frequentist stats and Bayesian stats are easy to abuse: Bayesian stats gives a false sense of comfort because people don't worry enough about their choice of prior, but at least Bayesian stats is explicit about the prior!. I won't say that hypothesis testing is complete garbage, but it is quite dangerous and frankly dishonest to reduce things to a p value and pretend that's the end of the discussion.

astrophysician··on On moving from statistics to machine learning, the final stage of grief (2019)
I appreciate your comments.

> Bias has a very specific specific mathematical meaning in statistics--the difference between the expected value of the estimate (under the sampling distribution) and the true value.

Right, I'm aware of what bias is, and how it's defined, and while the statement you make is true, it misses the point: regardless of the camp you're in (frequentist or Bayesian), inference involves a prior, and that prior will affect your inferred parameter estimates (it may bias them, it may not, but flat priors do not guarantee unbiased estimates). I agree, under various contrived scenarios, you can show your parameter estimate is unbiased when using a flat prior (yay!) but what happens if you're using the wrong parameterization of your model? A flat prior on \theta is not a flat prior on \log\theta or any other transformation of theta, but if you're a frequentist what do you do about that? If you're not conscious of the prior choice you are making, you can easily introduce bias you don't want even with a flat prior.

> There are many examples of parameter estimates in classical statistics that have zero bias under that definition.

Elaborate. I assume by "parameter estimates" you mean MAP (or MLE = MAP with flat prior). E[\hat{\theta} - \theta] may be zero with a flat prior, but E[\hat{\log\theta} - \log\theta] wont be, so the bias in your parameter estimate depends on the parameter you really care about.

see e.g. [1] for an example where flat priors do actually bias inferences.

> Also incorrect. Bayesian and frequentist methods focus on different conditional probabilities and can give very divergent results even in simple cases. See e.g. Lindley's paradox [1].

OK so let me clarify because I agree that my wording is wrong: while I agree with you that there are differences between Bayesian and frequentist statistics, they are philosophical; they answer different questions: ironically the wikipedia article you linked to actually says it pretty well: "Although referred to as a paradox, the differing results from the Bayesian and frequentist approaches can be explained as using them to answer fundamentally different questions, rather than actual disagreement between the two methods."

[1]: https://mc-stan.org/users/documentation/case-studies/weakly_...

"Although flat priors are often motivated as being “non-informative”, they are actually quite informative and pull the posterior towards extreme values that can bias our inferences."

astrophysician··on On moving from statistics to machine learning, the final stage of grief (2019)
“Non-Bayesian” stats uses bias in the exact same way that Bayesian stats does, because Bayesian stats and frequentist stats are mathematically equivalent. When someone thinks they’re not using priors, they’re wrong, they’re usually using a flat prior on the model parameters, but that adds bias just like any other prior! A flat prior on theta is different than a flat prior on log(theta) or some other parametrization, and flat priors are often times the wrong choice, so this notion that Bayesian stats is some “special” type of inference and that there is some way to do inference without bias is just a very large misconception.
astrophysician··on Santa Cruz, California bans predictive policing in U.S. first
> what are you arguing for? no analysis? or do you wish to change the analysis? Or why do you claim that analysis is premature to remain on topic?

There's a difference between a team of academics/statisticians analyzing policing and crime data and then debating the resulting conclusions, and a widespread set of predictive policing tools that are designed to be used by feet on the ground. The former is great! and important. The latter is the problem.

> saying "inherent bias" is very vague and has more value as a political dog whistle than anything else... unless you'd like to suggest what the biases are and point out how you have this special knowledge and other crime experts don't.

The entire point I'm trying to make is that it's very _difficult_ to figure out what your data biases are because usually the data is _all you have_. My point isn't that I have "special knowledge" about where the biases are. I do not have any such special knowledge. Domain experts will have more "special knowledge" and by that I mean through their experience they will have hopefully learned difficult lessons about the limitations of certain types of data available to them, but even they will not have perfect knowledge about where all of the biases in their data are.

> for example, what is the bias in incoming 911 calls?

again, I do not know! Do you? Do you feel 100% sure to say that the propensity of someone to dial 911 would not be dependent on socioeconomic factors, or on the nature of the crime, or on the alleged perpetrator? Do white people feel more comfortable calling 911 on black or brown people than black or brown people calling 911 at all?

You can try to correct for certain biases if you have access to all of the relevant info (e.g. in the above example, we could in fact try to answer this question if (1) we had the actual background rate of crimes being committed, independent of 911 calls, (2) all of the relevant information (e.g. race/income of caller and perpetrator, the crime type, etc.), and even then we would be missing other biases that we may not be consciously trying to "correct" for). But even this is subject to lots of error (how do we get the actual baseline level of crimes? hint: you can't), and will miss other factors that we haven't thought of.

astrophysician··on Santa Cruz, California bans predictive policing in U.S. first
no one is objecting to Bayes' Theorem here at all. I am directly addressing your argument: there is _inherent_ bias in policing data, and doing a Bayesian analysis will make it very easy to fool ourselves into trusting the results while also allowing everyone in the room to absolve themselves of taking any responsibility for the consequences of doing so. Pretending we know how to debias policing data _broadly_ (i.e. we know how to do it in every locale and in every instance that the data will actually be _used by practicing law enforcement professionals_) is a perfect recipe for disaster.
astrophysician··on Santa Cruz, California bans predictive policing in U.S. first
Yea but you're ignoring biases in the data that are difficult or impossible to disentangle from the underlying effects. What happens when you see an area with 50% higher violent crime rate? Is that because there are a large number of crimes? Or is it because it's a neighborhood that is disproportionately targeted by police? How do you even attempt to control for that?

It's not that Bayesian stats can't help or can't offer insights, it's that it can easily reinforce underlying societal biases, and then it's harder to question because "it's science" and no one has to take any responsibility for its effects.

astrophysician··on Growing Anomalies at the Large Hadron Collider Raise Hopes
> Turbulence is far more important than HEP

maybe ("importance" is somewhat subjective and I think it's totally valid to have that opinion), but I would argue HEP and maybe early universe cosmology are more fundamental meaning they study the most fundamental building blocks of physics itself. But "fundamental" is not the same as "important" so I'm not trying to suggest these fields are more or less important than others.

astrophysician··on Why is AI so useless for business?
You could say the exact same thing about AI: people who don’t really understand it can dominate a given “AI” culture at a given company, and then others come away with the wrong impression of what it is and what it can and can’t do. To say AI is a grift is a bit silly, but at the same time it’s quite understandable that a lot of people would have this opinion.
astrophysician··on Facebook executives shut down efforts to make the site less divisive
As a total outsider following this from a distance, I sort of feel for Facebook and other social media platforms facing this problem -- they've run up against a fundamental issue for which there doesn't seem to be any satisfying solution. Misinformation and propaganda are rampant on their platforms definitely, and echo chambers that reinforce divisive worldviews have probably deepened real societal divisions, but how do you actually implement a policy to stop this? What is "propaganda"? What is "misinformation"? The entire core of Facebook's existence is advertising, which means user engagement and reach is the only thing that drives your bottom line; they want to drive users to Facebook and keep them there, and keep them engaged. They've just happened to discover a universal human truth along the way, which is that people like feeling validated, and people like being a member of a tribe. Facebook is the way it is because thats what users want, whether they will admit to it or not.

Anything that Facebook does will be perceived as making a political and/or moral statement, which they obviously are trying very hard not to do, because as soon as you take a position you alienate half of the population (at least in the US). They've apparently decided to go the route of burying their heads in the sand instead of trying to make things less tribal and divisive, which in all honesty is a pretty understandable position to take, and yet even while actively trying not to piss off conservatives they have still landed in hot water over perceived favoritism towards the left. They are damned if they do and damned if they don't.

So honestly, what is the proposed solution here? What would you do if you were in Zuckerberg's shoes? Do you campaign for regulations that take this issue off of your hands but that let the government call the shots somehow? Do you look at your board members with a straight face and tell them you're going to tank user engagement for some higher, squishy moral purpose for which there is no clear payoff?

astrophysician··on Why is AI so useless for business?
> I took envelope to mean the operating conditions under which the control system is stable. For many control systems this isn't really a confidence interval except insofar as the parameters of the control system are imperfectly known.

Ah gotcha, totally agree.

> It has more use as a marketing term than as a technical term with precise meaning.

yea exactly; not a problem for me that this is the case (buzzwords are just a fact of life), but really what should be happening underneath any "AI" solution is finding and implementing the right solution to the problem. If the right solution is classical control theory or linear models, but the business leaders insist on deep learning because to them that's what AI means, that's where there's a problem.

astrophysician··on Why is AI so useless for business?
I mean, part of the problem is that no one can seem to agree about what "AI" even is anymore; you described a control theory model that "calculate[s] the envelope" which I'm interpreting to mean some sort of confidence interval/posterior, but to me that's exactly what you should be doing whenever possible in an "AI" model, as in, control theory (it sounds like with Gaussian Processes or something) is the right "AI" approach to that problem.

To me, linear/logistic regression is AI, as are neural nets, as are pretty much any statistical model. These days "AI" seems to be roughly interpreted as "whatever a DS person deems as a solution to our business problem" at least in my very limited experience. It's all a bit silly since AI has come to mean so many different things that may not fit the traditional definition of AI from e.g. the 90's, but as long as people aren't pidgeon-holeing themselves into particular methods, I'm happy to go along with slapping an "AI" label on pretty much anything.

astrophysician··on Introduction to Support Vector Machines in Machine Learning
> I don't see the misunderstanding you mean. I said if you think a selection of the least-length or sparsest solution is ad hoc, then so is your choice of prior. Solving the same system without priors would be analogous (actually mathematical equivalent) to solving an inverse problem without regularization. Or failing to solve it in the ill-posed case.

Oh totally, I agree with that.

> As for deterministic, I mean not probabilistic. As in linear algebra and "curve fitting". As for "nice-to-have", I mean you can do machine learning without having any of the statistical understanding we've talked about and instead making various choices simply "because they work".

Yea I agree with this too, at least in principle. No issue with solving a lot of problems from a non statistical perspective since many times statistics is not the clear “right” choice. E.g. understanding that L1 regularization corresponds to a “Laplace prior” doesn’t give you that much deeper of an understanding of what you’re doing, since most people use L1 regularization to encourage sparsity. Also, if you’re more comfortable with a non-stats perspective on things, no problem approaching problems in the way you prefer.

Summary: I agree with everything you’ve said here. All that’s left is I think a difference of opinion about how important it is to understand the Bayesian perspective and I think that likely comes down to (1) the types of problems you typically work on, and (2) personal preference. I find personally that understanding the Bayesian interpretation is extremely helpful for building a deeper understanding of a wide variety of ML algorithms but I totally concede this is not necessarily a hard truth. So I stand by my advice, but will definitely agree that there are alternatives. I took the route of understanding ML without Bayesian stats first — really didn’t understand or know Bayesian stuff for a decent amount of time after I got into ML. I’ve found the Bayesian perspective has helped tremendously but that’s just me.

← PreviousPage 2 of 4Next →