HNHacker News
TopNewBestAskShowJobs

tgbugs

1,061 karma · joined April 11, 2019

submissionscomments
tgbugs··on Ontology of Psychiatric Conditions: Taxometrics
Having received a flu shot is a discrete measure. What we care about at the individual level is their response to that shot, which is continuous, minimally something like P(got-flu | got-flu-shot). What we care about at the population level is the probability that someone actually got the shot given that the records says that they did and the probability that they did get a flu shot given that the records say they didn't or that the records are absent.

The conceptual data model for vaccination in the US is a table with ~370 million rows and a column for each vaccine with the type being the date the individual last received that vaccine or a null. It is nearly impossible to construct this table with any reasonable level of data quality.

Even measurements of height are subject to error. The question is whether the error is larger than the level required for some particular use case. Thus it is not that height is somehow more discrete (height varies over the course of the day due to gravity), it is that our uses for it aren't affected by the very low uncertainty that its measurements entail. However, if you are building a spacesuit for someone and use a measurement of height taken at the end of the day in a gravitation field, they will be extremely uncomfortable when they get taller in space.

Thus even when there are measurements which are nominally discrete, if the uncertainty is greater than that of the use case, then the fact that it is nominally discrete doesn't matter.

tgbugs··on Ontology of Psychiatric Conditions: Taxometrics
The author is nearly there, but they are lacking a few key pieces in their ontology and thus their language traps them by forcing them to conflate concepts that in fact need to be decoupled.

The required concepts are as follows 1. condition or underlying biological etiology 2. diagnostic codes 3. symptoms 4. the patient 5. the person assigning the code to a patient 6. the impact of the symptoms on patient quality of life (this one is nearly always left out, I left it out before I noted as I was wrapping up the post)

There are a number of relationships between these that I won't go into in detail, but to give one example, there is "some underlying condition may give rise to some symptoms."

The problem the author complains about is an inevitable artifact cause by discretization of underlying continuous variables into discrete categories. This is a direct consequence of signal detection theory and is why you see things described as spectrum disorders, because diagnostic codes are applied to sets of symptoms that are produced by multiple different underlying etiologies, including underlying conditions which would fall into the category of "normal" in the population. This is exacerbated by the fact that there is significant variability in how the person assigning the codes assesses the symptoms. An example example of this is that medical professionals consistently underestimate the amount of pain experienced by black Americans [1], and this is pain which is about is unambiguous as you can get.

The question then becomes, can we measure the various biases in these systems to try to recover what is really going on. Usually the answer is no. In the case of Alzheimer's disease post mortem pathology is absolutely required in order for a diagnosis to be confirmed, and we don't do enough autopsies to improve our models for mapping behavioral diagnosis to the pathology (which is definitional). This situation may improve over time, but as long as we cannot directly measure underlying etiology, those who may or may not suffer from an underlying condition will continue to be subjected to the current, clearly inadequate, system.

1. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4843483/

tgbugs··on Heap-based buffer overflow in Sudo
I firmly agree that sharing passwords is bad, however I can't imagine a threat model where password sharing among multiple parties is more risky than giving multiple people sudo. As zests points out, practically there is no difference because anyone with sudo can change the root password, they are already root, so the auditing use cases is moot if the logging system can't distinguish between two users sudo sued into root at the same time.

Having multiple admins that need to be able to administrate a system might seem like another case where sudo simplifies things, but isn't that what the wheel group is for? Yes there is an M:N issue between administrators and root passwords and we all know that reusing root passwords across boxes is just asking to get pwnd, but if admins can already ssh into a user that has wheel on a system then that implies that there is another existing authentication system that surely could be used to provide the password in a centralized manner. There is complexity in such a service, but that is only required for the M:N case and if it has issues then it can be fixed once. Busted sudo? You have to push emergency updates to (checks notes) ... literally every EC2 image on amazon. If admin:server is 1:N then password manager, copy, paste, and cut sudo out of the loop entirely.

tgbugs··on Heap-based buffer overflow in Sudo
Yet another vindication for one of my long-standing practices. I try to avoid installing sudo at all cost on my systems because all it does is increase the attack surface.

Despite this, the wisdom of the crowd is that you should never su to root, for ... reasons? Fat fingering is a thing, but if you can accidentally be in a root terminal without realizing it you have done something horribly wrong.

Heck, from a certain point of view if you have someone in the habit of repeatedly typing sudo over and over again then all sudo has really done is open up every single terminal to be a gateway to the nether realm of super user privs. And in this case, more attack surface.

tgbugs··on Unlocking LUKS2 Volumes with TPM2, FIDO2, PKCS#11 Security HW on Systemd 248
It is a straw man that people use to try to try to make systemd seem like a solution, the only issue is that the problem never actually existed, just like people who try to argue that "x is dying or not attracting enough new users" in order to try to present a change they want to make as some how "solving" the problem of users leaving (e.g. "we should switch development to github"). When you look at the numbers and look at the existing systems, the stories are a complete fabrication to try to prop up a fallacious argument or process, tool, or project that is usually worse than what is really out there, not what its proponents claim is out there.
tgbugs··on New superconductor microprocessor yields a substantial boost in efficiency
Significantly less the the energy it takes to cool current processors. Nearly five years ago I was talking to a professor who was at a conference on the future of computing and there were folks there from the big 5 and he said that all of them had done the math on how much it cost to run their data centers vs run the cooling for cryogenic super conducting computers, and they all concluded that it was vastly cheaper to run at cryogenic temperatures, at least in part because they would drop the cost of running the cpus essentially to zero because they would all be superconducting.
tgbugs··on Superintelligence cannot be contained: Lessons from Computability Theory
This is quite similar to why Christian theologians eventually had to give up the idea that doing good works could lead to salvation, because someone could force God to save them by doing good works thus limiting God's omnipotence. The protestants tried to come up with some hocus pocus about being able to know that you were one of the elect by reversing the causality here "Oh I'm not forcing God to save me, it's just an indication that he already did." It is amusing to see that it is the human that takes on the role of God in the AI box game.

This is also Hume's "is does not imply ought."

From this we conclude that only idiots get tricked by the AI in the box, because it is only possible to prove that you _are_ a superintelligence, it is impossible to prove that you are not. This is also a simple application of Russell's teapot. Now, the point of the story is pretty clearly that human beings are idiots when it comes to this kind of thing, but didn't we already know that?

tgbugs··on Debugging Lisp (2015)
Having done years of development in Python I can say, all the time. The thing is that in languages like Python you just learn that it cannot be done, or that if you do you are inviting in chaos and the creeping sanity destroying little unreproducible bugs. As a result the development workflows simply avoid ever doing that and pretend like they don't miss it. When you actually have the ability to do it, you do it all the time during development and on many other occasions. You don't need fancy multiple failover distributed systems in order to keep everything working while you take a node down, you just switch out the class like it was nothing.
tgbugs··on JupyterLab 3.0
I talked to Fernando about this at Neuroinformatics 2018. In principle generating dependencies from a notebook should be as easy as parsing the ast, finding all the import statements, and then using sys to lookup what package provided that module. In practice I stopped using notebooks entirely due to many of the issues described by pavlov above, though in my case it was mostly because it was at a point in time where the Gentoo python ecosystem was having a rough time dealing with the 2/3 transition and there was no minimal amount of work that could be done to get the jupyter server up and running.

As for the reason why this still hasn't happened? I'm guessing because it is because the people who care about reproducibility have found other ways to achieve it, and as a result have no need nor incentive to implement it as part of the notebook format. Also because, if someone is running notebooks at all they know enough to bootstrap the jupyter server, which is non-trivial, or they are using a managed environment (be it remote, conda, etc.) where they are blissfully unaware that all their code has massive implicit and undocumented context.

In the mean time I have implemented the start of an equivalent system for org files because the implicit environment is much worse given normal Emacs workflows. I have encountered some significant issues and considerations which the notebook ecosystem would have to deal with, and which the default python packaging ecosystem (or any ghettoized language specific packaging solution for that matter) is not equipped to handle.

The primary issues is that only language agnostic package managers can actually bootstrap the environment that is required. What do you do if someone doesn't have a toolchain to build numpy and they are on some strange arch where there are no wheels? Do you just fail? The 80% solution here is the one we have with conda or managed solutions. The 90% solution requires that all the language communities stop pretending that they are the only language in the world and stop wasting time on their bespoke package manager.

tgbugs··on How to open a file in Emacs: a story about Lisp, technology, and human progress
The author quickly moves on after finding that expand-file-name is the issue and routes around the problem. The last time we hear from it is when it is patched out. But what is really going on?

The problem is that many parts of Emacs and libraries built on top of them have not been made fully TRAMP aware. There is an abstraction boundary that is being crossed, assumptions are being violated, etc. because all paths are treated equally, despite the fact that most parts of Emacs can identify that a path is remote (or could be remote).

Fixing the underlying issue and providing latency aware variants of commands is not easy (you usually can't just run elisp on a remote machine), but when we get there Emacs will be even more powerful!

Here is an excerpt from some notes I made back in October.

Consider these cases.

I have 10k files that I want to move on a remote server. I have 10k files that I want to move on a local server.

Where do I put the for loop?

I want to put it so that I only have to call ssh once.

Similar issue with using find and subshells. If I have to invoke a subshell there is significant overhead.

Having even a little bit of knowledge about the latency of running a single command, or the knowledge that certain commands commute under loop reordering would make it much easier to produce efficient code, basically

  #+begin_src elisp
  (do-for-values value '(a b c d e f g)
    (please-do-this-on-the-remote #'command value))
  #+end_src
should be rewritten to

  #+begin_src elisp
  (please-do-this-on-the-remote
    (do-for-values value '(a b c d e f g)
      (command value)))
  #+end_src
However, this is only safe if the values are from the local and not the remote and are not functions that are dependent on local state (that doesn't really matter in strict languages though).

The best way to approach this might be to detect and warn before running. Or not, sometimes you do just want to be more efficient, and if you don't want the optimization then disable it some other way. The reason is because sometimes it is much easier to write the latency multiplying version, and the conversion to the latency efficient form is trivial for a computer but extremely easy to screw up for a human being.

EDIT: Hah. I though I recognized Richard's reply. It was to me.

tgbugs··on ECC matters
A relevant Bryan Cantrill talk segment on this, which heightens the paranoia around this. Namely, firmware hiding correctable errors and only reporting uncorrectable errors.

https://www.youtube.com/watch?t=2104&v=fE2KDzZaxvE

tgbugs··on WSL2 corrupting Git repositories and shell history
While we're on the topic of WSL2 causing issues, I will add one that I noted to the pile. If you have WSL2 installed then the first bash on PATH is the WSL2 version of bash. For whatever reason, this version of bash has a major impedance mismatch with Emacs and org-mode. From windows native Emacs (not an Emacs installed in WSL2) if you try to run an org babel block that contains bash code, whole commands will simply be ignored. The end result is that if you blindly execute bash blocks in Emacs on windows without checking which bash is being used there can be disastrous results because a seemingly safe script like `pushd some-folder; rm -r ; popd` suddenly becomes `rm -r ` without warning. I'm guessing that it has to do with mismatched line endings since mingw bash (aka git bash) doesn't have these issues. Also, you can't rely on the ordering of your PATH environment variable to protect you because updates can change it.

tl;dr WSL2 bash is not bash but it pretends to be and there are terrifying changes to the semantics of bash scripts as a result.

tgbugs··on Detecting the use of “curl – bash” server side (2016)
I have spent a stupid amount of time trying to come up with a somewhat more secure approach and wound up with `scurl` which requires a cypher, a checksum, a url that points to an immutable file, a target path, always writes the output to disk, and will not write to the target path if the checksum does not match [0]. For an extra level of paranoia add a check to ensure that the target directory exists and is only writeable by the user currently running the command. As far as I can tell there isn't a simpler way to do this that doesn't have some hidden pitfall, and I'm sure even this way has pitfalls, because I only just realized that I had not accounted for the race condition on the target path. There are variants of this which can stream, and there was even some work to try to integrate this functionality into curl itself [1], but they are 1) not portable 2) more complex, longer, and harder to audit and 3) don't protect you anyway. Why not? Because you can't know that a file doesn't match until you read the last byte, at which point all of those bytes will already have been piped into bash. Oops!

I have also spend a stupid amount of time trying to figure out a reasonable workflow that streamlines auditing and update of the immutable links used in this pattern [2]. The same issue is faced by anyone who pins their dependencies. Auditing dependencies is a huge bottleneck, to the point where it is often neglected entirely and/or has the effect of preventing updating dependencies altogether.

Immutability of the data behind the link is critical to prevent checksum failures due to legitimate changes to the underlying data, but also means that you can't take advantage of being able to point users to a single unchanging url. For one type of security the name has to change with the content. For usability and another type of security it is vital to be able to quickly push updates. If you have users with an immutable url pointing to a bad version the only thing you can do is take down that version, but you can only do that if you control the server. If you point to a raw github url that bad version will never go away. Thus it is critical to have tooling in place that makes it easier for users (aka other developers) to find and update those immutable urls in a way that minimized the audit load.

The sandbox wont save you this time!

0. https://github.com/tgbugs/orgstrap/blob/master/get-emacs.org...

1. https://github.com/curl/curl/issues/1399

2. https://github.com/tgbugs/orgstrap/blob/master/reval.org

3. https://blogs.sciencemag.org/pipeline/archives/2008/02/26/sa...

tgbugs··on AlphaFold: a solution to a 50-year-old grand challenge in biology
My conclusion reading this is that a gradient is a gradient is a gradient. If you can minimize one, you can minimize them all. The hard work would seem to be figuring out how to transform into a gradient that your hardware can solve. It will also be interesting to see the kinds of systematic errors that will come as a result of the biases in the training set, and whether it can be used to predict what the structures would look like under slightly different conditions (e.g. pH).
tgbugs··on Let's Solve the File Format Problem (2019)
Consider a use case where you need high concurrent write performance and you don't want the things you are writing to be accidentally dissociated from the file. If you have to write to a zip file there will be overhead every time you append a new "file" because you have to rewrite the central registry at the end of the file.

Basically there is no way to say to you operating system "hey, can I get a no-nonsense, don't what to have to come up with some other conventions that will be broken, way to have some additional storage and file system paths that are just for this file so I don't have to worry about name collisions?" I already have a perfectly good unique name under which I could stash information, but the posix standard basically prevents me from using that name as a way to safely namespace things that I need per file.

However, to your original point I have partially implemented this [0] using zip files on the read side. I treat the internal paths inside the zip file as if they are rooted at the zip file itself. I haven't implemented rewriting though.

0. https://github.com/tgbugs/augpathlib/blob/master/augpathlib/...

tgbugs··on Let's Solve the File Format Problem (2019)
I have used python's pathlib to create a common abstraction across ADS and linux xattrs, but have had to stop at the lowest common denominator that is the linux implementation. It is good to know that some of the other nixen don't have the same restrictions because it means there might be a shorter route to interoperability (especially in light of some of the other threads on the linux file system that have popped up over the last couple of days). The second piece of the puzzle is also exactly as you point out, which is xattr support across all the common tools. Many of them have it already, however it is usually not enabled by default, so it is extremely easy for a user to accidentally copy only the main file data. To give an example of the issue, vim's default save implementation copies files in a way that removes xattrs. Thus, even if xattrs can get closer, the fact that they still require tools to implement correct behaviour makes is a bit like the situation with cooperative multitasking.
tgbugs··on Let's Solve the File Format Problem (2019)
Even with current abstractions this is the case. I'm not worried about users that have sufficient knowledge to intentionally modify a file. Users who know enough to find, much less modify an sqlite database can make their own mess if they want to. I'm more worried about cases where something like a sidecar file gets deleted or overwritten, or fails to be copied along with the main file. Some programs and file formats come up with their own way of embedding sidecar metadata in the main file, but that is a complete hack that has to be used to work around the fact that the sidecar file cannot be made inseparable from the main file while still being just another file. Alternate data streams on windows can sort of do this (xattrs on nixen aren't really up to the task) but they are completely non-portable.
tgbugs··on Let's Solve the File Format Problem (2019)
They say file format problem, but one of the reasons why there are so many different file formats is because current file system abstractions are fundamentally insufficient for a wide variety of use cases, particularly ones where you need to ensure that a user can't accidentally separate or delete a file. How do you keep a cache directory that backs a file with the file if the file system doesn't let you put folders "inside" of files? This is only one of many issues that have resulted in the proliferation of file formats to meet use cases with different performance characteristics that all have the common requirement for data to be bundled together an not be separable. File systems that can decouple the notion of access and data locality from inseparability simply do not exist right now, so we are left with the horrible abstraction that is the archive or zip file.
tgbugs··on The AMA is a tax-exempt hedge fund and licensing corporation
Don't forget that CPT is also a massive money maker and completely proprietary. Want to do open research to improve patient care and health and use CPT codes? Probably not going to happen [1].

1. https://www.ama-assn.org/practice-management/cpt/ama-cpt-lic...

tgbugs··on Booting from a vinyl record
They definitely do, but we have much better historical data about the time course and about how to care for them so that they don't go as quickly. There is also the possibility to stamp them out of something that is more resistant to decay.

Some of my thinking was that so long as the decay was decorrelated multiple copies of the same record could provide redundancy, though that applies to many other candidate media for these kinds of projects as well.

The best option for long term archive is probably still characters printed out on paper, but getting that back into a working system can be quite a bit harder depending on the size of the codebase you want to restore. Of course all these assumptions are probably wildly inaccurate about the relative probably of events, e.g. of someone having a record player and a functional CPU while also not having a scanner. However, another objective here was to reduce the number of steps that had to be performed by a human being that could go wrong. A run of the mill office scanner doesn't quite fit the bill because it might have been cannibalized to be the CPU and because scanning in 40MB of printed pages is much lower bandwidth than the record. So end the end the calculation comes out to be the relative likelihood of having a record player and a CPU but not a modern automated book scanning system, and there the odds seem very much in favor of the record player.

tgbugs··on Booting from a vinyl record
I recently became curious about whether vinyl records might be a good choice for long term widespread backup of the information needed to bootstrap back to a full working Turing complete runtime, sort of as seeds for some future where much of the knowledge about computing had been lost. Somewhat absurd scenario, but interesting from a technical point of view due the the constraints you have to optimize for.

Depending on what assumptions you make about the effective bandwidth available on a 33 rpm lp record is somewhere between 225MB and 15MB. That is easily enough space to fit a full fledged implementation of Common Lisp on somewhere between 1 and 4 records (SBCL's working tree is 40MB, and with its .git folder it is 152MB). There are countless other factors that would need to be considered, but I still like to imagine a sci-fi story about the search for the 5th record of lp-lisp needed to reboot civilization! The fact that someone has actually done something even remotely related to this is fantastic.

tgbugs··on Native Intelligence
This is one of my all time favourite pieces. Whenever it comes up, I am also reminded of another piece which recounts what happened to the Native American populations and politics as if it were an invasion by space aliens following a plague that wiped out 95% of the population, can't find it at the moment but will keep looking.

Edit: here it is courtesy of the previous HN thread. https://slatestarcodex.com/2013/11/28/the-story-of-thanksgiv...

tgbugs··on Git is too hard
Compositional UIs are extremely hard to learn. You have to be a composer! Quite a high bar, but once you can clear it, you can do things that others consider impossible. Like writing a Symphony Fantastique. Wrapping the composable interface so that the 80% use cases don't require the user to be a composer is the challenge here, and what is interesting is that GitHub seems to be failing so miserably at this according to the folks in this thread.
tgbugs··on Millions of animals missing from scientific studies
Yes. I did not mention this in my original comment, but internal accounting and oversight by IACUCs is pretty good. They know within a couple of cages how many rodents are on campus at any given time (module reproduction etc.). If we want the public record to be able to account for this then it would likely have to be in another venue, because animals whose data does not go directly into a paper could be "involved" in the exploratory work for tens of papers. How do you prevent double counting, how do you know which animal whose data was not used was reported in which paper? Mostly you don't. The IACUC has it, it is buried in lab notebooks, etc. and if someone needs to be disciplined for misuse it is on the IACUC.

This is different from the UK where the 3Rs are much more strongly enforced, to the point where I remember asking a question back in 2015 to the then head of UK animal research about reproducibility, and getting the answer that he wouldn't approve the use of animals just to replicate an already completed study. In the US in some fields animals will be used just to replicate a result because another lab needs to know for sure that it is real before expending even more animals for a potentially useless follow up study. The way the numbers play out in practice, we would be much better off doubling if not 10xing the number of animals used in initial publications to avoid the 10x replication studies that will be done inside other labs to make sure that the result is real. Of course if we did this then the publication rate in many fields would be cut in half, or decreased by an order of magnitude.

tgbugs··on Millions of animals missing from scientific studies
The original article saying that more unsuccessful research should be published, and I agree. The sentence quoted in gp is a bit hard to parse, but it is just another way to say unsuccessful research shouldn't be hidden and ignored. The context for the sentence is a bit more from the funding side, where I usually only half jokingly say "99% of all funded grants are successful!" Which is the complete opposite of reality, where 99% of experiments fail.
tgbugs··on Millions of animals missing from scientific studies
Full disclosure, Anita is a lab mate.

One of the things that is latent in this article is that in the US you are supposed to have done a power analysis in order to justify the number of animals you are using in your study. Almost no one does this, and it is not surprising -- if you are doing cutting edge research you are in unknown unknown territory and any power analysis is likely to be no better than a guess. In a sense it is farcical that exploratory research needs to pretend that it is always successful.

Countless animal experiments are uninterpretable simply because they were poorly executed by first year graduate students. There is no other way to learn. Write them off as animals used for training if we want full accounting, but researchers' time is scarce enough as it is so asking them to publish uninterpretable results is a non-starter.

On the other hand I think it is important to publish as many results as possible to avoid file draw effects etc. Data publishing might be one way around this, but at the moment few labs anywhere in any field have the know-how to publish raw data for negative results, even if it is just sticking files in a git repo and getting a DOI from zenodo.

tgbugs··on Humans Are Bad at URLs and Fonts Don’t Matter
Great link. In a broader context this is makes me wonder whether the development in-group nomenclature (slang) developed for similar reasons, to defend against collisions in terminology with others where misunderstandings could lead to disagreements and fights. You can still speak the same language, but if someone overhears you (tries to interpret what you type into a url bar) they don't understand unless you have intentionally specified the mapping to the global nomenclature. Of course this would require that browsers stop trying to redirect you to who knows where when you type in a single word in the url bar so that not typing in the local name would lead to a failure.
tgbugs··on Org Mode's new site
We had this on the features page. Not sure why it was removed, but here is the commit [0].

0. https://github.com/tecosaur/orgmode.org/commit/54ad9895065cb....

tgbugs··on YouTube-dl has received a DMCA takedown from RIAA
This should serve as an example to people who want (for example) the GNU project to relax their contributor copyright assignment practices. It only takes a single moment of complacency and the vultures will swoop in. In this case it was because of sloppiness around which urls were used in a demo. Yeah, all urls are the same to the guy who is building a url slurping service (which is why the take down will likely be defeated), but there is an entire profession dedicated to nothing but nit picking, loophole hunting, and general levels of pedantry that we have just come to call it lawyering. Layers are like cliffs or handguns. Stay as far away from them as possible, they are facts of life, but you really don't want to have anything to do with them unless you have no other option.
tgbugs··on Cryo–electron microscopy breaks the atomic resolution barrier
Direct Detection Device.
← PreviousPage 7 of 9Next →