1,061 karma · joined April 11, 2019
That said, there are a number of cases where hierarchical taxonomies are vital in building information systems. Some examples.
A use case where you need a way to guarantee that a space is completely covered without duplicates (important for accounting, or creating menus). Having a single non-overlapping reference space (like a map) is critical for clear communication.
A use case where you need to capture some semi hierarchical knowledge from domain experts, e.g. an org chart, or the parts of a car, or the parts of the brain.
Org charts are the perfect example of the tradeoff. Consider the US Department of Defense (don't actually do this you will loose your mind). The chain of command is famously a single parent hierarchy, and if it is not, it is a sign that something can go wrong due to conflicting orders. However, let's say I wanted to know which branch of the military funded a certain research project based on the office that wrote the RFP. This question is pretty much impossible to answer for an arbitrary RFP, and the time and effort needed to maintain the full associative ontology and keep it up to date is stupefying huge, AND it is not even clear that such an ontology would actually be pointing to anything that was actually meaningful in the real world (beyond ill defined social and economic relationships between arbitrary groups of primates).
Hierarchies are an effective way to collect knowledge from domain experts in a systematic way that does not require them to know that they are writing down a bunch of axioms, but instead can just draw a diagram -- multiple hierarchies are very important here, because hierarchies from different experts usually agree at a high level, and then differ in the details, often due to the use case for the knowledge or from the experimental perspective, not because there is some fundamental ontological difference. In this sense having multiple hierarchies is a way around the problem of ambiguity in the meaning of "is a".
I have personally made this mistake on two or three occasions where I conflated file/module behavior with class behavior because I wanted a python file to act like it was a bit more declarative. Unfortunately this leads to a world of eternal pain. You can work around it, but you should have made everything a python class and pretended like the files/modules don't exist or at least have staggeringly different semantics hiding behind that innocent little `.` operator. Python simply cannot support the desire to solve a problem in a certain way because of the structure of the problem and forces you into using its happy path patterns if you want it to work in slightly different run time contexts.
Two relevant posts from instagram engineering on the subject which suggest that best practices for avoiding these kinds of issues are non-obvious and easy to miss.
https://news.ycombinator.com/item?id=20708889 https://news.ycombinator.com/item?id=21284669
As the maintainer of about 5 little services with this structure I have vowed never to write another one. The memory overhead alone is a source of eternal irritation ("Surely there must be a better way....").
Echoing other commenters here, the real cost isn't actually discussed. Namely that there is a solution to some of these problems (re long running tasks?), but it carries with it a major increase in complexity. Its name is Celery and oh boy have fun with the ops overhead that that is going to induce.
A while back I did some unscientific benchmarking of the various worker classes for python3.6 and pypy3 (7.0 at the time I think?). Quoting my summary notes: 1. "pypy3 with sync worker has roughly the same performance, gevent is monstrously slow gthread is about 20 rps slower than sync (1s over 1k requests), sync can get up to ~150rps" 2. "pypy3 clearly faster with tornado than anything running 3.6" 3. "pypy3 is also about 4x faster when dumping nt straight from the database, peaking at about 80MBps to disk on the same computer while python3.6 hits ~20MBps"
I won't mention the workload because it was the same for both implementations and would only confuse the point, which is that there are better solutions out there in python land if you are stuck with one of these systems.
One thing I would love to hear from others is how other runtimes do this in a sane and performant way. What is the better solution left implicit in this post?
No surprise, it is hard to find people who want to do this job and are good at it. I see the demand in the academic world ('scholarly infrastructure' is a very niche place) where it is nearly impossible to hire someone who can do this work, so hearing that it is also impossible in industry means I guess it is time to start training the undergrads :/.
I have an idea for a curriculum that could teach some of the principles for this kind of work (give them the gentoo handbook for a start, and see if they can follow it to get a database up and running from a box of parts), but I suspect that mostly it would act as a way to filter out people who simply don't like the activity, and you also have to have some amount of interpersonal skills in order to understand the use cases of your colleagues ....
Anyone who cracks this problem will have solved a far more general one in the process.
The curriculum presented here is an enormous community resource and I hope the other institutions recognize the desperate need for a course like this and allocate the resources to teach it.
No need to be better than your competitors when you can simply eat them and steal their powers!
I don't have a workflow for scraping and archiving snapshots of external links, but if someone hasn't already developed one for org I would be very surprised.
In another context I suggested to the hypothes.is team that they should automatically submit any annotated web page to the internet archive, so that there would always be a snapshot of the content that was annotated, not sure whether that came to fruition.
In yet another context I help maintain a persistent identifier system, and let me tell you, my hatred for the URI spec for its fundamental failure to function as a time invariant identifier system is hard to describe in a brief amount of time. The problem is particularly acute for scholarly work, where under absolutely no circumstances should people be using URIs or URLs to reference anything on the web at all. There must be some institution that maintains something like the URN layer. We aren't there yet, but maybe we are moving quickly enough that only one generation worth of work will vanish into the mists.
Further note (edit). I think it is unlikely that a single state will ever be able to mitigate alone unless all other states stop or drastically reduce CO2 release. Maybe one state could threaten to set off a bunch of nukes in the upper atmosphere to destroy electronics and power grids around the world as a threat to encourage compliance, but that seems extremely far fetched.
As an overview I use the heading level in the google doc as the corresponding * level (e.g. h1 -> , h2 -> *, etc.). Nearly every construct in the google doc has a 1:1 mapping with some org mode convention (there are some things that I still write in org-syntax in the google doc, such as figure captions and stuff like that.
If I need to export to markdown from org I usually try pandoc (I very rarely do this though). Importing from markdown I use pandoc as well, newer versions do a pretty good job and only need a few tweaks from time to time. Latex export is straight forward using org-mode's built in exporter [1] via the export dispatcher. For full pdf export I use lualatex via texlive with all the pain that that entails.
0. https://github.com/tgbugs/gdocorgpy 1. https://orgmode.org/manual/LaTeX-export.html 2. https://orgmode.org/manual/The-Export-Dispatcher.html#The-Ex...
While there is a learning curve, the peace of mind of having a single tool that enables me to use any tool I need to use was such a relief after years of other painful writing and publishing workflows.
If you are starting a PhD today and are going to do anything at all with code, I seriously suggest you learn org-mode. By the time you finish, not only will you have a PhD, but even if you don't you will have the power of one of the most amazing authoring tools under your command! (I would say, "Learn org-mode, simplify your life!" but emacs tends to lead to other complications).
So if lisp failed for some internal reason then I don't think it was for the reason the author specifies. What should worry people more is that a language that is better in 99% of cases can fail for reasons that have nothing to do with its technical merits, but purely by accident of history or as a result of politics unrelated to those technical merits.
Trilobites were the most diverse clade in the history of the earth. Trying to rationalize why they went extinct by studying their anatomy is going to be fruitless. Same with LISP.
In direct response to the suggestion about file paths for verbs. Allan Kay says in one (possibly many) of his talks something along the lines of 'every function should have a url.' The one of surely many challenges is how to ensure that the mechanism used to populate file system paths with nested functionality (e.g. /usr/bin/ls/all to `ls -a`) don't trigger malicious behavior during service/capability discovery. Being able to more deeply introspect file data and metadata as if the file were a folder could potentially be implemented as a plugin, and I worry about the complexity of requiring a file system to know about the contents of the files that it hosts, or that the files themselves be required to know about how to tell the file system about themselves. Existing file systems adhere to a fairly strict separation of concerns, since who knows what new file format or language will appear, and who knows what file system the file will need to exist on.
Said another way I think that the primary issue with the suggested approach is that it is hard to extend. The file system itself needs to know about the new type of object that it is going to represent, rather than simply acting as an index of paths to all objects. If there is a type of object that is opaque to the current version of the file system that object either has to implement a file-system-specific discovery protocol (which surely would have fun security considerations if it were anything other than a static manifest) or the user has to wait for a new version of the file system that knows what to do with that file type.
Some thoughts from my own work. (partially in the context of OJFord's comment below)
Treating files and urls as objects that have identifiers, metadata, and data portions and where the data portion is treated as a generator is very powerful, but the affordances around the expression local_file.data = remote_file.data make me hesitate. When assignment can trigger a network saturating read operation, or when setter doesn't know anything about how much space is on a disk, etc. then there are significant footguns waiting to be fired and I have already shot myself a couple of times.
The more homogeneous the object interface the better. However, this comes with a major risk. If the underlying systems you are wrapping have different operational semantics (think files system vs database transactions) and there is no way to distinguish between them based solely on the interface (because it is homogeneous) then disaster will strike at some point due to a mismatch. To avoid this everything built on top of the object representation has to be implemented under the assumption of the worst case possible behavior, making it difficult to leverage the features of more advanced systems. As with the affordances around local.data = remote.data, if I have a socket, a local file, a remote web page that I own, a handle to an led, a handle to a stop light, a database row in a table that has triggers set, the stdin to an open ssh session, and a network ring buffer all represented in the same object system, I have as many meanings for file_object.write('something') as I have types of objects, and the consequences and side effects of calling write are so diverse (from flipping bits on a harddrive to triggering arbitrary code execution) that it is all but guaranteed that something will go horribly wrong. At the very least there would need to be a distinction between operations where all side effects could be accounted for beforehand (e.g. writing a file of known length to disk has the side effect of reducing free disk space, but that is known before the operation starts), and operations where the consequences will depend on the contents of the message (e.g. DROP TABLES), with perhaps a middle ground for cases with static side effects (e.g. the database trigger) but that would not immediately visible to the caller and that might change from time to time.
The distinction between files and folders is quite annoying (non-homogeneous), especially if you want to require that certain pieces of metadata always 'follow' a file. This is from working with xattrs that are extremely easy to loose if you aren't careful. Xattrs are a great engineering optimization to make use of dead space in the file system data structure, but they aren't quite the full abstraction one would want. It is also not entirely clear what patterns to use when you have a file that is also a folder -- do you make the metadata the outer file and the data the inner file? Or the other way around? Having the metadata as the outer file means that you can change the metdata without changing the data, but that the metadata will always 'update' when its contents (the data) changes. However, when I first thought about using such a system, I had it the other way around, and a system with that much flexibility I suspect would have even more footguns than the current system.
Another issue is the long standing question around what constitutes an atomic operation. Everything is simple if only a single well behaved program is ever going to touch the files, but trying to build a full object-like system on top of existing systems is a recipe for leaky abstraction nightmares.
While I was working on this I came across debates from before I was born. For example hardlinks vs symlinks. There are real practical engineering tradeoffs that I can't even begin to comment on because I don't understand the use cases for hardlinks well enough to say why we didn't just get rid of them entirely.
0. https://github.com/SciCrunch/sparc-curation/blob/master/spar...
0. https://docs.racket-lang.org/syntax/Parsing_Syntax.html 1. https://www.greghendershott.com/fear-of-macros/ 2. https://summer-school.racket-lang.org/2018/plan/
edit: yep here it is https://doi.org/10.1145/3184558.3191636
My initial thought was that the primary challenge here (which the author addresses in part) is in estimating the time it takes to complete a task. If you already 'know' how much time something is going to take then is there really uncertainty? How much uncertainty? The author seems to be going in the right direction, but there seems to be something more here about sources of ignorance that touches on the number of special cases in the problem space, or something of that nature. I wonder if there is any work trying to infer the number of special cases, or 'practical realities' of a problem space (maybe a kind of roughness, inhomogeneity, or irregularity?) that will ultimately be the major time cost.
Another thought is that 'bad' negative results don't have the provenance required to rule anything out, but if you know exactly what was done then you have much stronger evidence about where the problems might lie.
Finally this is deeply connected to another issue which is that sometimes we don't have the resources to devote to solving the really big problems so we never even try. The economics of cutting edge research only serves to drive us further from the hardest questions because we don't have anywhere to start because we haven't the faintest idea why we fail.
Somehow this reminds me of my first play-through of Darksouls -- the only thing that kept me going was the belief that it could be completed. My repeated failures were required for me to slowly gather enough information to see where I was going wrong. Funding basic research that is truly at the edge of the unknown is like only funding noobs that have never played before, or maybe more like funding good players from one game that come to another, they'll get there eventually, but they have to be able fail, and if you make them play Ironman mode then we might as well give up -- the game is too hard.
I have actually found it hard to go back to vim because emacs has been much easier to customize and many of the features I use routinely are no longer in my vim config.