arcfide

> Not my experience at all - if anything I'd say visual aspects tend to be a marker of less rigorous communcation.

I would point to the field of combinatorics, the traditional proofs of both the ancient Chinese mathematicians as well as those of the West, both of which took on various elements of geometry and spatial reasoning for a significant number of their proofs when other tools were not yet available. The development of algebra I see as a chiefly visual and ideographic one, even tangible or malleable one. The development of UML diagrams another. Flow charts another. We have the abacus and Chinese counting sticks, as well. And finally, while poetry is not specifically rigorous, it is efficient in a way that few other communication methods are. And we find a great deal of "visual cue" elements in that field. In physical sciences and statistics, visualization is a very important tool. Mathematical notation itself is largely spatial and visual at scale.

As for legalese, I would argue that legalese is perhaps well designed for experts to be complete, but not for clarity. Comprehensiveness is different that clarity of rigor. And as for philosophy, vocabulary is not enough. And you'll note that some of the best notational systems to arise came from the philosophy departments in working on logical systems. Those are all usually notationally represented using ideographic, rather than natural language forms. And even some Eastern philosophers who wrote very verbosely tended to make their arguments from visualizations in the mind to make their point.

Musical notation, again, has evolved into a spatial, visual notation. A large number of traditional writing systems were ideographic, including ones we now consider alphabetic/phonetic.

It's a fair point you make. Regarding semantic density, what you talk about is density maintenance at a single point, that is, the density of a single function. I'm not saying that APL is unique in that respect. I'm saying that APL seems uniquely well suited to remaining at the same semantic density throughout the entire code base, and that the semantic density found in APL code more readily, IME, handles the shifts in common versus uncommon concepts at the same density level than I have found in other languages.

You're right that you can change piecemeal, but without knowing your specific case, it's hard to say what you focused on integrating, and thus, whether you were just integrating the language or integrating a style.

If you're willing to talk more about this, feel free to email me, I would love to see more details of your experiences trying to integrate and get more details on your project. This is a part of my future research agenda, and so gathering information on what has worked and didn't work for you, and making an attempt to understand all of that in your case would be very valuable.

As for the issue with big migration, I absolutely agree that it makes it hard to study. However, that's where academic research can come in. I'm specifically taking a look and building a set of research studies to isolate specific factors related to the APL phenomenon and understanding their relevance and how they place a role in the HCI of programming languages and developer culture. There's almost no one doing these kinds of studies, so the main problem at this point is that we have anecdotes, but not a lot of large, well done studies to deal with the issues. I hope to change that.

As for working with customers, are you aware of any other large projects that directly involved customers in their programming? That actually expect their customers to read source code, when the customers are not programmers themselves? I'd be interested in seeing the results of such an effort in other languages, as the only places where I've seen this level of customer integration used for non-computer science/programmer type customers (that is, those with no programming background) is in APL.

I think a big problem with people trying to work with APL code is partly training. There is very little out there on how to do "good APL" code. You get trained all through your development career on how to work with traditional programming languages and what are considered best practices there, but it is not at all clear or obvious to me that the best practices that you learn are actually the right ones once you start moving far from the traditional programming language bent. I would include not only APL in this, but Prolog and Agda, for instance.

I would even go so far as to say that some best practices in other languages are anti-patterns in APL. If your team has made a concerted effort to avoid and educate developers to avoid anti-patterns in APL and they have actually worked at changing that aspect inside of your development team(s), I would be very interested to learn more about this. It's hard to find good case studies in this material in the wild, and so if you'd be willing to share some of yours, I'd be very interested in getting good data.

Please contact me by email (arcfide@sacrideo.us) if you would be willing to discuss further.

Update. I'll be giving this live session on Monday the 13th at 3 PM EST. Stay tuned for a post on HN near that time for the link to the live session.

I'd be very surprised if your industry really did use complete pseudocode and only elided error handling. On the other hand, you're sort of assuming in your conclusion that pseudocode is the "better way" for languages because that's what people use, but you're leaving out the initial bias. I would argue that if you made current industrial languages more like pseudocode, you'd probably do better, yes, but it's a local maximum derived from an assumption of what the end result will be.

In other words, people use pseudocode because it's close to the code they intend to write and represents their current notational expectations. It's an enforcement of legacy methods of thinking.

But many people have admitted that there is a problem with writing pseudocode style programming for modern hardware performance, where taking advantage of parallelism is important.

Furthermore, I would argue that academia is relevant because it's one of the few places where the ideas are more important than the executable. If the ideas are communicated clearly, then you've succeeded. If we really want to program for the human, then we want our programs to be focused on the communication of ideas, and not machine-focused. And the reality is that if you take the machine away, and focus on human-to-human communication, without any "industrial" bias (expectation of machine execution), then rigorious idea communication is almost always pictorial, visual, and ideographic. Fruthermore, the notations that people develop and have developed over time to communicate ideas never end up looking like mainstream programming languages. As people work with ideas, math notation is the quintessential notation for communicating human ideas rigorously. It is highly evolved for human consumption, and manipulation, rather than machine-focused.

I believe there have also been some studies on how people describe processes without any computing background, and it's inevitable that many of the core "serial" programming concepts are not "natural" in human though, but a very acquired taste.

Again, I would be surprised if you put a bunch of industry or non-industry professionals up to a white board and had them illustrate their ideas rigorously to one another on just that whiteboard, that they would naturally gravitate to any real programming language. And I doubt strongly that they would actually continue to use pseudocode at scale on the whiteboard.

I disagree a little bit, but agree with you in part. You're drawing a distinction between codebase tersity and business processes.

See my other reply here about Direct Development. One of the better ways to do APL development is to write your codebase in such a way that you don't just onboard other developers, you onboard the customers as well. You don't just talk about business processes and then have some IT team turn that into code, the customers themselves work off the code together with the developers directly on solving the issues of business process. The result is that the code literally reflects the needs and expressions of "why" because that's the document or artifact in which the customers write their business processes down. And at that point, tersity matters. It's important that semantic density of the code keeps computer science-y stuff away from the customer, and brings the domain vocabulary of the customer to the fore in the code itself. You can't do this if you have your code littered with words and vocabulary that the customer has to parse and work with. The words that the customer sees, i.e., the variable names, should, ideally, only come from their own domain. In this way they can follow the code and see how the data flows.

That's the way to make business processes the core component of everything. You're reducing the development cycle down to a singular point, or very close to that.

There are a number of papers on this, but this idea of "user pair programming" has been successfully used in a number of APL projects, and is often the basic interaction/development methodology through the community.

And I would argue that the most important thing about the code base is that it accurately expresses the solution to the needs of the user, that it is capable of shifting with the needs of the user as quickly as the needs of the user change, and, finally, that it be as easy as possible to verify with the user that the code in fact is an accurate solution to their needs.

You could say I care more about whether the users can get what they need out of the code, including verifying that the code is what they intend, than about how easy it is to onboard other developers who are separate from the customer. That's best achieved by making the code easier for the user to read, rather than easier for the broad range of developers to read.

There are some good points here. I'm fine with reduction in constant factors when it comes to productivity. I personally find that those constant factors are the bigger issue in day to day work anyways.

And part of the problem is that people aren't thinking of the whole context and picture when they talk about conciseness. Conciseness is not the end goal in and of itself. It is a design constraint that creates pressures to help mentally force you to achieve the really important things. You can't blindly chase conciseness. You have to have a reason for choosing conciseness and fit that within a broader aesthetic with the big picture always kept in mind.

I think this is one of the reasons that people balk at APL and fail to integrate it into their code bases. We don't teach people how to design or write "poetic" style code. I believe one famous author (Perlis? Kay?) talked about as "lyrical" programming. It's not something we teach in schools, and it's not something that most people ever learn intentionally. So, people see APL and they get the take away that the reason this code is so neat is that it is so "short." So they often do one of two things. They try to make their own code short, or they see the APL operators and think, "I could do that in language X" (I did this at first with Scheme). But as you've discovered, that fails. It doesn't really seem to work well when a lot of people try it. But why? Why do we get such great results in some cases, and such poor ones in others? Why does it seem so hard to have "APL in Language X?" There are two or three major ideas that I think contribute to this.

The first is what makes APL code unique. It is not its shortness. The tersity of APL is just the surface characteristic of a set of synergistic design aesthetics that result in short code that can actually be useful. Yes, it is short, but it's not shortness for shortness sake. That brevity of expression is a part of a bigger picture. Iverson described some of how they saw this aesthetic himself in this "Notation as a Tool of Thought" Turing Award lecture. However, I think he misses a few things that were just "obvious" to him. One of those ideas would be the concept of the design tension between generality and specialization.

The second idea is this concept of semantic density. One of the reason, I believe, that APL code is unique, is because of the ability to remain largely at a single abstraction level and be so productive with minimal abstractions. The regularity of semantic density is really important. It allows you to make your variable name choices higher impact than they might otherwise be. But this semantic density issue means that by definition, inserting some small bit of APL code into a larger code base is fundamentally breaking semantic density. It's easier to deal with uniformly verbose code or uniformly terse code than a mixture of the two together.

Finally, the contribution of APL style coding is not primarily one of semantics, but one of design. That's why it doesn't work to just implement and APL semantics with terse naming in some other language and expect it to work. The semantics and domain and abstractions available to the APL programmer are a part of the whole, not the whole itself.

So, I'm not surprised that you struggled with integrating APL style programming with other styles. It's a case of trying to have your cake and eat it, too. It's also a reason that some groups can come to loathe the APL code base, because very often they are used to doing it another way, and the APL code base interferes with their approach. Of course, it's always possible to write bad APL code, too. But the style I'm talking about isn't strictly about just APL.

What most people encounter with integrating APL into other code bases, IME, is a fear of deleting their code. There's incremental development, and then there is glacial development. When I've worked with a few people on code integration, one of the things that trips them up is that they tend to think so much at the micro level, that they don't get any benefits from APL, because they don't understand how to leverage it. What they end up doing is trying to fit APL into their architecture, because they are too afraid to change the way that they do something. They want to try little "piecemeal" integration here and there. But that doesn't work, because having some mysterious three line piece of APL surrounded by hundreds of lines of other code doesn't work nearly as well. You get almost none of the benefits of APL style coding and all the potential social issues.

You can always write verbose, old style code in APL if you want, and then it will integrate fine, but you are just writing Python in APL then, and your gains will be minimal, at best.

Instead, you have to shift your granularity of integration. You have to think of replacing whole, independent units in your code base at a single time. This sounds scary when you first mention it, because people are imagining some massive change in the system. However, if you can take a single unit or module in your system at a time that is truly independent of the rest, then you can choose to have a separate style for that code and a separate aesthetic without incurring the same costs that you would have elsewhere. It would be like having part of your system written in Python and the other written in Haskell. Sure, you've two languages to work with, but as long as you're not switching back and forth all the time in the middle, you can focus on doing good design for each of the languages.

If this is done right, then the selected code base should go from a large piece of code to a very very small piece of code, and the time it takes to maintain that code should be minimal, requiring maybe one or two people to work on maybe a hundred lines of code instead of thousands or tens of thousands or something like that.

And really, to do it really right, this group needs to be working directly with customers on this code. Then you start to see the wins in APL style.

So, the shift in APL style programming is as much aesthetic and cultural as it is semantics and technical. If the methods aren't fundamentally forcing a change in the way you think about your code and the fundamental complexity of your code, then you're not using APL correctly.

Because some languages are better tools of thought than others for certain disciplines. Linguists have demonstrated that language itself has a shaping on the way in which people approach and see problems.

While I could have chosen Japanese, it wouldn't suit the purpose as well.

Moving this into the domain of programming languages, the point is that working with APL as a notation fundamentally changes the way you see problems. It facilitates a style of thinking and working with code that promotes the ends I'm working on.

I'm sorry that you don't like analogy, but analogy is important to me, and I'll have to throw another one in here. To me, it's more about constraints that shape design than anything else. I'm not writing "English" style stuff in Japanese or Chinese, I'm writing, say, Chinese Poetry in Chinese. It would be exceptionally difficult to achieve the same result in English. Could you literally express the same content? Sure, maybe, but it would depend on what you counted as important. And the result would be very very very verbose indeed, and would no longer have the value that the same thing in Chinese poetic style would have.

In PL, I could have written the same compiler in CUDA, and I would guess that it would take at least 10,000 lines of code or more to make it work.

I could try to write it in any other number of methods, and I would argue that not only would the results have been more ugly, they would have been much less maintainable. I could have implemented the same literal algorithms with the same literal content, but to get the Human factors that I want, I would have a very very hard time of it.

This is a big problem I see in the PL community. We don't, as a whole, understand or care to study the impact of notation on our thinking. It's all just "syntax" and we can choose what we like. But that's not really true. Just because I could write an APL library in Scheme does not mean I will be able to duplicate the efforts of APL in Scheme. Going the other way, I've for a long time tried to imagine some way of getting syntactic abstraction a la Scheme's syntax-case into APL. I can't find a way that wouldn't basically be useless, because the human factors involved change the game.

So yes, you an translate Chinese poetry into English, and no one will consider the translation to be as good. Now, what most people are suggesting is that you can create Chinese poetry by starting from English and writing the same thing there. There are a host of reasons why that doesn't work.

It's not just about meaning/algorithm/semantics, but about the experience of working with the code day to day and how that affects your ability to work in your problem domain. The design of this compiler enables better, simpler collaboration, and more adaptability and flexibility than other designs I have tried. It obviates certain documentation burdens within the team that is involved in working on this code. It simplifies deployment, maintenance, and all sorts of other things.

Perhaps the biggest boon to working in this way within the team involved in this code, is that everyone, from the managers down to the users of the compiler, are able to discuss and have conversations about making things happen, handle bug reports, and deal with architectural design issues, all without anything else in front of them except for the compiler code itself. We don't need other documentation, we don't need diagrams. We can all literally, from the top to the bottom level, work off the one single code artifact. Everyone can get what they need from that single code base, without needed extra levels of "human documents." Because the code itself is a sufficient entity for human discussion.

You're asserting that this isn't multi-programmer friendly. I'll agree that it's not "code monkey" friendly, but I disagree that it is not oriented towards multiple programmers. And the APL language has almost all the features you would expect from a modern multi-paradigm language, including branching, control structures, recursion, exceptions, objects, frameworks, interfaces to other languages, and so on and so forth.

But APL was designed from the beginning to enable human communication. I would argue that almost all programming languages fail to be a good human medium of communication. The evidence I give in support of this assertion is that if you look at how people write when they think the computer won't need to see the code, such as in academic publications on computer science, see what they use in the paper. Almost all of the people who implement their ideas in one language or another fail to include the entire code in their papers, and they usually include some mathematical notation and diagrams to explain their ideas instead. They may include some small snippets of code, but they rarely if every include the full code. Dan Friedman being an exception that proves the rule, if you will.

If you then take a look at how APLers communicate when they have ideas, you see code all the time, all day long. The APL community is the only one I've seen that regularly can write complete code and talk about it fluently on a whiteboard between humans without hand waving. Even my beloved Scheme programming language cannot boast this. When working with humans on a programming task, almost no one uses their programming languages that primary communication method between themselves and other humans outside of the presence of a computer. That signals to me that they are not, in fact, natural, expedient tools for communicating ideas to other humans. The best practices utilized in most programming languages are, instead, attempts to ameliorate the situation to make the code as tractable and as manageable as possible, but they do not, primarily, represent a demonstration of the naturalness of those languages to human communication.

I look forward to convincing you of the simplicity of the code base. :-)

The sentence was a bit of a tongue-in-cheek sort of rhetoric. In particular, if you look up most of those words in the relevant domains, they're all standard practice ideas that are well understood around their various parts. Importanntly, none of the words or anything said in the sentence is really complicated, and if you were familiar with all of those ideas, then it would be an easy sentence. However, I wrote it in a way so that it appears to be obtuse and a bit ridiculous on the face of it. Basically, attempting a bit to mirror the code itself. The sentence very concisely and neatly describes the architecture of the compiler, but only if you know what you're reading.

One of the biggest issues with reading the compiler is that most people will have a "part" of the picture based on their backgrounds. If you have written compilers before, the overall compiler design and the strategies at a macro-level used for it will make perfect sense, but you'll balk at the data-parallel programming style. If you are an APL programmer you'll be very familiar with the basic tricks being used in the compiler and you'll easily be able to see at a micro level what's happening with the code, but not having the background in Programming Language Design (such as you might receive at Indiana University's PL course path), the overall design and the intent of the whole system won't be intuitive to you. This means that likely for anyone new coming into the code, there will be parts of the code base which feel very foreign to them. Of course, I don't know of a way of doing something new without making people learn a thing or two to understand it.

Fortunately I'm not the only one in the world that has experience in all of these areas. And even better, the code is straightforward enough that once you do learn the basic skills, it's easy to work with. But there are precious few APL/Array oriented implementers out there, and probably less than a handful of compiler writers out there that specialize in this area of languages. I hope that will change and that I can convince people that this approach is actually a very neat and easy one to work with. But that's not easy to do.

It does. Furthermore, he's "simplified" APL in K to require less infrastructure, with fewer primitives, and the like. Combined with some clever, and some would argue, devious programming practices, he's able to keep things pretty small. I don't know if the interpreter is still that small, though. If someone reminds me, maybe I can talk about scrolling. :-)

Since I believe Whitney wrote the J incunabulum, I suspect that it looks very similar. The code is actually quite simple and straightforward if you take the time to read it.

I would tend to agree. It's certainly a generally good rule of thumb. However, I've honestly struggled to find a way to document the naming conventions that is useful. Every time I've wondered about a particular name, it's faster for me to go to the definition sight of that name than to seek documentation, and the documentation for the naming conventions might exceed the size of most of the compiler, simply because it's hard to write out all of the aesthetic and stylistic choices that are almost self enforcing through an overwhelming pressure of context when you're actually programming in the code. Basically, while I am a huge fan of documentation, with this style and the approach I'm taking, I've found it exceptionally difficult to create up to date, meaningful, and reliable documentation of any sort that isn't totally useless. Now, I do have public API documentation, but internal developer docs have not proven to be helpful at all for any reason in this compiler. I've tried on multiple occasions to make it happen, and it just doesn't work here. If you have a way to do so, or even maybe if you want to talk with me and make it happen, all the better, but when the code base changes so fast all the time, it's just very hard to keep documenting this fluid thing that keeps changing. In some ways that includes the naming conventions.

I'm quite open to people providing ideas on how to properly document this project outside of "big idea" documentation that I'm currently doing with the paper publications. I've yet to be able to find anything that works. The standard best practices don't seem to flow well at all, but maybe I've missed something.

At this point, if I get sick, the code doesn't move much. If I were permanently disabled, this someone else could take over. I have people contribute bugs, tests, and other things fairly often. If you had to temporarily work on the code base and weren't familiar with the background of the project, I would say you'd be lost. It's just not the sort of thing that you can start tweaking things here and there so easily, because almost everything that needs changing is a matter of addressing architectural or serious questions that require you to really understand the project. Because of the way the code is written, there's basically no "code monkey" type work. That means that you only do meaningful work, but it also means that only people who are knowledgeable architects can work on the code. You can imagine the same thing in other code bases. Imagine that you didn't need any of your lower-level programmers anymore for work because there was nothing for them to do. Now imagine how the bus factor changes on the code when only your chief architects are necessary for working on that code base. That's very nice in one dimension, but it does create quite a different picture.

You're right about the code examples. I figure that people were already posting some code snippets. I wanted to give the big ideas rather than any specifics. The reason for this is basically that if you take any single line of code out of context, it's a bit hard to explain why I'm doing the things that I'm doing. It's very much a macro design, which is why I am offering the live session to go through. It's sort of, but not quite, an "all or nothing" thing. if you let me sit down with you and go through the entire code base, then I can explain how it all fits together and why things are the way they are, but if you just take a single piece of code out, you're missing the picture.

If I took a single compiler pass, out, for instance, you'd have between 1 and 12 lines of code to look at. I could explain a few features, but how would I explain that when you look at this piece of code you're able to see it entirely in context? Well, I can't, because the code it completely out of context at that point. Or what about demonstrating how the naming conventions exhibit structure informative regularity? Again, I can't, because that's a visual design element of the code. It's something you have to "see" by looking at the whole painting as it were.

The naming convention is actually a great example. Out of context, there's apparently no rhyme or reason to it. But in context, it forms a key component to the visual regularity and continuity throughout the code. The names are an important part of how you can see the structure of the code. It helps to orient you in the big pie. But if I were to quote a single line here, there's now pie to look at, no sky to navigate by. It's just a single constellation. By analogy, it does less good to say, here's the Big Dipper, it's useful. But why? Because it's easy to find amidst the context of starts and its shape helps you to find the North Star. But on its own it doesn't seem as valuable. At that point it is just another constellation. The same thing happens with this code.

So I'll go through and explicate it all in detail in the live session, where I can provide the "painting" and workflow in its entire so people can see how it works. Then you can see how my comments here match up with the code.

As for what is harder and what is not, it's not really so much a matter of easier and harder. By replacing all of the normal techniques with equivalent ones, it's more just programming in a different style that nets more benefits. I wouldn't say it's fundamentally easier, because the hardest part of any problem is reasoning about the problem itself, but I do think that you get a lot of benefits for writing in this way that is not really harder than the traditional methods, either. There's a sort of 1 for 1 replacement of traditional programming techniques with new techniques. The new techniques solve similar problems, but are more parallel, and more concise.

I've structured the points free style so that it's basically like working with any expression, I just am working with expressions that build functions instead of expressions that build values. The compiler is very functional in style, and the entire core of the compiler is just a single data-flow graph if you get right down to it. I'll discuss this more in the live session, but they operate in the Nanopass style over a mostly monotonically growing (along the field axis) matrix representation of the AST whose core "columns" correspond to the core columns of the Quad-XML format. Rather than a single "attributes" column I flatten the attributes column into multiple main columns, but otherwise its the same, and the "Xml" function in my compiler helps to convert to Quad-XML format and serialize the AST for those who want to store intermediate AST results.

The challenging part, which is part of what the research is one, is doing tree transformations in an efficient manner. Because this is a pointerless representation, you have to be careful in how you design the structure to ensure that you maximize locality and parallelism. If you read my "Key" paper in the publications:

https://github.com/arcfide/Co-dfns#publications

You'll see how I manage one of the most tricky elements. By using the techniques I describe in that paper, I can perform arbitrary computation over any group of sub-trees in the AST selected by their parent-child relationships in a data-parallel fashion. This is largely accomplished by converting that depth vector in the Quad-XML format into a "path matrix" which allows for computing and reasoning about the parent child relationships of any two arbitrary nodes in the AST without reference to any other parts of the tree, without pointers. I can then optimize that path matrix representation for either ease of construction or for performance over certain types of common operations.

That's really one of the most significant elements and sort of blows the whole problem wide open and actually makes it possible to do what I'm doing so easily.

Once I replace the standard recursive transformation idioms with this new set of Key/Path Matrix idioms, I use the Nanopass architecture to allow refactoring. Nanopass is a style of compiler construction that builds on the idea of functional programming. So, yes, the compiler itself is very very functionally oriented, and that's a very big part of the refactorability of the code. But also, since I have done so in a way that results in so few variable names, that's also a major component, and it means that code is often highly "independent" and can be removed or deleted easily.

Sorry! I just realized that I forgot to answer the question about file names. The filenames themselves are a bit of a cultural homage to historical APL development. They are a little bit of a part of my push to stay small, because if I go beyond 26 or so files, I'm in trouble. But it's also a little bit of a "self documenting" element. There's a famous example of the style of C coding that I'm doing here from the author Arthur Whitney, the K developer. He famously whipped up a little J interpreter prototype that was about a page of code and Kenneth Iverson spent some time studying that code to understand its structure and layout and found it interesting. Whitney famously tended to write software in a very ascetical style and just used single letter names for his files.

The use of single letter names in the files here is a bit of an inside joke, referencing back the style of programming of Arthur Whitney, signaling a bit of a historical "stylistic" or artistic connection, while at the same time being the first "alert" to the programmer that they are likely to see something along the lines of Whitney style C code inside of the files. It serves both as a chuckle to the APL community as well as a documentation of how you might want to prepare your mind before reading the code.

Oh, and on another note, I've found that it's mostly programmers and computer scientists who struggle the most with the code. I've tried this style of programming out with high school students with little to no programming background, and they were able to pick it up and use it to do more in 12 hours than most students in an entry-level undergraduate course did in the first half of their semester.

Don't worry about offending me with that comment. I have a pretty strong belief in why I'm coding this way, so I'm glad to have the opportunity to work with people like you who find the code scary and disgusting and see if I can't either change your mind or change the code to be better.

Firstly, about this being a personal project, it's actually a bit more than that. It serves as a research platform for a research agenda around the usability of programming languages and the HCI and pedagogy of computation, yes. However, it's also a commercially funded compiler that is commercially licensed and distributed. The compiler is still in early stages, so it's large a boutique offering at this point, but that's scheduled to change this year or maybe the next. And yes, the development team that works with me on this compiler has read the code, and while they are not as fluent in it as I am, they understand how to work with it and we can talk about the compiler and work through issues in the compiler that comes up.

Indeed, the fact that the compiler is so easy to track through at a macro level has allowed us to avoid needing extra documentation throughout, because when we have a question about some level of architecture design, we can usually pull up a page of the compiler and work through it without needing any other documentation.

And, the point of the post above was that small code is a useful metric for pushing for simplicity. There is a difference between obfuscation and small code, but my code is not obfuscated to those who need to work with it. It is obfuscatory for anyone who expects to read it like a normal program.

At the heart of this is the meaning of readability. You've implicitly defined readability as being a state for code bases that allows anyone else to read and understand the code. That's a high bar. If I write a standard proof of the uncountability of the real numbers, it's a rather high bar to say that everyone should be able to read that proof.

Also, if you look at the way that the Clang codebase is engineered, for instance, if you take any one snippet of 50 lines or so of code, it's all nice, neat, and readable. But when it comes to understanding the entire compiler as a whole, the codebase is completely unreadable. It requires external documentation to understand almost any part of that code at a macro level.

But Clang uses best industry practices and is, on the whole, what most people would consider very cleanly written code. And yet, it is essentially impenetrable from a macro level without other documentation.

Instead, I'd submit that readability is something that we should consider valuable for those who have the relevant pre-requisite understandings of key ideas and concepts when looking at a new code base.

Part of the problem is that this code base is introducing new, research level ideas into the coding space. There is a fundamental difference between it and other compilers in the approach that it is taking, and thus, you can't just look for the same patterns.

I've already touched on malleability elsewhere. SICP has a classic quote about the importance of malleability in code (amoeba vs. pyramid programming).

I'll give a simple description of the architecture. If you understand everything said in the following sentence, then you'll have no trouble understanding how the code is written, and if you don't, then learning these sub-domains of programming skillsets will go a long way in helping to clarify the design.

It's a three part dfns->C++ offline batch compiler overloading the standard Quad-Fix system function in APL interpreters to compile whole, closed namespace scripts built of pure functional dfns on the Dyalog 15.0 primitive vocabulary sans-guards through a PEG parser to a core compiler written in a Nanopass compiler architecture over a linearized Quad-XML style matrix AST representation where each pass is written as a data-flow, data-parallel function train leading to a single dispatch code generator with a runtime library header prepended to each output file containing implementations of each implemented APL primitive.

Those would be the basic set of techniques and skills that are being put to use in the compiler. If you already understand Nanopass, PEG parsers, the Quad-XML tree linearization format, function trains, and so forth, then the structure and format and design of the compiler is obvious and easy to work with after about 5 minutes of orientation. If you don't have that background, then understanding that part of the compiler is rather a difficult one. In addition to this, there are new techniques being used and applied to solve problems in this compiler itself, and those are being documented through the papers that I'm publishing on these techniques:

https://github.com/arcfide/Co-dfns#publications

Most people don't have a strong data-parallel, array style programming background, which makes the micro-level code the hardest part to understand for them. However, if you are experienced in that background, then working with the compiler passes is not difficult, provided that you take the time to understand the core idioms in play.

So, in summary, I'd say that you're right that the code looks horrendous, because your heuristics are designed for code that is completely, almost assuredly, fundamentally different than this code. However, like I said, come to the live session and see me explicate the architecture of the compiler. I'll explain a lot of the ideas I mention in the above sentence enough to allow you to walk through the code easily. If you still think it's scary, okay. I'd appreciate some ways to make it easier to work with it on a day to day basis.

There is a development method called "Direct Development" which has arisen as a term to describe the organic programming model that many profitable APL endeavors have followed. It helps to eliminate the issue of metrics by eliminating the divide between the programming/IT unit and the users of the software. In companies that are using Direct Development, the metric is "are they giving me what I need?" And the way they accomplish this is by pair programming with the users of the software themselves. That's right, the users of the code actually participate directly in the development of the code. They write the code in such a way that the code itself becomes the specification analogue and is in fact the shared knowledge between the coders and the users. Not the comments, but the code itself. The users read the code, work with the programmers, and they make changes on the fly (with appropriate QA).

If your users have the confidence that they can walk up and get any feature they need implemented with you, that's about the best metric of success I can think of.

I can provide even better. If you look at the Publications:

https://github.com/arcfide/Co-dfns#publications

You can read the "Key" paper that walks you through one of the core data structures in the compiler and how it works. If you see the "rn" binding in the compiler code in "e.cd" you'll see that code put into the first compiler pass of the project.

If you want to see more than that, I can email you a copy of an updated version of that paper which is currently under review (not likely to see publishing this round). It includes descriptions of the implementations of the "lf" and "fe" compiler passes.

This is a commercially funded project, there are other people reading and working with the code. There's just rarely a reason for them to commit any changes.

You can go through the Dyalog meetings and see how APL scales up and down along the spectrums.

I'm glad you think my compiler is a small system. The problem I'm solving is one that people said was simply too difficult and impractical to pursue. If I have made it so simple as to be dismissed as trivial, then that's good. :-)

I'm happy to walk you through the compiler in the live session and let you decide for yourself just how maintainable it would be if you had to pick it up. But this code base has been designed with maintainability in mind from the beginning.

How big is a big system? You've called this a small system, but it's a compiler with commercial backing/funding that compiles a language used in production systems, and is, to my knowledge, the only compiler able to express core compilation algorithms in an efficient manner on the GPU. It's rapidly moving to the self-hosting point, and at that point we will have a complete compiler that compiles a real language that runs completely and entirely on the GPU, from parser to generator.

To give you an idea of this task. A basic scan primitive implemented efficiently on the GPU in the neatest and cleanest code that I know of published in the literature is 100 lines of code. If you compressed it, you could probably fit it into 50 - 70 lines of code. That's for one simple operation that takes anyone a single line of C code to write.

This project has taken a real compiler (it's not a C++ compiler, of course) and is putting it on the GPU. Is this a small system?

I would put it in the realm of the sort of problem that can only be meaningfully solved by simplification.

However, this isn't the only code base around. There's another company who has a larger team of APLers who maintain over 1 million lines of APL code in production. At that scale they have to make different design choices than I do, but they also say, if they can do it in APL, they do, and they wish they could do everything in APL. They are one of the only groups, to my knowledge, who has been able to see a net gain in value from implementing a static type system on top of APL's core. So, in terms of scalability, yeah, maybe you need something more (like a static type system) as your code grows, but if you manage to need 1 million lines of APL for your problem, then you're in a good place.

Still, just come to the live session and we can discuss all of the issues that you see with maintainability. If you can see a way to make the code simpler and easier to reason about at a macro level, I'll be all for it!

V for Vector is an appropriate, but not the only, interpretation for that letter.

In the case of this compiler, I take an opposite convention. Most single character names are globally meaningful, and their meaning rarely, if ever changes across the whole compiler. As names get longer, they progressively represent more local elements. This is done in a way that reveals that nesting structure, but is also done because over time I realized that it was harder to remember from one patch to the next what the local variables were meant to do, rather than the global variables, which were almost always the same all the time and were much more likely to be in mental cache. Therefore, I used more "information", that is, more characters, for local names that I would more likely forget the meaning of later, than for global names that were universal and almost always on my mind.

And yes, I did try doing this compiler in many, many other styles, including C, C++, Nanopass Scheme, ML, Java, Cleanroom, traditional APL style, and so on and so forth. They were all unreadably verbose and difficult to work with and very hard to make forward progress on.

Thanks for the comments. I would encourage you to attend or watch the recording of the live session once it is done, as it will give a more thorough answer than I can give here as to why. I've talked a little about the motivations for this in my above longer two comments, but briefly here:

1. It matters how much code you can see and work with at a time.

2. This code is using research level new algorithms for solving certain problems in the core compiler that are not a part of the standard toolbox of most programmers, and so, even if you did have the code laid out differently, it wouldn't necessarily be any easier. An early version of one of these new algorithms has been published and is listed in the set of publications:

https://github.com/arcfide/Co-dfns#publications

3. As mentioned above, keeping this code at the same semantic density as the core compiler allows you to read this code in much the same way that you can read the compiler code, making it easier to jump back and forth whenever you want throughout any point in the compiler and not adjust to multiple languages. This applies to the use of single characters.

4. The letters v, z, and s are all semantically meaningful to those working in APL, but they are also globally standardized "for the most part" throughout the compiler so that you know what we are talking about whenever the letter "z" shows up anywhere in the compiler. There are some exceptions, but these are obvious, and local, and can be that way because the use site and definition site are within a couple of lines of each other. So, yes, it's quickly obvious from the context.

5. I have tried many, many times to introduce comments. They almost never help, and almost always hurt.

6. If I were to change the style, it might be easier to understand what a given snippet is doing, but this wouldn't aid in your ability to understand what the whole compiler is doing or how it all fits together. It would be a false sense of understanding because it would be divorced from the broader context. The point of the design the way it is is to encourage you to focus on the macro level design issues, and allow you to see the whole context of the whole compiler more easily.

The live session would be the first start, obviously, but you can also see the Publications area of the README:

https://github.com/arcfide/Co-dfns#publications

Some of that deals with the micro and some with the macro level ideas, but there are some key elements in those that will be necessary to appreciate the whole thing.

It's something I've been working on for a while, but because the architecture is under constant flex, it's actually more valuable to be able to know how to "experience" or discover the architecture in the compiler code itself than to have a separate document to follow, since it's very easy for that document to get out of date quickly. I am building up a set of documents that discuss some of the core idioms and ideas though, and I hope to have something come of this live session that I can maybe put into an interactive document that people can work with.

Done.

Indeed, many of these ideas as expressed in APL come from the 1962 book, "A Programming Language."

I find it unfortunate that these ideas are only now beginning to find some general acceptance in larger, more mainstream languages.

Understandable. However, it won't be necessary to know APL to understand what I'm going to talk about. I might delve into a bit of APL, but only from the perspective of how to work with the code. My hope is that the more general lessons can be taken and used anywhere, and improve the code of projects elsewhere.

Respectfully, the APL of today is not the APL of the 80's. APL is tremendously well suited to many of the problems that face other programming languages, such as the parallel programming usability problem. Modern dfns based APL has incorporated lessons learned from languages liked Scheme (Lisp) to make function syntax and control flow more functional.

And APL has been and continues to be used to solve problems from web programming to graph processing and everything in between. While it finds immediately obvious advantage in numerically dense calculation heavy code, my research is specifically interested in demonstrating and scaling APL beyond the "status quo" of what people think APL is good for. The Co-dfns compiler mentioned in these threads is precisely such a counterpoint, demonstrating how to leverage APL in a domain space traditionally dominated by OOP and Lisp style languages, and more recently by certainly highly typed languages such as Agda or maybe Haskell.

The difference is that good APL is often more or less "parallel ready" out of the box. Good code in other languages is often anything but. I am working through making the case that "parallel first" languages such as APL are actually pedagogically and programmatically better than "serial first" languages. To this end I have conducted at least one exploratory pilot study on the matter:

http://video.dyalog.com/Dyalog13/?v=kIItfQJEVdM

It is very common for large APL projects to require only a single person to maintain most of the source for a large portion of its life. While this doesn't help the bus factor much, it's not the red-flag it would be in other languages.