The Subtext Programming Language
github.com
github.com
- Spreadsheets don't separate code from data, which is an important aspect of programming. Blending the two is a nice shortcut for spreadsheets, but it ruins maintainability.
- Naming is important for programming, but spreadsheets encourage users to not bother with names and fall back to cell coordinates like "A10" instead. It's a great shortcut, but again, it ruins maintainability.
- Spreadsheet formulas operate on scalar values, while programming languages operate on nested data structures. Data structures are a key to managing complexity, yet the spreadsheet paradigm has only a very limited view of them.
If users are ever going to be able to use a spreadsheet-like interface for serious programming, most of those issues will need to be addressed in a way that doesn't hurt usability. That's a tall order! Users already struggle with function calls in formulas. At some point it's smarter for users to just pay a programmer to figure out how to build it.
There is no gulf. Spreadsheets just are programming languages. You've certainly identified problems or at least quirks of spreadsheets as a 'language', but I think people do a huge disservice to their own ability to understand programming as a field when they try to carve out exclusions for languages and environments on the basis of who uses them. Lots of things no one would ever argue aren't a programming language have all these flaws and more.
When people make a big ass complicated spreadsheet, they are programming. They are doing so whether they're a seasoned professional or an accountant or a lawyer or a homemaker.
What develops typically don't get is that the paradigm of the programming language and environment matters a lot, triple so for people without formal training in coding. Most people who can code with a spreadsheet would be totally unable to create a very simple shell script, because the paradigm of imperative, syntax-led classic programming languages require building a mental model of a whole Turing machine in your head, which requires years of training. That's the gulf the GP was referring to; not from which people happen to use them, but from how they think about coding.
The functional-reactive paradigm (and spreadsheets in particular) avoid that need, because they're highly composable so you can truly think of each part of the program in isolation, something which side-effects and program flow in imperative programs won't allow.
I'm calling out an implicit bias that always comes up in discussions like these and I see lurking behind this sentiment, because these two things are inextricably related, not an accidental correlation.
They think better in this paradigm because its a better suited paradigm to their actual work, and it fits better with things they understand already because of their experiences and knowledge.
When we prioritize the "thought patterns" of professional programmers we forget this, and then we lament the lack of understanding for our field from people outside of it. Our solutions, when we look at it this way (with a stark divide between "programmers" and "non-programmers"), end up demanding that they come to our understanding instead of us to theirs. And that's basically always doomed to fail.
This is important.
I concur, that's a good way to put it. The "way of the programmer" requires an unnatural way of reasoning that takes a lot of effort to master, and requires a lot of short-term memory effort to mentally simulate how the code behaves.
I don't even think this programming style is truly required for professional programmers either; more a by-product of how programming started as a way to feed instructions to mono-processors in the context of mathematics and physics computations on limited hardware.
Symbolic business logic doesn't typically depend so strongly on the specifics of runtime execution, and recently there has been a very active movement to build new programming environments which emphasize the relationship between code and data, rather than between code and intermediate states of computation as in classical programming languages.
- Spreadsheets don't separate code from data, and that's the single most important reason why they're easy to use and allow non-developers to build automations. They work like an always-on debugger, so normal users can work like a programmer works when exploring and/or fixing a program's behaviour. Practically no other coding tool for non-developers allow this.
- Naming is important for code reuse, not necessarily for one-shot functions; positional reasoning can work equally well there. Spreadsheets do allow naming cells and accessing them by name when needed; but the option not to, which classic PLs don't allow, lowers the entry barrier. Programmers could also name all their parameters 'a', 'b', 'c'; that they learn to use better names is a matter of discipline, not a limitation of the tool.
- Scalar vs ranges: this one is actually the most limiting part of spreadsheets. They are great for tabular data structures (doh!), but nested structures are tricky. There are research projects to improve this (Microsoft hosts a lot of them on this line, specially around ML and Programming By Example), but in the end you need to overcome the paradigm and apply spreadsheet-like programming concepts to tools allowing hierarchical, nested structures. That lies the future of computation for the masses.
- You can give things names after you create them instead of before, and it's not clear that argument names are necessarily either necessary or sufficient for code reuse. Subtext makes it easy to change argument names whenever you want.
- It's reasonable to describe Subtext as a programming-by-example environment, and it does support nesting—in fact, it's rather pervasively nested. An alternative approach is to solve your problems without hierarchical, nested structures. SQL and co-dfns are two interesting directions there. I wrote an overview of this kind of thing five years ago: http://canonical.org/~kragen/memory-models/
As I said elsewhere, it's a pity that its chosen license works directly against widespread adoption. I may take a look at how it's done to gather some implementation ideas, but I would never depend on it for any project that have a chance of growth.
Computers were alien, and the culture projected all its fears onto them: rightists churned out endless movies about slave revolts with computers and robots in the role of slaves, while leftists tried to bomb and burn down computer centers as oppressive instruments of the System.
FORTRAN and COBOL were alien to experienced programmers, who seriously doubted how optimal the output assembly would be.
VisiCalc was alien to Fortran programmers. You're still pushing back against the remnants of that rejection today.
MacOS in 01984 was alien to those of us who'd grown up writing BASIC. It provoked hostility, too!
Pascal was alien to me because it didn't have line numbers.
Online chatting was alien. We called ourselves "geeks".
I still have anti-JS JS from 02000 on http://canonical.org/~kragen/hotlist.html because JS was alien and I hated it.
When we pioneered AJAX and Comet later that year at KnowNow (one of three independent inventions), it was alien, and in fact it took four years before Gmail took AJAX mainstream; Comet took even longer.
So don't worry about being alien. You're shooting for a revolutionary change, not an incremental improvement. Revolutionary changes are difficult under the best of circumstances, but they are impossible to achieve without being alien. Your weirdness budget isn't unlimited but it's necessarily larger than for an incremental change.
I know it would be a headache to achieve it, but it would be pretty useful to have an open-source runnable version of any of the older Subtext prototypes.
I have some ideas on my own on building a new style of computing environment that doesn't feel alien, and I've arrived to a lot of the same conclusions that you describe in the "Notable features" section.
My approach is to focus the environment in content-editing first (i.e. making it feel like a "structured content editor" more than a IDE), and representing code primarily with a metaphor of content transformation over collections (think Google's OpenRefine or MS Power BI), rather than as processes running on an application.
I have a core idea at the base of the structure for this system, which is to reify function applications as a named object in the environment, much like your idea of treating function results as collections of values accessible as raw data. This reified object, which I call a wit for memorability, acts very much like items in your Subtext, except that:
* Each wit has a unique identifier (a 'label') which can be used to annotate data directly through direct manipulation, like giving a name to a cell in a spreadsheet. The user can build collections by applying the same tag to all the values in the collection, manually or supported by a PbE process. The wit represents the data structure containing all those values; but at the same time it may represent the process used to generate them.
* There is no distinction among scalars and collections. I think this is the most unique feature of my system; you may apply a function to a wit (identified by its label), and it gets lazily applied to all its values, thus creating a new collection of transformed values. This is similar to named ranges in a spreadsheet, but it doesn't depend on a positional grid. And single values are treated the same as collections, so there's no need for loop constructs; the user can simply build templates with wits as variables, which then get applied to all their possible values in their input domain. (Microsoft has a similar feature with their research on FlashFill, which was posted here a few days ago[1]).
* Wits may be used as identifiers of attributes, to extract content from records. That content may be previously stored values or computed on the fly from their formula and current inputs (just like in your "Subtext is both a PL and a DB" line). I.e. wits can represent relations among entities, not just code applied over data. This blurs the metaphors of function application and data structure, which I believe is a good thing for easy-to-use computational environments.
* Since code is seen as data collections, you can build repeatable behaviors by changing the input on a sequence of transformations. For example, recorded recent user actions are not seen as a sequence of code steps, but a sequence of intermediate processed states of the transformed collection (with each step being itself represented by a wit, and thus accessible as data). I think you have a similar idea with your "functions seen as expanding to traces", but I envision using the intermediate steps as possible inputs to new functions (facilitating creation of pipes and recursion by demonstration).
I've felt my ideas validated when seeing that you've created a system which has evolved to a very similar design space; your Feedback mechanism is a neat solution to the representation of interaction and user updates, for which I didn't have a concrete solution (other than the abstract idea of representing all state as updates on functional-reactive streams).
Unfortunately my design process involves primarily paper prototypes and lots of concept notes; and also I don't get paid for it, so there's very little in a shareable form. Do you have a public discussion forum where these ideas can be discussed in the open? It would be good to share my detailed concepts and see how they apply to concrete use cases that you may have in mind for building your system.
[1] https://news.ycombinator.com/item?id=28548165 "Story of the Flash Fill Feature in Excel"
I took inspiration from this [1] classic article by Henry Lieberman, where tags are applied to elements of a composite as a way to edit its structure; and I combined it with the idea of a templating system to process or generate items in bulk (all templating systems do this, my idea is to abstract it into a generalized, always present interaction mechanism).
[1] http://acypher.com/wwid/Chapters/23ProblemSolving.html Making Programming Accessible to Visual Problem Solvers
I liked that book so much I bought it in paper form; also YWIMC. Lieberman's work in particular has always been inspirational. I think he sits down the hall from Jonathan, but I could be misremembering.
This morning I was working on a problem that I was thinking would be ideal for a system like this: I was taking notes in prose about potential blowing agents for waterglass or other geopolymer foams—things that produce gas when the polymer is heated to its softening point. For example, chalk, ammonium chloride, sulfur, aluminum hydroxide, sal mirabilis, saltpeter, or carbon plus silica. Each of these blowing agents produces a certain number of moles of gas per mole of blowing agent, has a certain molar mass, a certain density, a certain cost, and a certain activation temperature, plus certain material-compatibility concerns—you can't disperse hydrated sal mirabilis in an aqueous solution, for example, or chalk in an acid solution, if you want them to be intact in the unblown geopolymer preform. These are "givens" that can be found in a literature search or, in some cases, a search on MercadoLibre.
In the prose document, I would like to be able to tag the gas production per mole, molar mass, density, activation temperature, and material compatibility in such a way that I can render a table of the candidate materials in the document itself, and then sort it by gas production per mass, per volume, or per dollar, and filter it by activation temperature. Gas production per mass is calculated from molar mass and molar gas production; gas production per volume is calculated from gas production per mass and density, and similarly for gas production per dollar.
Is that the kind of thing you're thinking of, or am I imagining wits very differently from what you really have in mind? I look forward to reading your notes! (I'm not in the Slack. I hate Slack. It's ransomware.)
A step beyond would be to type the reaction equations into the prose document and calculate the gas production per mole from that, and use a database of atomic weights to calculate the molar mass from that as well. With a database of the formation entropies and enthalpies of the compounds, I could not only calculate how endothermic the decomposition is (or, rarely, exothermic) but plot an Ellingham diagram as well, which would be pretty useful as a double-check on the sanity of the reported activation temperatures from the littrachaw. Together with combinatorial search functionality, it might also enable me to anticipate problematic side reactions, or how the activation temperatures would shift if the foaming was carried out at 1 MPa.
Some of the quantities I'm calculating on have units attached, such as temperature and density, and some have intervals of uncertainty, such as price, and it would be useful to have the computer track those things instead of the human. Confusing a Gibbs free energy in kJ/mol with kJ/kg can be a problem; similarly gas production per gram or per cubic centimeter.
Some of Jonathan's work on Subtext has focused on variants of or alternatives to test-first programming; in spreadsheets people sometimes do test-first programming (for things like the equilibrium-temperature calculation I mentioned above, for example) and then delete the tests. In a context like the blowing-agent survey, being able to hide them instead would be better.
Finally, ideally, I'd like to be able to transclude at least the givens and possibly the calculation results in other notes: "According to my notes in [0], aluminum hydroxide is stable at atmospheric pressure up to 220°, ..." perhaps eventually refactoring those inputs into a centralized database so that I don't have to remember which note has the boiling point of sulfur in it.
The calculation part would be easy to do in Gnumeric or localc, but those handle the prose, units, and intervals very poorly at best; and they would be hopeless for the equation parsing part and, at best, clumsy for the database queries and Ellingham-diagram plotting.
Yes, that seems be a very direct application of my wit concept, with one wit for each magnitude, placing each wit as a column of the table and having one row for each material. In the end, a wit is basically like a user-friendly version of a typed pointer in C, with a 'visual' GUI for combining them like an Excel pivot table or a OLAP 'data cube'. You can read the value at the end, or you can iterate it over a collection to process a list of values; the meaning is located in the code that uses it, more than in the object itself, and you can share the reference to the same value in many different places.
Of course there are already several systems where you can define a data structure in code and show it rendered as labels on a document (see idyll[1], or the original Explorable Explanations), but I have the idea that using such system shouldn't require any code for basic or moderately-complex programs:
1) the user should be able to define the data structure through direct interaction, like on a spreadsheet (e.g. Airtable does this fairly well). But the raw data could be extracted from existing documents, like a web scrapper, with the scrapping expressions defined through point-and-click over the actual data. The now extict kimonolabs[2] had a good interaction model for this part. **
2) this structured data could be transferred between applications with a copy/paste-like universal protocol (most current applications require coding against APIs for this),
3) calculations can be defined and re-applied to new data collections through simple formulas and more direct manipulation, possibly with Programming-by-example assistance tools. The people from MS Excel are doing this, but they lack the universal API - everything happens inside Microsoft applications.
For your example, you could store your database of atomic weights as "hidden wits" somewhere else, extract the desired value by naming the label (wit labels are global variables under a particular namespace), and feed its values to the expression where you want to calculate a function application with the data.
> The calculation part would be easy to do in Gnumeric or localc, but those handle the prose, units, and intervals very poorly at best; and they would be hopeless for the equation parsing part and, at best, clumsy for the database queries and Ellingham-diagram plotting.
The good thing of wits is that you could chain them and transfer information between applications, with each app handling what it does well, and returning the data back to the origin; i.e. any workflow that you could do manually with copy-paste. In this sense it's like a Unix pipe, but in a visual environment and transferring structured data (which can be de-composed at the receiving end, extracting one wit at a time). They are also like those awkward connector lines in dataflow visual languages, except that instead of a line you have a label with the wit name at the origin and destination.
Again MS has Power Automate as a tool with these capabilities (I think its way more complex and powerful than the classic Apple Automator), but again Power Automate restricts you to a closed runtime environment of applications that support it, which you glue only through code.
> Is that the kind of thing you're thinking of, or am I imagining wits very differently from what you really have in mind? I look forward to reading your notes! (I'm not in the Slack. I hate Slack. It's ransomware.)
I don't like walled gardens either; I've heard Matrix.org and Element.io are lovely this time of the year... :-P
I've bought the domain http://witclay.com, though I never managed to clean up my notes and publish them over there. If I manage to gather an audience in that group, I will probably summon the will to publish my use cases over there so that I can show them in the few next months, providing some concrete examples that are more intelligible than just abstract ideas.
[1] https://idyll-lang.org/docs
[2] https://youtu.be/RS3ckeDIOmE?t=157 Kimono on tables
https://youtu.be/jQ_hSNj1gI0?t=49 Kimono on structured text
** The wits here would be the yellow bubbles with the extracted data. Again, many of my insights I've seen realized in other systems; my idea is an orchestration of all them into a single user-centric system that made all them compatible, at the user fingertips.
With the example interactions in that website you can get very close to what I imagined my system to be (though I would use a "select+tag" metaphor more often than the cumbersome drag&drop).
I still have the idea to allow spreadsheet-like processing on top of those metaphors, but the essence of the system is quite similar to that one; many information systems are converging in that same direction toward a similar model of interaction.
[1] https://alexanderobenauer.com/labnotes/000/
Things like
// passing values through check
x do {
check >? 0
check even?()
+ 1
}
"It is an interactive medium for data processing by people who don’t want to learn how to program. For example, scientists who want to process data without becoming “data scientists”. Or people who want to build simple custom apps without learning to program."
[This fighting text with text is indeed a CSFi soap opera :)]
But the inspirations differ https://inflex.io/blog/bibliography
What if you add formulas?
I have no idea if Subtext 10 has legs. It kinda reminds me of Fran, so that's a good start (direction).
All I'm saying is that if I was gonna tackle this problem, I'd start with turning Excel files into into source code.
What I've been hammock thinking is a way to bridge the gap between spreadsheets and what we tend to consider code. Why can't we iterate spreadsheets to something that makes it easier to: diff, review, separate logic from data, track in version control, write new functions/macros, re-use, package and distribute.
I think as more people will need to use computers more, that everyone will need to have access to some form of programming skills. I think spreadsheets are a successful mechanism for people to rapidly manipulate data, but they deteriorate quickly with complexity. A tool that doesn't throw out the baby with the bathwater could help people ramp up to programming much easier. If you teach someone to write a function that operates on a medium they already know like a spreadsheet, it is a lot less intimidating than writing a .py file, and downloading python and learning what compile is.
Spreadsheet-like IDEs are the most logical direction to meet that vision, specially when you want to attract non-programmers; figuring out the language syntax and the runtime execution model are the hardest parts of learning to program, and spreadsheets minimize both.
I put my bet on code notebooks, of the functional reactive variety. They have the same interaction model of the spreadsheet, intermixing data and code in the same environment, and they allow you to grow a program iteratively with cells acting as reusable objects - a simpler metaphor than OOP, but equally powerful. Data scientists are already making good use of them, and it's easy to think that these notebooks will be getting more and more of the debugging and DevOps functions of traditional IDEs.
> I hope not. Open source works best for software used by other programmers, because that is how the contributors get their cred. But the economics of startups conflict with the goal of liberating software development. Perhaps a hybrid form like a Public Benefit Corp or a Coop offers a workable middle way.
... Huh? The question is about open source. The answer seems to be about startups. Was this an editing mixup?
One thing I don't understand is the dismissal of operator precedence for infix operators. It seems to me that creates an unnecessary impedance mismatch for getting formulae entered into the computer.
Thanks for sharing.
As with spreadsheets, the absolute or relative location can fill in as a address. Perhaps there should be some relative addressing for code, as you hinted ^ with spaces around it is the previous result.
to me this is really the desirable core idea
https://github.com/JonathanMEdwards/subtext10/blob/master/LI...
I believe the lack of a patents grant clause is the only worrisome thing about CC licenses?
It would need to be a success to warrant alternative implementations, and the license is inadequate for a language to get widespread adoption.