Ruins of forgotten empires: APL languages
scottlocklin.wordpress.com
scottlocklin.wordpress.com
Joe Celko coined a phrase for this: think in sets. Not in loops, not in linear instructions -- but in sets, being transformed into other sets. It's a very powerful way of thinking. Most of the time, you don't need ordered evaluation and so loops are imposing unnecessary guarantees on you.
On the more general problem of our collective historical ignorance ... yes. That is us.
Well not entirely. That decades of the relevant literature is locked up in Fort ACM and IEEE Castle has a lot to do with it.
People who subscribe to the Relational Database dogma often claim that sets are superior to vectors, and yet despite asking a lot of them I never received an example -- whereas I have numerous real life examples in which vectors are superior.
So if it isn't necessary, why pay the cost?
Moreover, the order you want will change from case to case. Today I care about the alphabetical order of names, tomorrow about the order of dates, next week I am most interested in the top 100 transactions by value.
Imposing a single order imposes an access pattern which may or may not hold true. With sets I delay that binding until the last possible moment, when I understand it best of all.
> So if it isn't necessary, why pay the cost?
But if you use the APL/K approach, you don't pay the cost. There is in order to every result, but you can only rely on it if you know it (or force it, by some sort).
Specifically, my question is this:
The order of results of "select ... where" without an order-by clause in SQL is never guaranteed. The order of a ksql/q result is sometimes guaranteed, and can be used. Where's the cost of just giving every result row a running index, which some operations can rely on / change, and most ignore?
> Where's the cost of just giving every result row a running index, which some operations can rely on / change, and most ignore?
The cost is that you need to keep an index updated. A trite observation, but there it is. Sometimes that cost is worth paying, sometimes it isn't. SQL doesn't forcefully extract that cost, it lets the database programmer or administrator decide, based on local conditions, what is best.
Yes, I understand the theoretical argument. I'm looking for real world examples where that actually helps anything, or theoretical examples where it is superior (in the sense that "there is no way to assign order which would not move the problem to a higher complexity class").
> The cost is that you need to keep an index updated. A trite observation, but there it is. Sometimes that cost is worth paying, sometimes it isn't. SQL doesn't forcefully extract that cost, it lets the database programmer or administrator decide, based on local conditions, what is best.
My point is that ksql does the same -- except that in most cases, the cost is 0, so you get the order "for free"; when the cost is not zero, ksql doesn't make guarantees, and you'll need to ask for ordering, just like in sql. Every sql engine already assigns order in one way or another (ROWID, etc.), except it is unpredictable, non-standardized, and doesn't provide most of the benefits even though it is there.
And the "no order" default often does have a cost; e.g., if you are trying to store e.g. tax brackets in your database and find out which tax bracket a salary falls into:
Either you store both lower and upper bound of bracket in a record, which means you can't normalize (and it is hard to express the integrity requirement of "no gaps" as a constraint) - or you do normalize, and then every query is a self join or correlated query that many optimizers won't handle properly. Whereas in ksql, you store it normalized (just start-of-bracket), order by start-of-bracket, and then get the bin that corresponds to your question.
The main difference is that a relation is not a primitive. It is a tuple of primitives.
The point being that sometimes you care about those relations being ordered by date, sometimes by amount, sometimes by ID and so on and so forth. The database imposes no order on you and so you can impose one at will.
The database equivalent of maintaining order for every single-column selection is to have per-column indices.
Try that in production on a busy OLTP system and let me know how you go.
Where array/vector languages and SQL see eye-to-eye is in basically abolishing the loop and having a logical model where operations are applied simultaneously to whole collections.
From talking to you the difference is when order is imposed. In cases of time series analysis, where there are single numbers and a natural order, I see the value of array/vector languages (or one of the various proprietary temporal-SQL databases). But most problems of interest aren't vectors with a single natural order.
Incidentally, I don't see why you think the tax-bracket thing is such a killer case. Oh noes, a small join!
I guess that's where the misunderstanding comes. In K/APL, every column has an implicit "natural index" - the first item has index 0, the second item has index 1, etc. The thing that is always guaranteed is that, in a result set, the natural indices of the different columns of the same tuple correspond.
Some operations guarantee a particular natural index order, but many do not. An additional index on the values that allows you to find an exact value (e.g. hash) or range of values (e.g. a b-tree) may or may not exist, just like in a relational database.
This "natural implicit index" does not take memory or anything to maintain, but it is useful.
> Try that in production on a busy OLTP system and let me know how you go.
ksql runs my queries approximately 100 times faster than optimized MySQL (using MyISAM - faster, but no acid guarantees; ksql is acid).
> Incidentally, I don't see why you think the tax-bracket thing is such a killer case. Oh noes, a small join!
That was just for known terminology, not knowing where you come from. I have tables with thousands of such brackets. And while I haven't tried Oracle or MS-SQL for them since 2006 or so, it was incredibly easy to confuse their optimizer into doing a complete cross product on those tables and filter it - which makes it takes a second per query instead of milliseconds when the query is optimized properly (or microseconds using K).
This is known as an "as-of" join in time-series queries, and any database without explicit support for it usually takes thousands of times more resources than needed for such a join.
I'm against "complex nested data structures", though. In my experience they are abused and there is always a normalised alternative.
But you make a great point about ACM and IEEE. As a student (and now an engineer in industry) in the CS and applied math fields, I can't count how many times during research where I see a useful article behind an IEEE/ACM paywall.
Are you aware of any efforts/campaigns to demand more open access from these two behemoths, who pretend to be on our side?
>As a student (and now an engineer in industry) in the CS and applied math fields, I can't count how many times during research where I see a useful article behind an IEEE/ACM paywall.
I have never heard of a college that doesn't pay for this access for their students...
>Are you aware of any efforts/campaigns to demand more open access from these two behemoths, who pretend to be on our side?
If you are a computer science related professional you should be an ACM or IEEE member and thus have access to their digital libraries. If you want to have access to guild resources, join the guild.
The vast majority of educational institutions are not in the US and do not have anywhere near the funding to pay rent to the ACM/IEEE. Every dollar they spend on something like pay-wall access would be better spent investing in their students.
> If you want to have access to guild resources, join the guild.
Guilds are cartels. That's why we don't have them anymore except in the more neutered form of trade unions.
But also, the content ACM et all charge for is not their original contribution. Other people write the papers and give them to the ACM for free. Other people review the papers for the ACM for free. The ACM takes in about 60mm each year but only spends 40mm on it's various activities.
At one time something like the ACM was necessary purely for pragmatic reasons. But now we have the Web, and there is no reason someone should be earning a 20mm profit for artificially restricting the flow of knowledge in the discipline. By comparison the total budget of arXive is around 400k. The ACM could make the digital library free not make a dent in funding for any of the actually useful stuff they do.
It's also unconscionable that the ACM attempts to police 3rd parties linking to copies of papers published on author's homepages. This issue specifically is why I canceled my membership (after writing a letter of complaint that received the most self-aggrandizing response you can imagine)
I'm more charitable to the IEEE as they do a lot of standards work, which does benefit from having a centralized bureaucratic organization.
• The self-taught of all ages who may have never set foot in a university, or have begun informally studying an unrelated discipline on their own after leaving formal education. That they exist and can make use of academic research may sound far-fetched for people in some disciplines, but it's not at all unheard of, at the very least, for CS, statistics, and mathematics.
• Precocious youth
• As jasonwatkinspdx mentioned, students that do not live in the US
• That restricting access to these materials takes a hidden toll on humanity; unless they are made freely available to all, we will never know how others might have taken advantage of them.
First, you have to be a member of both, as the literature is divided between them.
Second, library access isn't included in the membership. It's an extra.
Third, the IEEE in particular partition the hell out of their database and you often wind up paying anyhow ("oh sorry, that's not in the IEEE-CS library, now f-you-pay-me").
When I taught people to play frisbee, I used to throw with my off-hand when teaching, because it made me pay attention to the fundamentals I was trying to teach. Learning J had some of that same feel. I felt like a newbie, but when I saw the kinds of audacious programs you could write, I was hooked. Highly recommended for a bit of fun in your spare time.
Anyways, making cellular automata is fun as hell! :P
That class was fourth in the undergrad sequence, coming after C, C++ and Java so it was quite a shock to, well, everyone. He took us from finding the average of a list all the way through to implementing a CMS in J (our department web server still runs on J because he's since retired and no one else knows how to maintain it).
This was easily the hardest class in the entire department. Students who sailed through everything else would struggle to pass. It's a fascinating language but not an easy one to grasp.
Hadoop is pretty crap - I know, I used to use it professionally. You know what else is crap? Unix. People are going on about cgroups as an exciting new feature, when better implementations were around what, thirty years ago in Real OSes.
It doesn't matter. Commodity wins, and sooner or later if there are good ideas to be had we'll find them again.
It is not about commodity. cgroups-like functionality was available in many opensource OSes for years (e.g. FreeBSD jails), but was easily dismissed by majority as worthless. This is exactly what the author was talking about referring to Lady Gaga -- the modern programming community is driven by fashion regardless of how good or bad the current fashionable technology is. I do not find it surprising however, as it is a direct result of the order of magnitude growth of the programmer's population, which is behaviorally much more close to the general population currently that it used to be.
Compared to what? Beethoven? Charles Mingus? That she can produce some shitty pop tune to be forgotten in a few months after release doesn't indicate much.
Source?
If you know of any others, please let me know. I certainly haven't found anything.
(Though note that the most recent version of K, K4, is a much cleaner and simpler language - I'm not aware of an open source version of that)
When you say "Unix" did you mean to say "Linux"?
"Real OSes"
Can you name an example or two? I'm aware of some old non-UNIX OSes that are still around but which over time have become largely ignored. I'm curious the "Real OSes" you were thinking of.
"How many can play around with a language like that at home?"
As many as have the desire and time to do so. Learners are limited to 32bit and no long running jobs in excess of 1 hour, but this does not stop anyone from learning the language. There's also an open source alternative of the intepreter based on an earlier version of the language.
For examples languages like Fortress contain primitives for parellelized loops, and most modern languages also contain enough functional programming that you rarely have to write for-loops.
Function composition and interactivity aren't a unique feature of APL languages either.
There are some other things that are important for programming today: Maintainability, richness of available libraries/modules, packaging, versioning and so on.
Where would I even look for existing APL libraries? My searches for a HTTP library for APL (for example) have turned up nothing, while searching for a HTTP library for several other languages that came to my mind brought up useful results.
Unique features of J (I'm writing about J because I know it, while I don't know APL) are that all the basic verbs (functions) can operate on arrays as easily as on atoms. There is no special syntax or feature for adding two matrices, for example, standard `+` verb does it naturally.
The second unique feature is a tacit - point-free - style by default. This means that function composition is the default, not function application. You can define new words by just slapping old ones together, without worrying about threading params through them.
The third unique feature (and probably unique to J, previous are found in APL too) are complex rules of composition (hooks and forks) as well as words that modify functions. You can express very complex things in very little space thanks to this.
J is very interesting, unique language which are very well suited to one thing - very fast and efficient array transformations - and completely useless for other things. I hear it's used in a financial and banking sectors, for rapidly prototyping and sometimes deploying statistical and mathematical algorithms. If you have a good enough use case, then learning J will pay off. For anything less than full time commitment, I think that R od Python+Pandas is better. Still, I'm very happy I tried to learn J, as it's completely unlike anything else and it helped me greatly to understand numpy/pandas for my main job.
To stretch his metaphor.
Let's take an example from RosettaCode. For instance, merge sort:
merge =: ,`(({.@] , ($: }.))~` ({.@] , ($: }.)) @.(>&{.))@.(*@*&#)
split =: </.~ 0 1$~#
mergeSort =: merge & $: &>/ @ split ` ] @. (1>:#)
C looks like that to most non-programmers. We forgot how much we have acclimated to already.
Is 中 a word? Or 中國 a word? Or 中國哲 is a word? One can't say until they read a full sentence, making sense of it and distinguishing words from context.
Similarly, can you clearly distinguish operators here ",`(({.@] , ($: }.))~` ({.@] , ($: }.)) @.(>&{.))@.(@&#)"?
Sure, we can acclimate to a lot. Assembler is also a language. But nobody gonna write Hacker News in it.
kona: https://github.com/kevinlawler/kona A free implementation of a k3.2-like language.
There are equally strange patterns that you are probably used to. For example, I see stuff like
PORTB &=~(1<<3);
1) It is extremely terse. A small block of code will do what pages of boilerplate class definitions, interfaces, setters/getters, declarations, typedefs etc. will achieve. There is usually no need to navigate through a spaghetti chain of virtual function calls.
2) It is interpreted. I can quickly (without compiling) cut out chunks of the code and run them to understand what they are doing. I can create variables on the fly, fill them with test data, and run the function. If it calls another function, I execute the name of the function and it returns the function code, since functions themselves are first class variables.
3) It actually becomes easy to read because it is easy to parse, for both machine and human. K/Q is read right to left, and so breaks some traditional conventions. For example, 5*6+3 will evaluate to 45, not 33. At first this can be disconcerting, but eventually you will realise that parsing a line of code no matter how complex it looks is quite simple - you mentally start at the rightmost token and start gobbling. This is probably why the interpretor is so fast as well.
As someone mentioned below, I'm not sure why you'd consider it "too cryptic". Anything is cryptic when you first encounter it. I suppose the difference is that Q/KDB provides an (extremely valuable) solution to the financial world - the handling of big data long before it was even called that (a tick database with billions or trillions of rows) so there is a valid motivation to learn the language and it doesn't feel cryptic at all once you've immersed yourself in it.
Python wasn't cryptic, but it is confusing unless you have a programming background.
the fact that "and" and "or" short circuit.
The fact that "1/2 is 0" (at least before Python 3), and that "1000+1000 is not 2000", even though "1000+1000 == 2000"
Or that "0.3*3 <> 0.9" (if you knew that <> means not equal)
Dijkstra, I think, viewed mere languages as an unhappy by-product of computer science and hankered to do it all with mathematics, actual utility be damned.
More closely, the Rogers model of diffusion[2] has 5 factors of diffusion (relative advantage, compatibility, complexity/simplicity, trialability and observability).
Generally even the most advantageous language transitions score well only in relative advantage. The other diffusion predictors are missing or weak.
In general, he was good at the math part of computer science, terrible at the computer part.
How the fuck is that "wonderful"? The comment was something along the lines of "I wish people wouldn't use that kind of language", and the response was in the form "If I insult you enough, I don't have to acknowledge your argument". Looking at the other comments, the one nitpicking about the choice of Lady Gaga is equally as useless, but strangely that one didn't get an angry reply.
The fact that this is even "a thing" with 30-odd comments about it is a sad commentary on the illness which afflicts modern civilization. If this isn't clear: I don't like people with such "refined" sensibilities, and they are invited to never, ever read my blog. If you're that sensitive, you belong by yourself in a yurt, where nobody's language or opinions will bother you.
Personally, I've never met an actual programmer of any race, gender or creed who actually is that sensitive. People that sensitive are mentally ill, and don't have jobs. I've met plenty of power-mad Cotton Mather sickos who think they have the right to police the words and opinions of others though. The fact that people think witch hunts over a fscking word are OK; that is not OK.
As for people saying "I wish people wouldn't use that kind of language" -- those are the scum of the earth.
It's okay to disagree--I didn't find the "men use Apply(), not for" thing to be particularly offensive myself, honestly--but there are many ways to disagree without resorting to the kind of hateful, caustic language the author used. This is not how well-adjusted adults disagree when having a discussion.
It never got the chance to be a discussion, as the author responded to the opening volley with a shotgun blast.
That's his prerogative, of course, as it is his blog, and he can take his ball and go home if he wants to.
The cause of women in computer science (which as student of a CS department, I wholeheartedly support) is not helped by people derailing everything in the manner in which the commentator did.
It's not a derail. It is a topic which came up because of language the author used, and which is an ongoing problem in the community of software developers. While you are free to believe in a platonic ideal of "pure ideas and discussion," it doesn't exist in this messy world of relationships and human language and politics. If you are still a student, I suggest you take a gender/sexual/racial politics course and try befriending some people with very much opposite viewpoints to your own. You may be surprised at your shift in thinking.
But leaving that aside for the moment, let's say the author's main intent was to "nip that derailing in the bud." What do you think the most effective, least hurtful way to approach that would have been?
1) Ignore the comment, 2) assuming he is the administrator of the WordPress blog he writes on, delete/hide the comment, or 3) lash out in a juvenile, sexist, hateful fashion.
I'm going to go with 1 or 2 being better than option 3 any day of the week.
> Apply() is the right way for grown-assed men do things.
sexist?
Am I not reading enough into it or something? As far as I can tell somebody was making a fuss about some non issue and was shot down for it.
Or it implies that whoever reads the text, man or woman, will be wise enough not to give a fuck for such minutiae.
It has been postulated, I think plausibly, that the commonality of such language is one reason why this gender disparity continues to exist.
Personally, I agree with this argument. If you disagree, that is fine -- but please show a little bit more respect.
The idea that "the commonality of such language" is the reason, is like the idea that video games and action movies create killers and criminals.
My point stands here equally well, for the same reason. (This idea I happen to disagree with, but that's not relevant.) The idea you mentioned is not obviously stupid. I don't see what dismissing the idea without refuting it adds to the conversation.
So?
>features exclusively discussion and pictures of men or languages invented by men
You've got to be kidding me right? The majority of programming languages are made by men, it is statistically likely any discussion of programming languages involve those made by men. You might as well attribute any discussion of programming languages as sexist.
I'm not sure if this response should have a /sarcasm tag on the end because it's more sexist then the original comment.
His reaction may have been juvenile and extreme, but I certainly don't think it was sexist. What makes you say that?
Scott Locklin is playing the game of taking proud ownership of his own labels. "I'm not a member of your cool modern hipster club, I'm just a geek who cares about fast computation." For comparison, see also "I'm a man hating dyke, meh, I get more pussy than you."
This is not true in the least. The "politically incorrect", as you put it, get free license to say and do sexist and racist things and its up to the people affected by that to speak up. When those people do speak up, usually their attempt to bring up that point is the subject of criticism.
> The person being accused of political incorrectness is automatically assumed to be a person of the worst sort - sexist, unthinking, mean.
That someone's tears and fears of being called out on a being a sexist is considered more of a problem then their sexist words and behavior is a key part of having privilege.
> For comparison, see also "I'm a man hating dyke, meh, I get more pussy than you."
That statement drips with misogyny, homophobia, and transphobia. If someone wants to take ownership of those aspects of themselves, that's fine, but that person deserves to be ridiculed for it.
No they don't. If they phrase factually correct statements the wrong way, society condemns and penalizes them. Larry Summers is a perfect illustration of this.
That someone's tears and fears of being called out on a being a sexist is considered more of a problem then their sexist words and behavior is a key part of having privilege.
It's not considered "more of a problem" by pretty much anyone.
Accusing someone of sexism is just a rhetorical weapon used by the privileged to make dissenters stop speaking truth to power. A person who's beliefs align them with modern liberalism has an extraordinary privilege - they do not suffer the presumption that they are bad people with evil beliefs. They can make content-free assertions, make ad-hominem attacks against those who disagree, while in contrast those who disagree must present mountains of evidence for their position.
(Oh yes I am being the jerk who uses the language of political correctness against itself.)
As for the statement you describe as "drips with mysogyny, homophobia and transphobia", it's a statement I heard made by a lesbian who was taking ownership of the word "dyke". If you wish to ridicule lesbians who claim "dyke" for their own (or blacks who claim "nigger" for their own) be my guest.
Given that reality, I'm coming down more on the side of stopping and listening when I hear a criticism of elements of our culture, rather than becoming reactionary. The criticism may not be right...but, it's worth listening and thinking about it, rather than just jumping down someone's throat with an angry and childish insult. Is it "political correctness" or actually language that turns people away? I don't have the answers...but, programming wasn't always a male-dominated field. The fact that it has become more and more so over time tells me something is wrong. Listening to ideas on what that wrong thing (or things) might be is probably a start on fixing it.
I don't think the gender equality in the industry is worsening, as you say. It certainly has worsened in recent years, but the strong movement in support of women in CS is gaining traction. My opinion on it may be jaded by the "white male privelege", but to me it seems the balance is starting to be reinstated slightly. Numbers of females joining the department each year has been increasing since I entered the degree, and recent statistics released actually showed there is now more women than men in the post-graduate CS degrees. Whether this is the case everywhere, I don't know (and doubt), but it's certainly a start.
Again, I'm not sure I understand the nuances of why someone felt uncomfortable with this particular piece of language; but if I assume good intentions, all it costs me is a moment of my time to think about it and try to understand. And maybe, by taking that time and showing a little respect for that view, I'll help someone feel welcome into this community I've been a part of for so long. I think it's probably worth the trouble...and, in the end, I'm pretty sure I'm a better person than I was a couple years ago before I got involved in this kind of activism.
I don't know how to convince others to stop being reactionary, and instead simply think about what's being said. Anti-oppression training might be cool at tech conferences, but I suspect it'd be the least attended track.
"I don't know how to convince others to stop being reactionary" If it helps, your views have helped me consider my reaction to this. I understand and empathise with why he reacted how he did, but my reaction to his post was probably a step further into inappropriate. Thank you.
Plenty of the good things in APL are now available in other forms anyway. NumPy with it's powerful set of linear algebra primitives and the huge set of implemented algorithms is just as great at data crunching. Want mmapped files? No problem:
http://docs.scipy.org/doc/numpy/reference/generated/numpy.me...
That's not to say APL is not worth studying, it is built on powerful ideas. But to study the masters is to criticize the masters, not only blindly admire them. And history is not all romantic.
You are correct that not everything is built on K - typically there are gigantic concurrent enterprisey systems build in java/c++/scala which shell out to a computational core built in K.
You are also right that Python/Numpy is getting to be a good alternative to K, giving you less performance but better cooperation. That's why BankAm is building their SecDB clone on top of it.
K itself isn't as consistent as I'd like (odd corners like the auto-casting arrays, especially dict arrays to tables, and skipping NaNs in sums), but it's a great place to start.
J really needs a table or dictionary data type, or at least the tools to make one yourself.
I had a bit of exposure to A+ then, trying to debug bits of the interest rate model subsystem. I hated it. We were mostly using A+ as the wire protocol at that point, and I had endless problems getting the APL fonts set up, and so on. Non-ASCII was a huge black mark.
Once we got the license and expanded the team working on the models, we decided to switch over to using kdb+ for the basic data store. That switch was like going from night to day. I could operate over the entire dataset, rather than just on a subset. I could query things on the fly with real aggregations. Things that were a huge pain (and slow) using SQL were suddenly fast and easy, like "calculate the average default rate grouped by FICO in 25-point buckets, for loans issued after 2005" or "backfill missing data from the first non-missing observation."
The first example can be done in SQL, but it takes a whole lot more typing and is (IMHO) much more error-prone. The last, I'm sure you can do it, but it's a nuisance.
I first got comfortable with kdb+ as a database query language. I read "q for mortals" by Jeff Borror, and we actually had Jeff around for a while for questions.
After that, it we started actually using q to estimate the models. It was great at the data aggregation and was a pretty natural fit. It didn't have anything built-in, but so we wrote a few functions for multinomial logistic regression, and that was good enough. That part didn't have a huge advantage over using something like R. However, the interaction with the large data store was far superior, better enough that it was worth it to write our own routines to do the estimation.
We even used q to run the models in production. That was more of a reach, but it worked pretty well.
So, I liked kdb+'s ability to handle large data sets in a speedy way. It makes a fantastic query language. I liked q's conciseness. It doesn't matter much for actual in-production code, but if you're doing research, every character counts. It seems shallow, but I think it's a real effect. Writing a half-line of q to do a paragraph of SQL is a huge win. I liked q's array operations, but by then I'd already been using Numeric and numpy and such for a decade, so those weren't really new. It was nice to be able to write them easily, as opposed to something like "np.concatenate((foo,bar))", but it wasn't really novel.
Basically, it was kdb+ that sold me on q.
The more expertise is applied to a solution, the less likely it is that people can cooperate on that codebase. The languages and frameworks of today try to minimize the need for expertise to get things done, because expertise doesn't blend well with communicating about code. I think part of the ideas in the article is that by running away from expertise we have often arrived at inefficient solutions. We scale out, with huge codebases that are like cities, sprawling and vast yet with each component isolated and easy to comprehend. Learning from the masters means understanding complex ideas which are based on scaling up, on building a project-specific grammar that encapsulates the problem domain so elegantly that it allows for one line features, at the expense of being incomprehensible to someone new to the codebase or the field.
But even if it was 1987, that's not older than Perl or Python.
[1] http://python-history.blogspot.com/2009/01/brief-timeline-of...
While the language is wonderful I would not use it for production systems today. It is my opinion that the language came to be too early and, as a result, has not seen innovation for decades. It also does not help that the leading APL implementations cost thousands of dollars to license. APL is still being used extensively in financial institutions in Wall Street.
J has been mentioned here a few times. Even though J is the work of APL's creator, Ken Iverson, whom I had the pleasure of meeting at various APL conferences, I consider it an abomination and a huge step backwards. I have a feeling Ken might have sought more commercial appeal by using ASCII characters rather than specialized notation. Ironic because of [1]. However, in making this switch, he created a horrible mutation that anyone can rightly recoil at. The language looks like a random vomiting of ASCII characters on the screen. Sad.
Many have commented on APL on this thread. In most cases it is obvious that these comments come from lack of or limited experience with the language. The power of APL comes from two things: Abstraction and Notation.
Abstraction is very much like objects in OO languages or specialized symbols in mathematics.
Notation --through specialized symbols-- is huge. Ken Iverson's own "Notation as a tool for thought" [1] covers this in far more depth than I could attempt here.
I'll just mention one example: music notation. A page in an orchestral score looks like gibberish to someone unskilled in the art. However, to a skilled professional it reads, well, like music. While I am not at that level, I can read guitar or piano sheet music and imagine, for lack of a better word, the music with reasonable accuracy.
That's what happens with APL if you really know it: You read it like music. Much of the criticism and misguided opinion comes from people who simply don't have this level of proficiency with the language. All other languages allow you to be a pretty decent "hobby" programmer in the sense that you don't have to be able to read it like music and you can still produce and maintain software. APL, for non-trivial work, requires true expertise, not superficial exposure. It rewards you with unparalleled productivity and expressive power. It really is a cool language, but you have to invest the time to learn it and read it like music.
Another way to put it: It's like vim. It's a shit code editor until you invest the time and effort to use it instinctively, to play it like a reasonably skilled musician would play a piano: without having to think about which finger to use to press a certain key to produce the desired tone.
That's APL. It's very different than using C or Java. It requires expertise, time and dedication. And perhaps that's the source of its demise. Today most programmers don't care about diving to those depths. Most want to learn the basics and poke around API and code libraries to get shit done and consider super-simple stuff like pointers confusing and complicated.
BTW, to me Python has a number of APL-like features that make it fun and productive. I really enjoy it.
___________
also compare "under" in J to loan patterns and all the resource allocation mechanisms in algol family languages
Also, our 10K LOC workspaces is potentially equivalent to hundreds of thousands of lines of code in something like C. Another way to put it is that a single programmer can do the job of five or ten programmers in another language. And do it faster.
What APL need that only more modern languages have?
Let me preface this by saying that I've been away from regular use of APL for years. I've forgotten a lot of the details but I'll throw out o few seat of the pants thoughts on this.
In no particular order (and not specifically related to modern languages):
The option for compilation would be very interesting.
Within a function, variables are global by default. Huge pain in the ass.
Objects.
A language-defined interface to C (or some other suitable language) for
when you need to speed things up.
An extension to nested arrays that allows you to create C-like structures.
I am going after the self-documenting nature of having structure members
have name-based access.
An extension to function definition to allow for more than two arguments.
This would require a little language re-thinking.
I would also look into making a hybrid that includes some C-like syntax
with an APL core. C/C++ -like comments and conditionals with bracketed
groupings of APL code would go far in making code easier to read and
organize.
Native C-like switch() primitive with APL-inspired extensions. For example,
switch could take a vector argument.
APL-run-time-definable comparison operator. Sometimes I want to apply a complex
evaluation function to data rather than the simple equal, not equal, grater/less
than, etc. operators.
The ability to enforce data types if desired.
A means of telling the compiler/interpreter to optimize code
--at the statement level-- for speed or resource (memory footprint) and
other criteria. Certain operations can explode into huge multidimensional
arrays which isn't always the best idea.
Real-time extensions integrated into the language.
Ability to use GPU resources for computation.
Standard interface for custom hardware-based acceleration (read: custom
FPGA boards).
True binary vectors and arrays.
More flexibility in multidimensional array indexing.
A rethinking of the workspace model to better support multi-developer
environments.
Genetic and Evolutionary computing primitives built into the language.
Built in primitives for multi-threaded/multitasked computing.
Built in primitives for network access and processing.
Built in primitives for multi-core and distributed computing.
Built-in primitives for data exchange. For example, ingest or output
nested array data from/to JSON, SQL, XML or other modern data formats.
Better pattern-matching primitives. I'm thinking at least regular
expressions.
A general cleanup of the notation to remove lame text-based "fixes"
from an era when rendering non-ascii characters required custom ROMs
on the graphics card. All APL notation should be symbolic. ASCII
should be limited to strings, function, variable, object and other
constructs that require text.
Make it open source and make the open source version far superior to
any available commercial version.
K4 is a much simpler and smaller language than APL - and yet the programs come out simpler and shorter, and usually much faster. Unlike APL, it can be tokenized and parsed in advance -- meaning that, at least theoretically, a compiler can be written. Arthur's implementation is a bytecode virtual machine, but AFAIK there is no type inference there.
e.g. there are only 3 conceptual data types in K4: atom, list=array=vector, and dict. A matrix is a vector of vectors (what APL would call a nested vector), vastly simplifying APLs nested and axis operators.
A quick comparison of the code necessary to calculate all primes up to a limit in K and APL:
(!R)@&{&/x!/:2_!x}'!R
(2=+⌿0=(⍳X)∘.|⍳X)/⍳X
I feel the same way about languages such as ObjectiveC. Another abomination. There was no reason to do that. Nothing whatsoever was gained by saying: "Hey, let's do the same thing but come up with a different way to write it." Sure, O-C has a few nice things here and there but it did not advance computing in any appreciable way as far as I can identify.
As for the other extensions you listed come with K, that's probably great. I'd have to dive deeper into the language in order to offer even a superficial opinion on that.
One of the reasons I abandoned languages such as APL, Forth and Lisp, languages that I used extensively for many, many years is that they became less and less practical and relevant. I can apply C to nearly everything from embedded to system work and even in modern hybrids where FPGA's are integrated with capable microprocessors. On the hardware front languages like Verilog are very reminiscent of C and, as long as you understand that you are actually describing hardware and NOT writing software, are easy to pick-up with the appropriate background. You move up to languages such as C++ and other layers open up. PHP, Python, Objective-C when I absolutely must and Java if I have no choice. All of these are very flexible and relevant tools that have, for the most part remained relevant and useful for years. That range of applicability will never be achieved with something like APL. If an APL-like language is going to come to the forefront it will be for very specialized applications where it makes sense. It will not be to run a shopping cart on a website or control a servo on my robot. That's just reality. I love APL. I devoted a huge chunk of my professional life to it. I can't see using it or the wanna-be variants for anything today. Sorry.
e.g., the first element operator is unary asterisk e.g. "x", which is like C's pointer dereference (which gets the first element ..). There is no symbol for "last element" - instead you use "|x" meaning "first of reverse of x". Similarly, there are no compress/expand operators; Instead, there's a "replicate" operator which unifies and simplifies both (and has other uses to boot).
In fact, it looks like Arthur used the number of ascii symbols as a constraint for the number of primitives - which are all single characters. (They can be postfixed with a colon to force monadic comprehension, but that's not part of the operator). As a result, K is as much notation as APL is. The domain is "writing real world programs" instead of "expressing algorithms", and it shows, but it's still notation that -- once acquired -- reads like math or APL.
> One of the reasons I abandoned languages such as APL, Forth and Lisp, languages that I used extensively for many, many years is that they became less and less practical and relevant.
I suspecting it's going to make a come-back. The APL / J / K computation model is much easier to apply to GPUs. But prediction of trends is very hard -- doubly so when it is about the future :)
Can you please elaborate on this and give a few examples?
def problem_1a():
return sum(set(range(3,1000,3) + range(5,1000,5)))
I'm wondering what your opinion is of A+ (dialect of APL from Morgan Stanley)? GPLv2 codebase, supposedly is/was used in a commercial context, so should be pretty stable. Unfortunately, the last activity on the project seems to be from 2008. To my uninitiated eye, it still appears to be the best thing out there for learning and dabbling in APL.
The A+ website is at http://www.aplusdev.org/
You are missing the point. I could proclaim that from now on we are going to use the /$ character sequence to implement the integration function in a hypothetical language. I could similarly continue with that idea and create a bunch of transliterations and cover a wide range of functionality. And, yes, people would be able to learn it and learn to read it. One example of this is reading regular expressions.
However, having that that, you have not created a new tool for thought all you have created is the ability to --as I described it before because I despise it-- vomit a bunch of ASCII characters into a file and call it a program.
Notation is powerful. The association of symbols or icons with ideas is powerful. The play, stop, fast forward, fast rewind and other buttons on your remote are easy to recognize, read and use. You could purposely avoid using symbols and use words: "FFWD", "STOP", "PLAY", etc. However, using symbols (notation) is by far the most efficient and communicable tool.
Going back to the integral, well, the existing mathematical integral symbol conveys the entire concept very efficiently.
J, to some extent, comes from an era when it was downright painful to even attempt to print or display APL characters. You had to replace character ROMs in printers, graphics cards and terminals. You had to replace print heads on other machines. APL characters before expanded character sets and the ubiquity of GUI-based operating systems were painful to implement and communicate. J attempted to solve this by transliterating from the APL symbol set to an ASCII-based equivalent. When that happened one of the most powerful aspects of APL was thrown out the window.
That's why I call J an abomination. The functionality offered by J is probably an improvement of APL. However, walking away from attaching meaning to custom symbols was a huge step backwards. Perhaps Iverson thought the reason APL was not seeing wider adoption was the character set, gave in and sought to fix it by going to ASCII. In my not-so-humble opinion, that was a huge mistake. ASCII is gibberish. Symbols, much like a page full of mathematical equations, means creating and using a new language that become part of your mental process.
You did mention that you did not bother to learn APL. That's the reason you don't get it. I am trying to use mathematical and musical notation to illustrate the value of notation. It is obvious I am not doing a good job.
Let's just disagree and move on.
FYI: The RaspberryPi port of Dyalog's APL is available for free. See http://packages.dyalog.com
In my own situation APL is a language looking for a problem - I don't currently have a use case for it.
http://books.google.ie/books?id=azvE8V0c-mYC&pg=PT121&lpg=PT...
Or, download it and run one of the intro labs; that's even better.
As a language, J isn't ever going to fit comfortably into a "Learn X in Y" format -- there are too many unfamiliar ideas, it isn't just syntax.
It's too bad. I find VMs interesting, but I wouldn't touch that C with a ten-foot pole. It's IOCCC quality.
The "shock" of looking at it is exactly the same as the shock of looking at APL or K programs to begin with. They don't look anything like code that most people are used to, but since they tend to be highly regular, the impression of noise is misleading.
Can you name some that are not?
Here's an offensive but accurate taxonomy. There are 6-month languages/technologies and 20-year languages. The 6-month set are optimized toward being quick to learn fast. Java, for example, inherited C++ syntax while cleaning up the OOP, so the C++ programmers of the time could pick it up in quickly.
Now, q and kdb are 20-year languages. It doesn't take 20 years to be productive with them, of course; it's that they're designed to optimize a person's productivity over 20 years, and not in the first 6 months. Unlike the various distributed tools out there (Hadoop, Hive) it will take you a long time to learn them, because they aren't like anything else. Superficially, q is Lisp-like, but the performance implications are very different and it doesn't look like a Lisp at all. (It parses right-to-left, for one example; also, it's written as a stream rather than in nested s-expressions.)
Many of these modern tools being bashed have grown up in a world where 20-year tools can't really be built; they'll never catch on commercially in a world where people change jobs every 2 or 3 years, when MVPs are more important than solid decades-proof engineering, and where employers are pretty averse to their people taking time to just learn new technologies.
There's too much volatility in the world for it to make sense for most people to learn q and kdb right away. You can get half-decent in a couple months, but it'll take much longer to learn why those tools are (for some use cases) so damn powerful.
This 6-month vs. 20-year tool concern is part of why I've gravitated toward Clojure recently. I haven't seen anyone else do quite as good a job of balancing those two need sets as Rich Hickey and the Clojure community have. Clojure is (becoming) a 20-year language, but it's no harder to become basically productive in than Python or Ruby.
Scheme, on the other hand, was also minimalist and consistent - s-expressions and about dozen special forms, and out of this everything else could be implemented, including closures, continuations, and classes and message passing, if needed (CS 61A 2008 does this for you).
Haskell is.. well, everyone know what Haskell is.
As for APL, in my opinion, the power of a language comes not from using esoteric math notation, but, like in CL or Scheme, from possibility to easy add new control structures (call/cc) and special forms (with macros) or even create a new languages, such as Arc.
We don't have six characters in the English language.
The only problem is these narcissistic hipsters, creating so much noise with meaningless blog posts about this or that way of writing abstractions. No surprise that I was confused - reading blogs instead of classic books is a waste of time leading to even more confusion.)
If you don't like for loops, yep, we've got that too-- use Scala or another functional programming language to write your MapReduce jobs. Or use SQL, which is a lot more powerful than APL, and a lot more accessible to businessey types than functional programming. SQL is a declarative programming language, by the way-- I don't see any for loops there.
The efficiency argument makes no sense either. Does the author understand that a just-in-time dynamically recompiling virtual machine is faster than an interpreter? If so, he doesn't mention it anywhere. You know, sometimes old technologies are just... old. You could at least compare Java and the JVM to something like Lisp machines.
People overestimate the gains that are to be had from mmap. I am currently in the process of adding mmap support to HDFS and I know what I am talking about. mmap gives gains, but only when the data is coming from memory, and only when you can reuse the mmap. Otherwise, you're better off reading even a 512 MB file via read() and write(). The reason is that syscall overhead is not that high on modern UNIXes (like Linux), and page faults involve a transition into kernel space anyway.
How could that possibly be true? What does "powerful" mean? Certainly not that it can compute things that APL family languages can't, and the design of APL/J/K etc. as languages is so much better that it isn't even fun. SQL is the COBOL for the 21th century.
Does the author understand that a just-in-time dynamically recompiling virtual machine is faster than an interpreter?
As memory latencies relative to instruction execution times go up, this factor becomes less and less relevant. An even worse problem is that your "just-in-time dynamically recompiling virtual machine" can't even vectorize properly, whereas APL has high-level array manipulating operators that make it comparatively easy to generate vectorized code. Your "just-in-time dynamically recompiling virtual machine" would have to "intelligently" recover high-level patterns from seemingly nondescript for() and while() loops to do that. Guess what: it doesn't! When you consider the fact that memory latencies matter these days, the fact that an AVX vector instruction can do a whole lot of work in a single cycle (or a few of them at most), and the fact that the integer units that would otherwise be idle can handle dispatching the next primitive operation in the meantime, there's a potential for even an interpreter to kick your "just-in-time dynamically recompiling virtual machine"'s ass.
I agree that the JVM does not use memory well. For example, it has a high per-object memory overhead. And there are things such as lock widening which are a real design problem. But even considering these things, the JVM still manages to consistently beat other choices such as Perl, Python, Ruby, and so forth.
Yes, vector instructions are great, and Java can't really use them. There are also other builtin instructions such as the CRC ones which would be nice to have at the language level. We have been adding JNI segments to make use of these. It isn't the syntax of loops that is the problem, but rather, the presence of side effects in the language, that makes auto-vectorization difficult. Probably annotations should be added to make this easier, like Intel has done for C/C++.
None of this means that APL is going to come back. The scientists and engineers who originally used it back when 4kb was a lot of memory are just going to use R, MATLAB, Mathematica, or another choice like that and gain a lot of the same micro-optimizations you love. People who want to do big queries over giant data sets are going to keep using Hadoop.
The one thing that could maybe dethrone Hadoop is a system that made use of GPUs (graphics processors). Some of them have 512 cores now-- that's a lot of power, and Java can't really harness it at all. But GPU processors are really restricted in how they can communicate and how much memory they can access, so it would not be as much of a general purpose solution.
If mmap doesn't bring you significant gains, you are doing stuff wrong. E.g. You serialize objects. Don't.
Hadoop is horrible. I've heard Spark manages reasonable performance (subject to the shackles that Hadoop compatibility entails), but haven't used it.
If you think SQL is "more powerful" than APL... Qualify that. Because e.g. APL is Turing complete, whereas SQL isn't.
> That's something which should be marginally faster when it's faster,
No. If used properly, it is crazy faster.
Let's say what you need from the data is element no. 1821942 in the stream.
Hadoop standard: read & deserialize & discard 1821941 items (just so you can get to the 1821942nd item). read & deserialize and use one item.
K standard: (equivalent to): seek directly to element 1821942, read it and use it.
except .. K does it slightly faster than that, using mmap: It just accesses the location for this element in memory. If it is not in memory, then the operating system will arrange for it to appear in memory in a way that's usually more efficient than read. If it is already in memory, then it is exactly one memory read.
Now, if Hadoop & friends stored their data in such a way that you could do a random access read (and used a read syscall) then mmap would _still_ be about 1000 times faster than a read() call for access if the datum is already in the o/s buffers.
The magic is not that K can read stuff faster -- e.g., it can't read a hadoop stream faster than hadoop can. The magic is that K makes it easiest to just store your data in a random-access mmapable way, so that mapping the data back in takes ~10ms, and from then on, you only pay for the data you actually use without having to explicitly read it -- and usually, you pay less than you would have paid if you explicitly read it.
I've been impressed with, for example, Symas's lightning mdb: http://symas.com/mdb/microbench/
However, it's not as if someone replaced my Toyota with a Formula 1 car. Properly used mmap is very nice but some people are getting carried away with the superlatives.
I hope this doesn't come off the wrong way but I really wonder if the "crazy faster" amazingness you're referring to isn't a function of comparing K to Hadoop. Which, you know, might just be setting the bar way too low.
Well, it all depends on your access patterns.
The bottom line is: in a modern (later than 1985 or so) Unix system, mmap is hardware based virtual memory; read() and write() can give you software based virtual memory. Who do you think is going to have better performance?
Compare and contrast:
a) mmap an 8GB file into 64-bit address space, start using immediately. only stuff you actually use gets read from disk.
b) using only read() access, read what you need, manage caching to avoid syscalls, etc.
It is unlikely b) can be faster at all, and it will only be competitive if the granularity of reading + the cost of cache management is low.
In cases where either granularity or cache management is expensive (which is common), mmap is likely to win big time.
Yeah, it's too bad this method doesn't exist:
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/..., byte[], int, int)
It's too bad that this doesn't exist: http://www.parquet.io
It's too bad that this doesn't exist either: https://issues.apache.org/jira/browse/HDFS-4953
You are amazingly, astoundingly, misinformed.
It's no surprise that you are wrong about mmap as well. It's only a performance advantage when the PTE (page table entries) are already populated. The PTEs may not be populated, even if the file region in question is resident in memory. And even then it's only about 2x, not "1000 times faster". There was no "serialization and deserialization" involved in our benchmarks, either.
And how, if you may help me, do you access object #n in the stream (for non-trivial n)? Not byte n in the stream, which this appears to give you - but object record #n ?
> It's too bad that this doesn't exist either: https://issues.apache.org/jira/browse/HDFS-4953
Your patch is two weeks old, and you expect anyone to be aware of it?
> You are amazingly, astoundingly, misinformed.
No, I have inherited a hadoop project from people who did it "by the book". And then rewrote it in C with mmap to gain a 20-1000 fold improvement in speed depending on workflow. That had been in 2011, but I'd be surprised if things changed so significantly since then.
> It's no surprise that you are wrong about mmap as well. It's only a performance advantage when the PTE (page table entries) are already populated. The PTEs may not be populated, even if the file region in question is resident in memory. And even then it's only about 2x, not "1000 times faster". There was no "serialization and deserialization" involved in our benchmarks, either.
Are you using hadoop map reduce, or your own HDFS code? Because every single non-trivial map-reduce jobs that I've met serializes and deserializes objects, spends 90% of its time on I/O, serialization, network, compression and stuff - and that's actually the recommended way to do so.
Which is why, incidentally, spark manages to be so much faster (100 times claimed - I've heard of >70) than hadoop - it keeps stuff in memory.
It's no surprise that you are wrong about mmap as well - your description is correct if and only if you are going to read the entire file. Which is standard for hadoop map reduce, but actually not that common in a well designed program, and certainly not common in the K world.
And you certainly did not understand the 2x vs 1000 times faster comment, which stems apparently from your (faulty) understanding that everything has to be read. In an mmaped file, if you need one byte from the middle, you pay for reading one block from the middle exactly once. You do not need to pre-read everything, and you don't need to manage caching yourself. That's where the 1000x times come from.
If you do read everything, it's not 1000x times. But you've possibly wasted 1000x times memory -- and 1000x times I/O to get to that byte in the middle that you need.
No, our HDFS mmap testing did not include MapReduce. I'm pretty sure I've been repeating that over and over as well. Why don't you write your own test program in C if you don't believe me?
How about we listen to this Linus Torvalds guy? Have you heard of him? [http://yarchive.net/comp/linux/o_direct.html]
Right now, the fastest way to copy a file is apparently by doing lots of ~8kB read/write pairs (that data may be slightly stale, but it was true at some point). Never mind the system call overhead - just having the extra buffer stay in the L1 cache and avoiding page faults from mmap is a bigger win.
The situation might improve when Linux gets support for hugepages on (regular) file-backed mmaps.
I'm having some trouble putting it together from context but I'm wondering if you're talking about manipulating files using mmap from... the JVM? That's kind of interesting.
This whole thread makes me very sad. I might have to write a "Mythbusters" post about Hadoop at some point.
And in a C (or K) mmapped file, you access the n's object by accessing a memory location. No index lookup, whether O(log n) or a hash table.
I'm indeed not familiar with Parquet - Hadoop failed me so badly I do not feel like I need to keep up to date with every new thing. I might take another look when when it's been around for a while.
> Right now, the fastest way to copy a file is apparently by doing lots of ~8kB read/write pairs
That again discusses reading the whole file. When mmap delivers the huge gains is when you don't actually need the whole file - rather, when you need parts of it that you discover while reading. Which in many workloads is the case (or would be the case if you did not have to conform with some framework's constraints about the order and kind of I/O you can do).
mmap gives you hardware virtual memory and assistance for caching; Every other solution is a software virtual memory and implementation for caching. While it has its own constraints (e.g., you have to store data in a directly usable layout, rather than "disk frozen" layout), if you do that, things do work exceptionally well when you do.