Hacker News Reader

Comment by tikhonj | Hacker News Reader

I wonder if there is room for some smaller languages optimized specifically for data analysis. In particular, I wonder how a carefully designed non-Turing-complete language would fare.

That would be a really cool project to work on: design a minimal language for expressing most types of data analysis at a higher level. If the language is sufficiently small and simple, I could see some very powerful tooling being possible for it.

Perhaps it might make sense to go even more specific: have a small language designed not just for data analysis but for analysis in a very specific vertical (say finance or bioinformatics). It would be awesome to let people express their ideas in terms of the domain and not worry about low-level details like loops.

View on HN

It seems like a good idea, but I wonder how actually useful highly specialized programming languages would be. Why?

1) Most data analysis tasks boil down to roughly the same things: accessing the data source --> data cleaning -->simple transformations --> (optional)stats/fitting/ML/specialized procedures-->pretty pictures and reporting.

2) Not everyone wants programming to be the main component of their job.

People who can take advantage of the flexibility that programming offers can usually take advantage of existing technologies. People who don't enjoy coding will always look for of-the-shelf solutions that have pretty GUI's with magic buttons that solve all their problems. I just don't think there is a huge market in between to be filled... in the domains that i've been exposed to anyway.

I disagree with your supposition. I think highly specialized languages exist and are highly useful to non-programming communities. I think there is plenty of proof of their usefulness and room for growth.

For instance, consider illustrator products or d3? Both of these are specialized ("deep") tools for creating pictures that I've used extensively in the "pretty pictures and reporting" stage you outlined.

Also of serious note are BUGS[1], JAGS[2] and (recently) Stan[3] as small semi-declarative languages for MCMC model building, fitting, and checking.

SQL is an obvious example of a component of the "simple transformations" step.

[1] BUGS http://www.mrc-bsu.cam.ac.uk/bugs/ [2] JAGS http://mcmc-jags.sourceforge.net/ [3] Stan http://mc-stan.org/

I think you are going in the wrong direction. It's too easy to paint yourself in a corner that way.

> It would be awesome to let people express their ideas in terms of the domain and not worry about low-level details like loops.

Yes! I think you want to build this functionality on top of a powerful language to easily handle the dirty ETL work too. This is the reason lots of financial companies use python with scipy, numpy, pandas, etc, on top of it.

I've been working on a guaranteed-terminating language in the vein of APL/J. Primitive recursion is possible, but with the looping all implicit due to array shape, infinite looping is impossible. The issue I'm not sure how to handle is that some algorithms take the form, "repeat the following until convergence: ..." with no obvious way for a machine to prove that convergence will eventually happen.

I thought this might be the APL language. Now I'm not so sure... But it is, there looks to be some loops here

http://www.apl.jhu.edu/~hall/Lisp-Notes/Loop-Summary.html

I guess "APL" in that context means "Applied Physics Laboratory."

I'm working on this direction in healthcare. I want to make a little specialized, non-Turing complete language for handling evented data streams from patients.

I think there's a lot of power in certain kinds of non-Turing completeness. Email me if you want to talk about it.

Stata is close to your ideal - it is popular with business and economists. However, no one can ever resist Turing completeness - that would require shelling out to another language for that one small thing every project needs (a different thing for every project)

Such a thing was already invented a long time ago and widely celebrated - SQL. It eliminated the need for loops, was designed for ease of use, had a lot of cool features added like ACID, took over the world and ran every kind of business and website. Sadly, because it was invented too long ago (1970s) many people thought of it as uncool and failed to realize it's absolute awesomeness.

Reply on news.ycombinator.com