All ideas are old ideas
chris-granger.com
chris-granger.com
Just think of everything that you know how to do which involves a filesystem. I regularly use editors, grep -r, cd, edit in one directory and run unit tests in another, revision control systems, etc. People have invested a lot of energy into these tools, and have strong opinions about which ones they prefer and why. (Perhaps you prefer an integrated IDE. Doesn't matter, my point remains.)
As soon as you move to an image based system, your entire toolset no longer works within the inside of that image. It is, of course, a programmatic environment and replacements for every filesystem manipulation you are familiar with are available. Sometimes those replacements are better (though better is often just in the eye of the beholder). But they are guaranteed to be different. And as you move from one image based language (say, a Lisp dialect) to another (eg Smalltalk), they will be different again.
By contrast if you move from one filesystem based language to another, you will still have a learning curve, but there is a large toolset that you do not have to relearn.
We've been working on some big changes to the architecture of Light Table that should set us up for figuring that out in the future. A blog post is probably coming next week about some of that.
Yes, the image interop part is a bit more difficult to solve. I'm not saying it's exactly like the switch from carriage to car, but I'm saying that some changes are worth losing existing skills
Consider also that core ideas that make image-based environments good can be separated from the image-based system and moved into non-image based systems. Two examples that I would give for Smalltalk are Ruby's borrowing of Smalltalks OO model, and various IDEs borrowing the idea of a refactoring browser.
Borrowing the other way is much harder. For instance consider the distributed version control patterns that git enables. Image-based software development communities are sufficiently small and fragmented that they are unlikely to all duplicate the functionality of git. And because of the internal differences, they each would have to reinvent it. By contrast once git became available, it was immediately available for use with C++, Java, Python, JavaScript, Haskell...
There are advantages to image-based environments, but do they outweigh the disadvantages? Before concluding that they do, you might want to re-read Worse Is Better.
A possible approach is to consider the file system the image - consider a UI for managing a VM image and processes within that image.
The difference is then essentially deciding what is the center of the system.
The older image based language systems assume they are the center and all must pass through them - like a monolithic kernel.
Perhaps what could be explored is more transparent integration between language runtimes and file system objects.
1) move from an information pull to a push. (Sidenote: This is something that 'kind of' works in OSX GUI, although it seems to me that 10.7 has broken this functionality for certain usecases. E.g. I'm connected to my host mac with my linux vm through SSH and when I've updated a file from within that VM, the host Finder windows get updated automatically and Preview.app windows get updated when I click into them).
2) Extend the functionality to runtimes as well, i.e. enable hotloading like in the new Unreal engine.
Intersystems Caché.
There's a CACHE.DAT, which sits on the filesystem as a giant monolithic file. It's just a big block device containing a B-tree. In some nodes of that B-tree you'll find binary code. There are specialised tools to manage all this. In other nodes you will find data. There are specialised different tools to manage all this.
There is the one virtual machine and it only runs one language fairly well, called MUMPS. There is an object-oriented addition to it. It's basically a huge bunch of macros that compiles down to... MUMPS.
You can, in a hackish way, sometimes call specific functions from a DLL but that's as far as it gets. However, effectively...
No libraries. At all. And that's the biggest pain.
No libraries. I want you to let that sink in. All those things on Github and SourceForge and everywhere you, simply put, can't use.
No libraries. Except the few that the vendor, Intersystems, chooses to eventually let you have... with often broken functionality. There will never be any UI toolkit or Javascript interpreter or HTML engine or nice XML parser or any of those things. There is a crappy JSON parser, but that's about it in terms of recent niceties, and it's only there because some other customer would have paid for it.
No libraries. And that's the biggest problem with these software image things.
No libraries.
The biggest advance that we managed to get in was installing hooks in the proprietary source code editor (no vim or choice of text editor here) to export and import the code from disk. On save and load. It helped a lot to be able to get all the classes into actual files that we could use SVN with and made things so so much easier.
Software images might be a good idea in theory.
But in practice, you are throwing out your file browser, your text editor, your libraries and everything, everything, else.
EDIT: To clarify, I know that Lisp and Smalltalk have a fuzzy-to-nonexistent line between code and data. What I want to know is how you'd manage that in practice. If I'm changing an existing piece of software, I don't want to expose production users to its incomplete state, so I can't change the running production site directly. (Right?) But if I'm developing locally, or even in a staging version, I don't want to transfer my test/sample data.
Maybe a better question is, "What is a typical workflow like for modifying production software?"
To quote an old lisp saying, code is data is code.
If I want to do an image deploy, what I'll do is load the libraries into the image as part of the build, set up a 'main' function that the executable will trigger, then main will read any config files and go. This image will be an actual executable for the OS. The receiver will execute the executable and not know the difference.
edit: In terms of version control, each deploy to production would generate a tag.
Though if you were making a program that should be deployed at the end user you would properly make it read and write files or communicate with a server over the internet and obviously that kind of data cannot be put into the image.
In a modern Common Lisp web server set up, you'll likely have a SWANK connection available for debugging and patching. You will have to write live-patching code to correctly lock, drop/unbind old definitions, rebind new definitions, and unlock. It will likely remind you of a DB migration script.
For a non-live upgrade, you'll have a process that starts up the new server image, gets it ready to receive connections, then switch the routing & drop the old server. It's not too different from other systems, IMO.
I think that may have been part of why the Lisp Machines died - not only were they obscenely expensive, but they were extremely expensive single user machines. This is fine if your entire development team is one MIT supergenius but becomes problematic when you merely have a handful of rockstars.
Typical lisp development today usually involves using git and loading files into the image using SLIME.
And, yes, LispMs were very expensive and cpu-performed less well than a risc box.
Or sometimes I probably just used the existing image. I don't remember. Unless your code depends on something being not defined, or you deleted a lot of old code, it wouldn't really make any difference.
If I ever had a lot of code, I could have built the image offline (though Lisp compilers are pretty fast). If I had a lot of code to build and database migrations to handle on a live database, it could have gotten hairy. I guess that's just the price I would have had to pay for the added flexibility of being able to build my image offline. IME, complex database migrations in production can get hairy on any language/platform.
It's not really that much different from Python or Ruby, is it? They have a "repl" and an "image", except they're much weaker and not formally defined, or portable, or able to be saved or loaded, or anything like that. But if Python 4 was released today and Guido said "there's now functions to load and save images!", it's not like every Python deployment in the world would stop working. Everyone would have a new mechanism they could use, or not, as they liked, but it wouldn't affect their existing procedures.
It's always refreshing to see someone explore the work of those who have had similar concerns or solved similar problems. Whether they have independently chosen a similar direction or are standing on the shoulders.
Provides a good reference for new comers wanting to understand what approaches have been explored before.
We live in exciting times. The confusion of (programming) tongues is in full swing; the CPU clock wall is with us; high level features such as closures and GC now have fairly efficient implementations - it seems there are now enough transistors to deliver much of the abstraction that was envisioned by the founders but gave way to pragmatic concerns in the early microcomputer era.
I am looking forward to IDEs that manage and simplify complexity, and hardware ISAs that throw a few more transistors at things like STM, tagged data and memory management.
I can see a future where I can edit code in a language foreign to me, but see an instant side-by-side (editable) version in a dialect I'm more comfortable with. Switch in another view and drag and drop states in my state machine via direct manipulation, and so forth.
Personally, I am wondering when light-table mode for emacs/common lisp will come out. :-)
light-table mode for emacs/common lisp would be neat. As a schemer I think what would be really awesome is to see light-table merged with the emacs/guile/geiser work. Geiser is also trying to resurrect the lisp machine.
Yes, it was awesome compared to using a timeshared VAX (or whatever), but as fantastic as my 1108 was for its time and no matter how much I loved it at the time, I like my development environments today a lot more. Examples: development and debugging of both server side and client side code (GWT on client, something REST full on the server side) with IntelliJ Java debuggers open on both client and server code - hopefully you may only need this once a month but useful when you need it. Repl based development with a smart IDE (like Clojure in IntelliJ or Ruby in RubyMine). Interactive Scala worksheets in the Eclipse/ScalaIDE (sort of like Chris's Light Table).
In addition to a lot of NLP work on my 1108, I was also just getting into neural networks back then and my NN simulations ran like mollasses on the 1108. Any of my laptops are many orders of magnitude faster for just about anything I do than my 1108.
I'm suggesting that barriers to modifying systems have had an accidental benefit of favouring standardization.
The basic problem is that it's hard to create systems (APIs, generalization, documentation, test-cases, integration with everything else). Brooks claims it's 9 times as much effort as making something that "works for me". If you can't amortise that over many users, it's usually not worth doing the job properly.
But if it's just for you, you don't need all that polish, right? Well, no. If you are standing on the shoulders of giants, it's really helpful if those giants are stable, function as you expect, interact correctly with other tools, and have clear documentation for when you've forgotten the details (writing it yourself doesn't make your memory perfect). SOTSOG is really important for building complex systems, because it enables you to push down details to a lower level of abstraction, so you have less clutter to deal with. It's getting the abstractions right that is so difficult (we've all seen it done wrong).
There are huge advantages to using standardized systems, that have been improved, tested, documented etc by others. If you want to do something different, it's very helpful if you can start with standardized components around the new part, so you can just concentrate on the interesting stuff, and not need to build the whole world from scratch. Of course, the existing stuff is not perfect - the question is whether it's worth it for you to spend the time needed to make something better.
Some issues in assessing this argument: These are ideas have been around for a long time, but haven't taken off. Highly modifiable systems like lisp, despite having clever and devoted users, lack the standardized libraries of less flexible languages like Java.
But there is one kind of area where such systems are absolutely brilliant - exploration at the systems level. Many (most?) programming language ideas have been developed in lisp, even though did not become popular in that form, because lisp is great for language exploration, but not for standardization. Similarly, an OS that can be modified on the fly, is great for exploring OS ideas. You can iterate quickly, see the results and so on.
Of course, it seems possible that one could have it both ways - a highly flexible system that also had standardized components. But that doesn't seem to happen. Flexibility and standardization are natural enemies, both in differences in the tools that support them... and the kinds of people who use those tools.
BTW: There's a middle way. Create tools that can be standardized on. i.e. libraries, frameworks, engines, DSLs. Light table could be perfect for that... but people who use it may want to make tools that are just as flexible as light table, and so don't support standardization.