Hacker News Reader

Comment by sja | Hacker News Reader

I'm a little hesitant to learn how to 'understand large programs', or how to structure them for that matter, from a repository that has 100 code files in the root directory.

I'm not sure what I'm looking at?

View on HN

Try the README: https://github.com/akkartik/mu/blob/master/Readme.md

It's a new programming language.

edit: The source seems to be structured intentionally in a recommended reading order. See the comment at the top of

https://github.com/akkartik/mu/blob/master/000organization.c...

Layers of code are filled in as you read down the list of files. It's an interesting concept, similar to literate programming. (And FWIW, these source files are C++ code for the compiler, not an example of Mu code.)

The author also has a blog post up here: http://akkartik.name/post/mu

Thanks for figuring all that out and laying it out so clearly!

More details on my flavor of literate programming: http://akkartik.name/post/wart-layers

Author here. For my part, I have never understood why people consider it to be a good thing to squirrel code into a bunch of different sub-directories.

a) It makes the build scripts more complicated, which means they'll be more likely to break on some poor noob, and that when they break it'll be less likely the noob will be able to tell how to fix it.

b) Invariably the codebase accumulates dependencies between directories that are uneconomic to reorganize. At least a flat directory has a shot at under-promising and over-delivering.

c) Maybe people do it to make the place look neat, they way I used to 'clean my room' by stuffing all my dirty clothes into drawers. But codebases that are messes at a deep level are less likely to be cleaned up if they look clean at a superficial level. And all codebases eventually turn into messes, the way we've done things so far.

Sure, directories can be used badly (and the Java and Ruby projects I've seen are often very difficult to navigate because so much is the structure of the language, rather than the structure of the project).

But they can also be used well, and the promise of a meaningful directory structure is really appealing.

Here's a question, given all of what I understand about your layered way of structuring code: how easy is it to spin off logical portions of a project? If my project accretes its own HTTP server and I want to turn that into a stand alone library, could I? Would it be possible to "rewrite history" so that the HTTP-related changes never appeared in the layers to start with?

Thanks!

Yeah, that's a good question. Even though the mu core is at the top directory, there are a couple of apps in sub-directories of their own. To run a single-file mu program you say:

  $ ./mu factorial.mu

To run a more complex app, you just give a directory name rather than a filename:

  $ ./mu edit

Files inside the directory are loaded in numeric order just like at the top level. You can also run just a subset of the layers for the editor:

  $ ./mu test edit/001*
  $ ./mu test edit/00[12]*
  $ ./mu test edit/00[1-3]*

(Notice how the number of tests/dots grows at each layer.)

Since the layers are just regular files there's nothing stopping you from rewriting history as much as you want. That ability was precisely what I built them for.

I've only recently started using directories, so I'm sure there's stuff here I haven't considered. Feedback most welcome.

That same line of thinking could be used to just put everything into a single huge file.

I have to admit, I like long files. Arc's compiler is all in one file (https://github.com/arclanguage/anarki/blob/master/ac.scm) and it was a big influence on me. It was just so great to be able to have it open on one side while I wrote programs in Arc.

You're right that there are points beyond which a single file becomes too unwieldy, but with a good editor that limit is quite high for me. Maybe 10k LoC. Directories can grow even larger before they start having problems.

It's not the same. I can open, process, or skim a directory containing a thousand files without much trouble or CPU activity. Having a thousand files worth of information in one file means I have to run through all that data at once. This has implications in data integrity as well if I'm making changes or worried about bitrot.

So, no, there's a difference. I'll also note that Daniel Bernstein of qmail & NaCl often used a filesystem how some use databases under the philosophy of "Why create new, complex functionality when you have well-tested code that gets the job done?"

I think the reason why people (I put myself into this camp as well) prefer directories for organization is because there is an entire ecosystem built around them. Backing up, recursive search, exploring via ui, programmable access, programmatic access using language of ones choice are all choices available to the user. Code organization isnt the only place this shows up, organizing music and photos has similar semantics - apps which try to do the organization via higher level metadata rarely put in the engineering effort to come up with robust solutions for all the above use cases I cited abode.

His idea is that the way to understand a large computer program is by running it, testing it. So he's made (what I understand) a very nice UI for introspection of a running computer program. You can see what is happening while it's running.

Designing a programming language is kind of like a 'rite of passage.' I think most of us have at least thought about how to make languages better.

Heh, I've already had my programming language rite of passage[1]. Mu was my attempt to not build yet another language. Hence the lack of syntax.

Thanks for your kind words! I'm gratified that somebody understood what I'm trying to do in spite of my crappy writing skills.

[1] http://akkartik.name/post/wart

I'm glad I got it right :)

I like your idea, and I think there's room for improved introspection in debugging.

At the same time, I think that if a program can't be understood without being executed, then it's already lost (in terms of readability).

Certainly, for example, if you have a threaded program, you need to be able to visually inspect and verify that there will be no deadlocks or race conditions when the program is run, because you won't be able to test every possible condition in a debugger.

It might partly be a personality-type thing. I can't imagine visually inspecting and verifying anything if I didn't write it to begin with. I wouldn't know where to begin thinking of possible ways to break it. On the other hand, it seems far more natural for the author to say, "here are the race conditions I considered, these points where a context switch would be maximally inconvenient. And still lo the program works." This way the reader doesn't have to recreate such situations from scratch. He or she just has to verify the provided situations, or notice a scenario that was missed. That seems like an easier ask from someone new to the project.

(Mu scenarios can't insert context switches yet, but it's planned.)

Perhaps it would help to think of the dichotomy as between the rules and the state space of inputs they handle, rather than between reading and running. Seemingly simple code can often hide surprising subtleties. Why is this line written just like so and not thus? How does everything turn out just right in this one situation? Tests help to record the right questions for the reader to ask.

  >  I can't imagine visually inspecting and verifying anything if I didn't write it to begin with.

It's a skill, you need to develop it. You develop it by doing it :)

  > I wouldn't know where to begin thinking of possible ways to break it.

For deadlocks, look at every single lock used. Show that one of the Coffman conditions doesn't apply, and you've proven that there can never be a deadlock.

For race conditions, look at every piece of shared memory (or other shared resource). That is where race conditions always occur.

For understanding a program, the key is to understand the structure. That is why people put things in subdirectories, to help communicate the structure of the program, and show which things are closely related.

Do you really look at every single piece of shared memory? When you inherit a large codebase from someone? How do you reconcile that with having to deliver changes in the first weeks? Everywhere I've been, people half-ass these things, with inevitable bugs. I think you're under-estimating the possibility that you're just a better programmer than me :/

  > Do you really look at every single piece of shared memory?

Yes

  > When you inherit a large codebase from someone?

If the codebase has lots of threading errors, then yes, it's the only way. If it's a large program, it can take months to go through and check every piece of shared memory (and removing threads along the way, removing shared stuff); but the alternative could take years.

In one case, I moved every single lock/unlock to the top of a file so I could quickly see all the locks and unlocks, and that each lock had a matching unlock, even in error conditions.

How did you get into such issues? Were there classes where you got started, or was it all on the job?

I think I'm not concerned so much about inheriting a codebase with lots of threading errors. It's more about a codebase that's almost perfectly right, but where I'm scared to change anything because I don't have a big-picture model of the concurrency in my head..

Do you have any code samples I can try to learn from? For example, I'm not sure how you would move lock/unlock pairs to the top of a file from different functions/scopes. Unless you were doing some sort of literate programming as well?

  > How did you get into such issues? Were there classes where you got started, or was it all on the job?

I took the usual college classes dealing with concurrency (in my school it was in the OS class), but it took several years in the industry to really feel confident with threads (it took less time to feel confident with networking). I wrote down my knowledge (for what it's worth) in this book: http://www.amazon.com/dp/0996193308

  >  I'm not sure how you would move lock/unlock pairs to the top of a file from different functions/scopes.

That solution might not work in every case. It worked in the particular case I was referring to.

  > It's more about a codebase that's almost perfectly right, but where I'm scared to change anything because I don't have a big-picture model of the concurrency in my head..

Hmmmmm, that's an interesting question. Usually when the codebase is written, the person who wrote it had an idea in his head that, "this is how things will be locked to avoid problems." I try to figure out what that idea was.

Sometimes there is like a critical 'zone,' where a thread acquires the lock when it enters, and releases when it leaves. For example, it could acquire the lock when it enters a class method, and releases it when it leaves the class. Then the class becomes the critical zone.

Maybe learning to think of 'critical zones' is the most important skill to understanding the big picture?

Yeah, I'm familiar with the notion of critical section, and ideas like Ada's monitors, though my understanding is mostly theoretical.

I'm glad I kept the conversation going so I could find out about your book. Purchased.

I'm looking forward to seeing what interesting things you come up with in the future.

> Wart's infix operators have only one hard-coded precedence rule: operators without whitespace are evaluated before operators with whitespace.

> n * n-1 # does what you expect

Very good idea!

Reply on news.ycombinator.com