Hacker News Reader

Here's one study showing that complexity grows exponentially with the size of the program:

http://www.cc.gatech.edu/sparc/Resources/readings/berry.pdf

Here's a study showing addressing changed requirements or ﬁxing program defects requires a program maintenance effort that is directly proportional to the size of a program:

http://dl.acm.org/citation.cfm?id=358228

Thanks, I'll take a look. This point from the abstract of the latter is highly reminiscent of the paper cited by the OP:

"Repair maintenance is more highly correlated with the number of lines of source code in the program than it is to software science metrics."

Edit: I read it. This must certainly be one of the studies people are referring to; it covers exactly the question of interest. The major limitation is that all the programs were in PL/I, so it says nothing about language-independence. The important finding is that line count was the most highly correlated variable with bug count of those studied, quite a bit more so than more complex metrics were (Halstead's E). It's also interesting that although the authors write, "There are some very large programs in this set," the largest program was in fact only a very modest (by our standards) 6572 lines of PL/I.

In the Computer Language Benchmarks Game, they compare GZip'ed code size. I disagree with removing comments, since comments tend to be important, and comments generally document weirdness and bugs inherent in the platform/library/language.

http://shootout.alioth.debian.org/help.php#gzbytes

Les Hatton has some of what you are looking for. See "Re-examining the fault density - component size connection":

http://www.leshatton.org/IEEE_Soft_97b.html

...and some of his other work:

http://www.leshatton.org/index_CS.html

Thanks. I'll take a look at those too.

> So, HN: what are these studies?

If you have access to the publication libraries of the ACM and IEEE, you'll find that they publish most of this literature.

(It costs money for both, unfortunately).

> My second question is about how to measure code size. PG said a few years ago: why not just count tokens?

Some schemes do so, it depends on how you define "SLOC". The classic problem is if-thens.

How many lines is this?

    if ( foo ) then bar else baz

Or this?

    if ( foo )
    then bar
    else baz

Or this?

    if ( foo ) then
      bar
    else
      baz

In the literature you'll see a distinction between "physical" lines of code and "logical" lines of code. The latter is close to token-based.

> So: what's wrong with token count as a measurement of code size?

I vaguely recall that some properties correlate with physical lines and others with logical. I can't recall what and which, sorry.

If you have access to the publication libraries of the ACM and IEEE, you'll find that they publish most of this literature.

I know where to find research literature. I'm asking for specific citations. Is the claim an urban legend? If "studies show" X, one ought to be able to point to the studies.

How many lines is this?

All three of your examples have the same number of tokens, so to judge by them alone, token count is not just a good measurement of code size, it's a perfect one. My question is what's wrong with it.

I don't see how "logical lines", whatever that is, can possibly be simpler than counting tokens. In fact I don't see how anything can be simpler than counting tokens, since it's easy to know what it means, a tokenizer is always available, and everything irrelevant to the program is by definition dropped.

> I'm asking for specific citations.

None to hand right now.

> I don't see how "logical lines", whatever that is, can possibly be simpler than counting tokens.

I wasn't rejecting token counts per se. I think that it's a useful metric too.

What I was trying to convey is that "logical lines" is the term used in the literature. Logical lines can cover token counts if you define 1 token = 1 logical line. Or it might not. Either way, you have to settle on a definition.

One way that I could see where actual lines is a more useful measurement is if there is any correlation between what percentage of the code you can see on-screen at any time and bugs.

Not saying that there is such a correlation, just that there may be cases where it is useful to measure by line count.

  I don't see how "logical lines", whatever that is

At least by the definition I'm accustomed with, a logical line is a statement or series of statements which directly and logically belong together, for example a function call or an arithmetic operation.

Logical lines are independent from physical lines as each logical line can be split over multiple physical lines (eg splitting up a long string IO operation), or one physical line can contain multiple logical lines (this is a bad idea in most cases though).

Since the definition hinges a bit on what somebody considers as "logically belonging together", the whole concept is a bit fuzzy. Consider this string formatting operation (Python):

somestring.split(somechar)[-1]).replace("foo", "bar")

Do you consider this one logical line? Or would your logical lines look more like this:

somestring = somestring.split(somechar) somestring = somestring[-1] somestring = somestring.replace("foo", "bar")

Both are valid interpretations of logical lines, but they are visually and conceptually quite different, which makes it - imho - a bit problematic to try and use them as a measure of code quality.

You make sense, but the concept itself seems so fuzzy and hard to nail down that I marvel at how it ever arose, given that there already exists an unambiguous and ubiquitous way to distill the logical structure of code free of textual artifacts.

I think token count hasn't gained traction because of languages with syntax and types. "Surely all the boilerplate in defining a class doesn't make it more complex," goes the argument.

I prefer to turn the debate around on its head. Rather than argue complexity metrics (boring) I say I prefer languages without boilerplate because they make complexity harder to camouflage.

One can argue that boilerplate doesn't add to complexity (though I don't agree). But no one can argue that it doesn't add to code size. The studies cited in this thread show either linear or superlinear growth of bug count with code size. If those studies are correct, doesn't that rather settle the issue?

Yep.

A common LOC metric for languages in the C family is to count semicolons ("real" lines of code).

I used to use that all the time. It's great because it's quick, and a pretty damn good sloppy metric. I remember joking that the extra semicolons in for loops and the lack of semicolons in if/while statements balance each other out. People tend not to use semicolons in comments, either, except in commented-out code (which some of us have a pet peeve against anyway).

It's very easy to correct for both for loops and if/while.

Yes, but then the metric is no longer so charmingly trivial.

I actually think raw byte-count is a pretty good metric. Documentation size and variable name length are also symptoms of complexity. Specifically I think wc -c $(find -type f) (how many total bytes) and find -type f | wc -c (how many files and directories) make good metrics. Obviously you need to filter out data files and such.

This has some problems (e.g. spaces vs. tabs, utf8, etc.), but all of these size metrics will be pretty loose.

One metric I've seen is gzip-compressed size, which has the nice property that it identifies the size of the incompressible elements -- ie it discounts repetitive boilerplate.

Another interesting set of metrics is Halstead's "software science" metrics[1]. They fell out of favour because initially they were hard to count and didn't seem to correlate with anything else.

[1] http://en.wikipedia.org/wiki/Halstead_complexity_measures

But repetitive boilerplate is exactly the last thing that should get away scot-free in a measurement of code size.

It depends on what you want to know. Pure physical lines is one thing, "size" is a another.

I want a way to measure how complicated a program is that's independent of language and obviously extraneous things like line length.

You may find that the Halstead metrics I mentioned are closer to what you're after.

I've changed my mind. I'm interested in what I originally said: what's the best way to measure code size, and what are those studies (if they exist). Otherwise we get into debates about size vs. complexity, which is actually less interesting IMO. Size as a proxy for complexity is good enough for me.

I never understood the gzip one. Repetitive boilerplate is bad; why hide it?

You're trying to understand the "true" size of the software in spite of the idiosyncrasies of a given language.

As I noted somewhere above, "size" is an abstract, dimensionless quality. It can only be approached through proxies. The more the merrier, I reckon, especially if they turn out to correlate with different things.

In the case of most projects, copy/paste code is not just because of the language. It's because of lousy programmers. I've seen large codebases which are made up of a full 40% duplicate code. There's no way to blame that on the language.

You're missing the point of the parent: There is no 'one true' metric. If you use different metrics (actual lines, logical lines, gzip'd size, etc) you may well find different correlations.

But this penalizes programmers who like to use long readable names. I'm not one of them (though I used to be), but they have a strong case here.

Take any program. Replace all the names with the smallest possible character sequences. Have you made the program simpler? Or smaller in any meaningful way? Surely not. I'd say what you've done is left its logical structure precisely intact (another way of saying that token count is a good metric) while reducing its readability.

This metric relies on the assumption that people are trying to produce readable code. IMHO long variable names are much more helpful in complex codes than simple ones.

Ok, but now I'm wondering if we have opposite views of code size. In my view, code size is bad bad bad. More code means more complexity. Any time you add code, you're subtracting value; it's just that (if it's good code) you're adding more value than you're subtracting. So a higher score in a code size metric is a bad thing to aspire to, and we should greatly favor approaches to writing software that -- all other things being equal -- lead to smaller programs. I don't think that programmers who use long names for readability should have their programs discounted as longer (and thus more complex). Just because their names are longer doesn't mean their programs are.

No no no. My logic is this: Take tight, readable code with short names a replace them with long names, and you'll have worse code. The converse isn't true because complex (bad) codes are more readable with long variable names.

Complexity -> Code Size Code Size -> Long Variable names (win for big codes) Complexity is bad

Therefore long variable names are a symptom of a problem, but not the problem themselves. Long variable names aren't bad, but they are still a good predictor of badness. Since size metrics are meant to predict badness, long identifiers should increase size metrics.

Oh, I see. You sound like an APLer. We have similar tastes, but many good programmers disagree, so I doubt that long variable names are a predictor of program badness. Not every long name is FactoryManagerFactoryManagerFactory.

Consider a language like K, in which variables usually have one-letter names. The real code-size win for K is not that. It's that the language is so powerful that complex things can be expressed in remarkably compact strings of operators and operands. (Short variable names, I'd argue, are an epiphenomenon. It's because the programs are so small that you don't need anything longer, and longer names would drown out the logical structure of the program and make it harder to read.) Token count is a good metric here. Both line count and byte count come out artificially low, but token count can't.

I came back to say I've thought about your argument a couple more times and I think you're on to something there. The idea that long variable names, even when they add to readability, are a secondary indicator of code badness (because the code is too complex not to be able to get away with short names) is a subtle and interesting way to frame the problem. I'm surprised it didn't get more pushback from the 95+% of programmers who take the opposing view. I suppose this little corner of the thread is a quiet enough backwater that nobody noticed.

But I still don't see how you get around the objection that, according to your preferred metric, if you replace all the names with arbitrarily small character sequences, you get significantly smaller code - yet clearly not better code.

Also reduced its maintainability.

"I've heard it said for years that studies show the number of bugs grows roughly linearly with code size and that this holds true across any programming language."

I have absolutely no reason to doubt this, but I suspect this does not look deeply enough at the process.

Bugs, normally, get fixed, especially towards the end of a project, and it is much easier to eradicate bugs, and verify eradication, in a small project, than it is in a large one.

Poor bug-fixing, in the late stages of a large, buggy project, may well introduce further bugs, as well as discovering latent bugs, masked by the original ones.

The Mythical Man-Month

That's a marvelous book-length essay, but not a formal study. Or does Brooks cite research on this?