Once I did it with a large RDF (XML) that was a list of books. Initially I loaded and processed it in Perl with XML::Simple, but XML::Twig let me deal with things one record at a time.
The other time I had a large Java linked list holding objects, where I grabbed the head, processed it, and then shrunk the list (removeFirst()) as I proceeded (I probably could have optimized the loading even more).
As a one-person project, punting on the big load optimization worked for me, as I had a working program even before optimizing. If it was a larger project, what time would be best to optimize might have differed.
I can see MIT/permissive licenses being less attractive than usual in an environment like that, because the author might perceive reduced chance of reaping rewards (versus aiding competitors) from publishing components.
If improvements led to greater-than-zero reward for publishers and consumers alike, though (as enforced by copyleft approaches), then perhaps the results would be different. All conjecture, really.