Clojure Don’ts: Lazy Effects (2015)
stuartsierra.com
stuartsierra.com
In my old team we had a semi-serious production bug result from this (it didn't actually kill anyone but stopped the system from being used) because we had an expression with a side effect inside a debug logging statement. The code worked perfectly in a debug build, but in release the DEBUG_PRINTF() equivalent was removed by the C++ preprocessor (so logically equivalent to "!debug && " in front of every line) and a crucial value didn't get updated. It took some tense minutes debugging it on site in front of the client before we figured it out... lesson learned!
The way I think about it is that it's not so much that your program executes a lazy stream of side effects, it's that your program doesn't execute side effects at all. Instead, your program returns a pure structure which describes how side effects should be executed, and the runtime executes a program based on this description. By the time the runtime is executing that program, the pure structure which describes how the side effects should be executed has already been forced, so the side effects happen eagerly within that context.
That said, I'm not much of a Haskell-er, so I'd be interested to hear from more experienced Haskell programmers on whether there's a better way to think about this.
I didn't actually say this. I was suggesting a lazy chain of IO actions for such problems. For example, ListT a IO, instead of a regular lazy list with hidden side-effects. The Haskell types help make the distinction clear.
> Instead, your program returns a pure structure which describes how side effects should be executed
Yes exactly, the pure structure can still be computed lazily on demand, providing IO actions.
My favourite Hackage streaming library for Haskell is "streaming" and it can work exactly this way.
Also, by it's nature ListT can't stream IO unless you use lazy IO aka lazyRead aka unsafeInterleaveIO which deserves its name.
newtype ListT m a = ListT (m (Maybe (a, ListT m a)))
It is a nice simple example of an effectful streaming type.
> Also, by it's nature ListT can't stream IO
This is only true of the deprecated one, which is also misnamed. The community is rightly re-using the name for the behaviour shown above. See list-t on hackage, or the implementation that comes with Pipes.
For example, this code will fail because the file handle is closed prematurely by withFile as hGetContents returns immediately without actually reading the file, deferring that part until the data is forced.
main = do
contents <- withFile "foo.txt" ReadMode hGetContents
putStrLn $ length contentsNot unlike the Streams/Collectors framework in Java.
That is, the only reason what you said is accurate, is because you said "effectful stream of bytes" meaning "eagerly effectful." Right? If you do all of that lazily, you can just as easily shoot yourself in the foot here.
(Please nobody take this as a damning criticism of Haskell. I'm not intending it that way.)
Returning the stream itself from the "withFile" and trying to run it later still is still a bug.
I like this approach better that conventional Haskell lazy I/O, which I personally find quite confusing.
You could argue that libraries shouldn't contain unsafe code either and go with a substructural type system, use indexed monads, fake regions or something similar. Whether the extra complexity is worth it is arguable, though.
We had a numeric counter that was enumerating the frame number of a simulation. When we removed the debug statement, we didn’t realize that no other code ever evaluated the counter - so instead of resolving each X+1 into an integer, Haskell kept it as a ever-growing nested series of unevaluated thunks: 1+1+1+1+1+1+1+1+1+1+... , and our efficient little program started filling up all its available RAM and crashing.
A good rule of thumb is to by default mark fields for basic types like Int, Bool, etc. as strictly evaluated, and leave larger structures (trees, lists, etc) lazy. But you still need to be careful.
The compiler trying to silently fix things is probably a bad idea. The current behavior is at least easy to understand; I'd hate to have a program that's working because the compiler could figure out that it could make something strict, and then I bump something mostly unrelatrd such that the optimizer can't be sure anymore, so I get a space leak.
Also, unintended evaluation can potentially cause high memory use as well (e.g. [1..1000000]), so the compiler also has to be careful about introducing excessive memory use.
The compiler does do some strictness analysis, but it's a hard problem.
I like the way idris does things -- strict by default, laziness controlled by the type system, and some nice support for automatic coercions.
If anything, eager evaluation will reveal the bug faster.
It's often not clear what is lazy and what isn't - concat is lazy for instance. It isn't externally obvious which of the very common operations are lazy, filter being lazy has bitten me in CLJS a few times too.
Adding to the source of the confusion is that the REPL will often realize your side-effects which then won't get evaluated in your runtime environment.
One of the errors I used to make earlier on was:
(map insert-into-db table-rows) foo >>= \_ -> bar
What makes this do the right thing for effects is what values of type IO actually are, and what bind means specifically for IO. It's helpful to think of an IO value as code in another (imperative) language. Bind takes two fragments of code in that other language, and stitches them together into a script that executes one after the other.
The key thing is that the order in which you compute parts of the script is entirely orthogonal to the order the commands appear in the script.
In Haskell, evaluation does not cause side effects, period. `main` is a value of type IO, which is executed when the program is run. The effects of that execution are independent of how the value `main` is computed.
Obviously, the computation itself takes time and space though.
This made me waste a whole afternoon once. I was even ready to submit a compiler bug!
I’d say it’s a good rule of thumb, but it’s sometimes justified. For example, `line-seq` returns a lazy seq of lines read from a given Reader; the appeal being you can process them one by one, without keeping them in memory all at once. But if you just want them all, you wrap the `line-seq` in a `doall` in a `with-open`.
My scraping library, Skyscraper [1], has a similar justification for laziness around side-effects: scraping a site returns a lazy sequence of elements, each corresponding to one page. It's terrifically useful to have that sequence be lazy, and there's unchunking code on Skyscraper to enforce full laziness. Incidentally, I'm rewriting it to be based on core.async, but it has a less functional feel to it.
Clojure provides doseq and run! for side effects on collections, and both return nil. One might get the impression that these design choices are intended to discourage the programmer from complecting transformations of sequences with performing side effects on their elements.
Most of the time, you can replace one with another, such as pmap (lazy, parallel) and have the program behave the same way. Using mapv for side effects breaks this assumption.