I spun up two Haskell teams at work, and now it composes about half of our codebases. Happy to answer questions about the experience.
263 karma · joined January 7, 2016
Providing lockfiles is a really interesting idea! That certainly solves the "we need your non-deterministic build tool to reproduce an exact build that we found" problem.
We haven't explored this route yet because a lot of our customers use tools that don't support lockfiles (e.g. Maven - Java in general has a lot of legacy stuff).
If you want to build off of our work, our dependency analysis bit is open source: https://github.com/fossas/fossa-cli
> We use the Gemfile to figure out gem group and pinned requirements (we run turn your Gemfile into a ruby AST since Gemfiles can be arbitrary ruby code [...]
Here's an example Gemfile:
if RUBY_VERSION < "3"
gem "minitest", ">= 5.15.0", "< 5.16"
else
gem "minitest", ">= 5.15.0"
end
This makes it impossible to statically determine the direct dependency requirements of a project.
> Now we have [your dependencies and their current versions], [target rails version], [all of the runtime dependency constraints of these gems]. We run this through a dependency resolution algorithm (pubgrub). If it resolves then you're good to upgrade to that version of Rails without changing anything. > > If this fails to resolve, it's because one or more of your current dependencies has a runtime restriction on rails (or another indirect gem being pulled in by the new rails version).
This actually isn't that big of a problem for Bundler, which uses pubgrub, which to my understanding is deterministic. A deterministic algorithm means you can actually take requirements and simulate builds. There are two places where I would be hesitant:
1. This determinism only works if you also know what the universe of possible dependencies looks like. In many corporate environments, this is not true! Many corporate environments use private registries that mirror public dependencies, and may have a set of dependencies available that looks different from the public registry, which means your simulated builds will resolve to incorrect dependency versions.
2. As you move to support other languages, many other tools use non-deterministic dependency resolution algorithms. In particular, NPM is famously non-deterministic, which makes it impossible to simulate an NPM build.
---
When you're trying to just determine that there exists a build that resolves properly, these issues aren't particularly painful. At $DAYJOB (and what I suspect you will want in the future), we are often trying to predict the exact build of a user given a new set of dependency requirements (e.g. so we can predict their new vulnerabilities), which means doing accurate build simulations is much more important.
FOSSA builds developer tools to help engineering teams manage their open source. We help enterprise customers discover legal (licensing and copyright) and security (vulnerabilities) risks in their dependencies, provide tooling for them to catch these issues in CI, and automate the tedium around policy enforcement and report generation. As companies adopt more open source, their engineering teams get bogged down by more distractions around compliance and security. We help automate away those distractions.
We build an open-source CLI tool (https://github.com/fossas/fossa-cli) that integrates with compilers and build systems to extract dependency and build information; a backend distributed system for analyzing dependency metadata; and a web application with a policy, reporting, and enforcement engine.
Tech we use includes:
- Languages: TypeScript, Go, Haskell, Rust
- Infra: Node, React, AWS, Postgres, Kubernetes
I'm happy to chat or answer questions at leo@fossa.com.
---
The concept you're asking about is called "effect tracking", and the argument for effect tracking is very similar to the argument for static types.
In a world without static types, functions can be passed values of any type, and can return values of any type. Adding static types makes this code easier to reason about because we can statically enforce (i.e. enforce without needing any runtime checks) that functions only take values of a certain type or return values of a certain type. This makes reasoning about the code easier because it limits the possible things that a function can take or return, so instead of thinking about "what happens if I provide any kind of value to this function?", you can think "I know this function takes an integer, so I only need to worry about understanding how it behaves when provided something that is an integer".
In a world without effect tracking (this is most programming languages today), functions can perform any side effect. Adding effect tracking makes this code easier to reason about because we can statically enforce which side effects a function does. For example, effect tracking makes it possible to express "this function takes a callback function as an argument, but that callback argument is not allowed to print anything". Alternatively, you can also look at a function and know for a fact that might do IO, or that it doesn't do IO. When you're debugging, this helps narrow down the scope of places that could be doing something wrong, because now you'll never have a function that _looks_ like it's doing something innocuous, but is secretly writing a log statement to a file, or pinging an API over the network, or drawing something to your GUI window.
---
Monads are a way to _implement_ effect tracking within an existing type system, rather than inventing a separate, orthogonal "effect tracking system". This allows us to reuse a lot of the existing tools and theories that we understand about types and type theory, and apply them to how we do effect tracking.
Exactly how this implementation works is another topic that gets a bit involved. But note: (1) there are ways to do effect tracking that are _not_ monads, or that do _not_ integrate with the type system that the language uses for values; and (2) monads have applications that are _not_ effect tracking.
It's also open source - feel free to make a pull request adding more nuanced controls.
My main complaint with Haskell is that it's actually a great language, wrapped up in (1) hilariously beginner-unfriendly syntax and tooling, and (2) a library ecosystem that does not quite understand the value of friendly documentation (although this is actually improving a lot). There are also some weird warts around the language itself, but those are mostly tolerable and not so much worse than the weird warts in all other languages.
My specific complaints about beginner-unfriendliness in syntax and tooling:
- The compiler will infer the types of names as generally as possible, and will run this inference both forwards ("this variable was assigned a string, so must be a string later") and backwards ("this variable is used as a string, and therefore must have been a string earlier") depending on what information is available for inference. This means that type errors are hit-or-miss; about 60% of the time, the type error will point you to your actual mistake, and the rest of the time it will point you to where the type inference algorithm got stuck in unification (this may not be the location you made your actual mistake, because the algorithm will try its best to infer concrete types working under the assumption that you didn't make a mistake, so it sometimes gets stuck trying to unify types at an earlier/later point than your actual mistake).
- The syntax is VERY minimal, and prefers minimalism over readability. In particular, Haskell has no explicit parentheses for function calls, and has implicit currying! This means that it is very easy to accidentally get the arity wrong in a function call (for example, `foo (bar baz) quux` and `foo bar baz quux` mean very different things!), and create an extraordinarily confusing type error (in the best case, an error about applying things to the wrong number of arguments; in the worst case, errors about constructing infinite types).
- This minimal syntax also means that you can sometimes type something that is _almost_ correct, but is actually valid syntax that means something totally different! Fortunately, your almost-correct expression will not type-check (so you won't ever mean something else without getting an error), but it's still a head-scratcher in the beginning to figure out exactly what the compiler is taking your code to mean. In some of these cases, you may even type something that is not syntactically valid vanilla Haskell, but _is_ valid Haskell-with-some-extension, and the compiler will unhelpfully suggest that you turn the extension on even though it may not be what you want (and also provides no explanation of the extension).
- The compiler tells you what expression has a type error, but never actually tells you the exact term that has a type error. You have to sort of guess at this by understanding the rough high-level of how the type inference works. It's something you have to really pick up, and is hilariously unfriendly compared to e.g. `rustc --explain`.
- Print-debugging works differently than what you'd expect. Nobody ever teaches you that `trace` is a thing (seriously, this should be thing #2 that is taught), and nobody ever teaches you "you're getting trace messages in a different order than you expect because Haskell is evaluating expressions with graph reduction, not in the order of the lines of code you wrote").
Really this boils down to: even though Haskell will do a very good job at preventing you from holding it wrong, you really do have to have a decent grasp on what you're doing to write a program. As a complete beginner, you can't just put something you think will mostly work and then rely on simple compiler errors to tell you where to spot fix (unlike e.g. Go, Java, etc.). Fortunately, this gets much better over time - you learn how to interpret compiler errors better as you get a grasp on the language, and you learn tricks for building incrementally correct programs up using `undefined` and friends.
My specific complaints about documentation:
- Contrary to this article, types actually ARE decent documentation in Haskell. The constraints you can do with types in a pure language are much, much stronger than in other languages you've used before. Trust me - this is a Blub paradox thing. It's just a qualitatively different kind of type safety.
- Unfortunately, although types do a great job at preventing me from plugging the pipes together wrong, they still don't tell me anything about the _semantics_ of a function. Lots of older libraries do a very bad job at documenting semantics, but this is getting a lot better.
- Arguments in Haskell functions are not named (or at least, their names are not exposed in documentation). This makes for some confusion sometimes - what am I supposed to pass as arguments to this `foo :: String -> String -> String` function?
- Lastly, a lot of libraries document very academically. Here's an example: https://hackage.haskell.org/package/logict-0.7.0.3/docs/Cont.... The documentation here is written assuming you understand how Logic computation works, provides no examples, and you're supposed to read the module documentation and actually go and _read the paper_ (in fairness, it's pretty short) that they've linked. This is a far cry from the NPM world (which, for all its faults, has really embraced accessibility in documentation), where everybody has a quick start.
Overall though, it's a really good language. Once you get monads (and a HUGE part of this is not the idea of monads, but the really unfriendly syntax and error messages of Haskell; IMO the _real_ lightbulb moment is grokking the difference between data, types, data constructors, type constructors, and typeclasses), a lot of things make a lot more sense. The testing story is amazing, the refactoring experience is amazing, the incremental development and REPL stories are great, and there are some truly astounding abstractions you can use that are just impossible to express in other languages. It's my regular side project driver, and my team maintains a Haskell codebase in production.
Lastly, a word of advice to anyone looking to actually make a good-faith evaluation of Haskell: try it, make something substantive with it, and stick with it until you actually grok it. Ask for help from the FP Slack! Seriously, don't just pick it up, get stumped by error messages, and give up. The tooling is very beginner-unfriendly, but fortunately the humans over at the Functional Programming Slack (https://fpchat-invite.herokuapp.com/) are VERY friendly. As a first project, I'd recommend doing the Advent of Code, or building a small web application (IMO with Scotty, which feels like Express, rather than Yesod, which feels like Rails. Yesod is almost certainly better in production, but has a different and IMO less instructive learning curve). If anyone needs help, follow along with my AoC 2020 (https://github.com/liftM/advent2020), although be aware that some things I do there are a bit fancier than necessary.
I find that articles like this that complain about things like type abstractness or the mathy concept names really are not representative of the actual pain points with Haskell, and are FUD more than useful. Yes, the tooling is beginner-unfriendly. Yes, you will probably need to ask some questions on the FP Slack on your path to learning, things that a friendlier language would have documented. But this mystical aura of "ooh, there are types and they are very abstract!" is not reflective of reality, is not a criticism grounded in the reality of writing Haskell, and IMO is misleading for people genuinely trying to assess the language.
1. It was easy to find people interested in startups at Google, but the quality of those interactions tended to be pretty low. The vast majority of people really did not have any experience with how startups actually functioned, and most people I talked to were more taken with the prestige and fantasy of being a founder than with actually seriously thinking about running a startup. I'm sure there are great and experienced operators at Google who are very competent and pragmatic startup folks - but those were generally not the people you found at internal events geared towards "people interested in startups".
2. I learned a lot cross-functionally when I joined my current team (which was < 5 people at the time). I was talking to customers in sales and support calls, helping to think about product and market strategy, and generally helping out on a variety of interesting things. You're definitely right that I wasn't talking to investors (at least, not on a regular basis - as an early employee, sometimes investors still want to chat to learn more about how things are on the ground) nor was I spending my days prospecting and finding BD partnerships, but I did learn a surprising amount about deal dynamics, selling B2B, and thinking about markets and customers in general. That said, this may have been because of the extraordinarily open and competent sales folks I worked with.
When I look on my list of "people I would co-found with", the vast majority of those people are the incredibly intense and competent folks I met working at the (now no longer early stage) startup I moved to. In general, I think about finding a co-founder less as "cast a wide net to find your one true partner" and more as "work alongside the most aggressively competent people you can find, and hopefully some of those people will mesh with you well enough that you would consider co-founding with them" (with the corollary that, if you find yourself in a job where you feel like you're mostly coasting, this is probably not a good environment for finding co-founders (although it certainly may be a good environment for you for other reasons)).
As usual, grain of salt - I was basically a new grad when I joined Google, and I expect the experience would be very different if I had a strong, senior network there. I also come from a position of unusual privilege - my parents are both engineers and I have lots of friends in engineering and I'm good at what I do, so I am unusually risk tolerant because "running out of money" or "being unable to easily find a job" is just not a thing I worry about.
1. Everybody is swamped. Sales is now scaling up, which means new customers, new demands, and new fires every day.
2. The product's complexity has now grown tremendously. No single person, not even founding engineers, can fit the entire system in their head now. Very few people are equipped to even dig in to the more esoteric pieces and understand how and why they work, because enough time has passed that any context that was undocumented is either forgotten or hidden deep in a random person's head, and very little was documented (not necessarily the wrong choice - the time saved on not documenting things likely contributed to the company moving quickly enough and surviving to this stage).
3. Hiring has been approximately solved, and occurs on an approximately predictable cadence. Your engineering team has a small but predictable stream of newcomers. Every year or so, the team doubles.
In this region, you have some interesting constraints to deal with:
1. Building new features is necessary to increase revenue, but each new feature has rapidly increasing marginal cost (due to overall system complexity). Therefore, we need more throughput.
2. Adding new engineers is necessary to increase throughput, but each additional engineer now provides rapidly diminishing marginal throughput.
There are many tactics for succeeding in this region. The general focus here is "increase marginal throughput per engineer". Some not-writing-code tactics include investing in solid onboarding, developing effective documentation at the system level, and narrowing focus on product initiatives (becoming more deliberate with "we will try experiments A, B, and C in this order" as opposed to "we will throw the kitchen sink at the problem and see what sticks").
From a "writing code" perspective, I think this is where craftsmanship really shines. Constructing abstractions that dramatically increase the productivity of each marginal engineer provides an enormous pay-off in this region. Of course, the correct engineering abstractions must also be coupled with the correct engineering team structure. The effects of Conway's Law in this region are felt very, very strongly.
Unfortunately, I think it is rather unlikely for someone to just be able to drop in to a company in this region and begin working on this kind of neat problem. I think the most likely ways to get to work on this are:
- Be there from early on (arguably, first 5 or 10 engineers). Having domain experience is extraordinarily helpful in understanding which systems will be force multipliers.
- Be very, very experienced. I can see a role for a senior staff engineer to be hired in at this point to help build these force multipliers. I think this person would need to have previous experience in companies of this size to correctly value domain experience and judge the right pieces to build.
- Join in this region, and work with on the team of one of the two people above.
Sometimes it takes a minute or two to load artists - Spotify's API rate-limits pretty aggressively.
This came out of a funny dinner conversation with some friends talking about discussing favorite musicians. Someone proposed that the musicians most likely to be spark interesting conversation were those that were unpopular enough to be "cool", but popular enough that someone else could have feasibly heard about them, and suggested that musicians with ~1M monthly listener count were the sweet spot. (Empirically, I think that's about 5-10x too high.)
Anyway, this tool should help you figure out which musicians make for interesting conversation. If you'd like to compare against me, my friend code is "uIrtZpcNPUZpikRlAVPg".
The reality is that humans are hard to secure, so defense in depth generally involves preventing compromised accounts from causing lots of damage, detecting them as early as possible, and controls for shutting them down.
Arch is like the CI/CD analogue of operating systems. Small upgrades makes breakages much less common, and much easier to repair. This comes as a trade-off of less battle-tested stability than e.g. Debian, but if you're able to make small repairs proficiently then Arch is the way to go. I'd strongly recommend Arch on a personal computer - it makes a good OS for a pet (although Debian, because versioned releases are much more thoroughly tested, makes for much better cattle OS).
At FOSSA, we've commissioned a new license called the Commons Clause (http://commonsclause.com/) that tries to address this exact problem, so that developers can create permissive software without being exploited by providers.
In an ideal world, the compiler would always know to do this for us. In practice, it's very difficult (in most languages) to determine whether this optimisation is correct.
In the README example's case, the optimisation happens to be correct because both mapping functions are pure. If they had side effects, then stream fusion might create different results than the original code. Let's make this concrete with an example.
Say our array is `[1, 2]` and our functions are `function A(n) { print("A: "+n) }` and `function B(n) { print("B: "+n) }`. Then _without_ stream fusion, we would see the following output: ``` A: 1 A: 2 B: 1 B: 2 ```
_With_ stream fusion, we would see different output: ``` A: 1 B: 1 A: 2 B: 2 ```
Since the mapping functions have side effects, stream fusion is not valid here.
Some languages can do automatic implicit stream fusion. Haskell is a great example of this: it's able to determine when stream fusion is correct because the language has an _effect tracking system_ where functions with side effects are distinct from functions without side effects.
Disclaimer: I work for FOSSA.
One interesting thing I've noticed about these kernels is their tendency for full-stack language integration a la Lisp machines and Lisp OSes like Genera. For example, Mirage integrates strongly with programs written in OCaML (the home page describes the project as a "library operating system") and HaLVM with Haskell. A quick search shows unikernels for Golang (Clive) and JS (Runtime.js) as well.