kyllo

Yes, so to me, the key question is not whether Swift can replace Python's role, but whether it can replace C++'s role, and thereby also making Python's role unnecessary and solving the two-language problem in the process.

I've found it generally best to push as much of that data prep work down to the database layer, as you possibly can. For small/medium datasets that usually means doing it in SQL, for larger data it may mean using Hadoop/Spark tools to scale horizontally.

I really try to take advantage of the database to avoid ever having to munge very large CSVs in pandas. So like 80-90% of my work is done in query languages in a database, the remaining 10-20% is in Python (or sometimes R) once my data is cooked down to a small enough size to easily fit in local RAM. If the data is still too big, I will just sample it.

I don't know that the ML community necessarily _needs_ a better language than Python for scripting ML model training. Python is decent for scripting, a lot of people are pretty happy with it. Model training scripts are pretty short anyway, so whatever language you write it in, it's just a few function calls. Most of the work is in cleaning and feature engineering the data up front.

Perhaps a more interesting question is whether the ML community needs a better language than C++ for _implementing_ ML packages. TensorFlow, PyTorch, CNTK, ONNX, all this stuff is implemented in C++ with Python bindings and wrappers. If there was a better language for implementing the learning routines, could it help narrow the divide between the software engineers who build the tools, and the data scientists who use them?

The author has a full-time job at Google and apparently wrote the book as a passion project, not for the money. The knowledge within is an incredible contribution to the public good.

Functional programming builds immutable semantics as an abstraction on top of mutable semantics. This abstraction layer includes internal optimizations that elide unnecessary copying, such as persistent data structures and "list fusion." When you're processing a list, the compiler can optimize away intermediate sublists. If you "copy" a list or sublist in Haskell, you don't actually get a copy, you get a new pointer to the same physical data, which is safe because it's immutable and garbage collected.

A key premise of functional programming is that the constraints of immutable semantics free the compiler up to do all sorts of performance optimizations, and the compiler can apply those optimizations much faster than a human developer can.

Also, when you really need a vector that you can mutate in place, you can still do that. Haskell has a mutable vector type. It's not the default idiom, but if you needed to sort a billion numbers in place, this is what you would use. Note all the function names prefixed with "unsafe": https://hackage.haskell.org/package/vector-0.12.1.2/docs/Dat...

Sure! I use pandoc to render Markdown, but you can use whatever Markdown processor you like in place of pandoc.

Makefile

    all: example.html

    example.html: example.md example.png
        pandoc $< -o $@

    example.png: example.dot
        dot -Tpng $< -o $@

example.dot

    digraph G {
        foo -> bar;
    }

example.md

    # Example

    ![example graph](example.png)

For simple graph diagrams it's easy enough to open up a separate graphviz .dot file, write up a graph specification, run it through dot or neato, and include the .png in your markdown file. And then automate this with a Makefile.

I've held many other white collar jobs in my career. Software engineering is different, the interviews are at least an order of magnitude harder.

Google knows and admits that their process has flaws and their HR team does internal studies on it. Just because Google does something, and Google is big, doesn't mean what they do is optimal.

The ability to acquire and apply technical knowledge, and ability to demonstrate random technical knowledge under time pressure, in a foreign environment, while other people are watching and judging you, are orthogonal skills and many engineers are great at the former and terrible at the latter.

There's also a ton of opinion and taste involved in software engineering, which can bias the interviewer's view of someone's technical ability. Not to mention all the unconscious biases against people who don't "fit the description" of a typical software engineer demographically.

The Googles of the world limit their own potential by rejecting many talented people who couldn't make it through their hiring filter for reasons that have nothing to do with their actual ability to build software. Then they express angst over how difficult it is to fill positions. It's a self-inflicted problem, and a hard one to solve. It's probably impossible to design a theoretically perfectly fair hiring process without also solving all causes of inequality in society generally. But it can be made better and more fair than it is, and many companies and applicants aren't satisfied with the existing hiring processes, so they are making efforts to change the situation, as they should.

I'm an R folk, and I'm even happy doing all of that in Emacs!

Ha! No, Microsoft--I was in the hardware supply chain group for a while, and The Goal was a common conversation topic there. I heard a few people call process bottlenecks "Herbies."

I also wonder if we'll see a greater push toward remote-first work, given the high carbon costs of physical facilities.

I wonder if there may also be a push to reduce some compute-intensive workloads, like training/scoring ML models or refreshing analytics dashboards that are not mission-critical and maybe not even actively being used.

The point is that within the boarding process, the slower (elderly, disabled, traveling with children etc.) people are the bottleneck, and letting them board first actually makes the boarding process take less time. The pilot can't even start waiting to get permission to fly until after everyone's boarded.

This reminds me of the book The Goal by Eli Goldratt. It's fictional, but there's a story about a boy scout troop that goes on a hike, and there's an overweight kid named Herbie who hikes the slowest. The troop leader teaches everyone that the entire group as a whole will reach the top faster (and stay together) if they keep Herbie in the front.

When you have a bottleneck step in your process, the whole end-to-end process will run the most smoothly if everything else matches pace with the bottleneck.

You get what you measure, for better or for worse.

But what tends to happen with stack ranking is that people only do what their direct manager values and rewards in the short term, and avoid other work that really needs to be done, or what the actual customer wants. Your boss becomes your only customer. You invent redundant new shit because that's more impressive than fixing your existing shit that's broken. This can be incredibly damaging to the quality of the product, and also to the careers of anyone the boss just doesn't personally like for whatever reason. It creates a monoculture of like-minded, demographically similar people who suck up and shit down.

You even can see pathological behaviors like managers purposefully recruiting low performers as sacrificial lambs to offer up at the next review time.

Companies got rid of the rank-and-yank system in the last decade or so for a very good reason.

It may be against Visa's ToS, but it's still a very common practice, especially among small businesses, probably because of enforcement challenges.

One is data analysis. I often have data in two or three different datastores that I need to analyze together (a column-oriented DB, one or more row-oriented DBs, a key-value store, a document store, a Hadoop/Spark cluster) and it's nice to be able to use the same code to open and manage connections to any one of them. My queries are and always will be vendor-specific, but my connection handling is all just generic ODBC code.

It's also great for WYSIWYG BI tools like Tableau or Power BI, which can connect to and fetch results from anything that has an ODBC driver. Without that, the BI tool vendor would need to write their own driver specific to each datastore on the market. So if anyone ever wants to build a generic BI/analytics tool product in Rust, this is a prerequisite.

It'd be interesting to see how std::map (implemented with a red-black tree) would do on these same benchmarks. A high ratio of inserts to lookups could favor the tree-based map.

I read it differently--this is not targeted at founders trying to validate a business plan, it is targeted at engineers whose job it is to build the product, and he's saying to quit fiddling around with your tools and focus on building the product--which is more fun anyway.

"It's important to learn how to use your tools well, but once you have, stop prioritising that. They're good enough now, you know how to use them. Don't get trapped in this local maximum of optimisation. Go and build stuff."

The constant variance part is important though. If you apply linear regression to a severely heteroscedastic dataset, the inferences you obtain from the model will be incorrect.

Yeah, it's not really new. A few years back, I interviewed at a startup that was close to two years in, and they described their app as "the most overloaded Drupal installation in existence," and were in the middle of migrating it to Rails.

For prototyping relatively simple, UI-focused apps, yes, I think low-code and no-code solutions do have a place.

The key is having the awareness and discipline, when the prototype is validated and starts to gain traction, to throw it away and build the real production app using what you learned, instead of just accumulating months to years of technical debt and cruft on top of a prototype. If you launch with a "permatype" you will eventually find yourself facing a very expensive and painful "2.0" project.

4-6 hours still seems like a long time for this, did you have sorted indices on the join columns? How high of cardinality (distinct counts) did the join columns have? Traditional RDBMS are pretty fast at joining when you have the right indices in place, especially when the cardinality of the join columns isn't very high.

And how much of that time was just getting the data into the tables? There are fast and slow ways to do that, too...

Linear regression doesn't require or assume the input data values to be normally distributed, just that the residuals are normally distributed and have constant variance.

The best thing about a highly expressive language is that you can do anything you want to.

The worst thing about a highly expressive language is that you can do anything you want to.

Expressive power vs. intelligibility (for the sake of common convention and communication in a team setting) is a real tradeoff, and the "principle of least power" does partially explain the popularity of slightly less-powerful languages like Java, Python, and Go, where there is generally just one idiomatic pattern for accomplishing a given task.

Yes, flexibly applying situational tactics to follow the principle and spirit of the strategy rather than the letter of the rulebook, is what leaders generally want their people to do. They communicate intent and let good judgement do the rest.

The way I have always thought of strategy, though, is that it should be directional. Say what input variables you think we should increase or decrease, in order to get closer to the optimum, relative to where we are currently.

Taking your example of cost vs. quality. Obviously we want to find an optimum. If a manager can state which direction we need to move in and why, to make progress toward the optimum, I will take their strategy seriously. But if they just tell their people to "optimize," I dismiss them as having no strategy. A manager who's responsible for formulating and communicating a strategy, has to at least have an opinion about which side of the curve we're currently on.

Early in my career I was an analyst at an ocean shipping company--lower rates meant fuller ships, higher rates meant better unit margin but lower utilization. Corporate's strategic message? "Fill up the ships with high-paying cargo." It became a joke among the employees.

Perhaps another problem with strategy is that we expect too much from it, because we neglect and devalue tactics. I often hear "strategic thinking" lauded and "tactical thinking" looked down upon by managerial types. But a strategy alone achieves nothing--you need good tactics to execute on it.

Better how? There are plenty of benchmarks showing that gcc's optimizations are more performant than llvm's.

Speaking of that, does anyone here have a ballpark number for what it would cost and what the minimum order quantity would be like, to get one of the major Taiwanese OEM companies to manufacture an initial run of a custom server product? My guess is that the capital requirement would be in the 10s of millions of dollars.

It's a cost model, they probably have a coefficient for what % of people they think will wait vs. go elsewhere while the website is down. When you're running a retail company, you have to at least attempt to quantify lost sales in order to model your business.

There's no way to measure lost sales with high accuracy, but that's not really the point, because it's most useful for modeling decisions and policies in a forward-looking context. Future revenue forecasts, the other term in the equation you're optimizing, are also inaccurate.

These functions are also generally "bathtub" shaped, and with the amount of money at stake, you can save quite a lot by being a little closer to the optimum, so they're worthwhile.