bobbruno

Hypothetically, if I wanted to learn how to code by studying open source examples on GitHub, should I have to go ask permission of each rightsholder to learn from their code? I agree that, if Copilot is based on a model that overfits to output the exact same code it read, the lawsuit has merit (and Copilot is not really ML), but the idea of ML is that the model doesn’t memorize specific answers, it learns internal rules and higher-level representations that can output a valid result when given some input. Very much like me, the coder, would output valid code when given a use case description, after studying a lot of open source examples of that. Should most programmers just be paying rights to all publishers of code they have studied?

If they can actually link output to specific input, the lawsuit has merit and more, GPT-3 is a lie. A neural network is supposed to learn how things work, not memorize a large number of examples and spit them out verbatim - or keep connections to specific inputs.

Copilot losing the lawsuit is evidence it’s a case of overfitting, not true ML.

Let me see if I can at least justify some of these things:

- SQL became the de facto standard for data manipulation a few decades ago, and I’m still to see a worthy contender. While the language itself shows its age, the concept of it being (almost) totally declarative of “what” data to get and “what” to do with it, instead of “how”, is unbeatable. A better syntax for achieving this is certainly possible, but you won’t beat decades of SQL code being generated easily. About “get the answer right once”, for analysis it’s often all that’s needed, and good SW maintence doesn’t apply that hard (it just won’t be needed again). For actual repeatable data engineering, I’d suggest either using generated SQL or writing in some abstraction (like PySpark). But don’t underestimate the productivity of SQL and the ability of modern parsers;

- Unit testing: while I generally agree, after 28 years, I have to say that unit testing is not as useful in data engineering. Essentially, it doesn’t matter how much time and effort you put on designing test cases, you’ll never beat the chaotic creativity of real users making real systems ingest all kinds of wrong data in patterns you’d never imagine. Now multiply that by having to cross-reference data from systems that were designed independently, and think of all the possible data errors that might come out of that. You won’t, you won’t cover even 20% of the ways things can go wrong. So, instead of “testing for all ways things go wrong”, a much better pattern is “write code that can capture unexpected errors and make reasonable decisions on whether to continue or abort - with some nice messaging and data debugging, please”;

- Poor support for version control: I 100% agree with you. DataOps is still not nearly as popular as it should be;

- Frameworks over libraries: let me tell you one thing - data is coupled. Uncoupled data has less value than coupled data, because it can’t answer more complex questions. Having different libraries that represent, manipulate and expose data in different ways will just make things much harder for the DE trying to generate the data. And the focus is handling the data, not the code. So, while you may have a bit of a point, I’d suggest that you’re looking at this from the wrong perspective;

Low-code: it’s one of those things that come and go. When I started, I’d use Oracle’s PL/SQL and C. Then came the first generation of visual ETL tools (Informatica, Datastage, etc) and they looked great with the graphic flows. Until you actually had to use them and all the clicking just to get to a point where you could actually add some logic showed. Then came the Hadoop era, and data engineering split into the DW people (who prefer visual, low-code to this day) and the Hadoop people, who went back to coding. This split stays to this day, with each side not recognising what the other does as essentially the same thing. Personally, I prefer code, but I don’t care what others prefer. The visual x coding discussion is, in my opinion, the wrong problem. The real issue is being able to create logic that’s clear enough that something as dumb as a computer will unambiguously understand. STEM learning environments for kids show that this is totally possible in a visual way - the thing is, most companies choose the visual tool in hopes that a person without the right mindset will generate good logic with it. That’s what fails. The rest is a matter of preferred UX - relevant, but not core or one-sided.

I’ve been working in this field for 28 years, and it can be quite confusing for a newcomer. Let me see if I can help.

First, the name Data Warehouse is overloaded. It can refer to the entire architecture (often called “Data Warehousing architecture”) or the central data store that (ideally) contains all the corporate data in a format suitable to generate datasets/schemas for specific needs (the data marts, if that need is BI/reporting/dashboards/data slicing and dicing). The other common component of a DW architecture is the staging area, where you land data from the source systems. So, datawise, a DW architecture has 3 layers:

- Staging: where you land your data from the sources. You may have some temporary storage (some choose to just keep everything, just in case) of data, and a few control tables, but it will usually look very much like the databases of each source;

- Data Warehouse: Bill Inmon defines it as “a subject-oriented, integrated, time-variant and non-volatile collection of data in support of managemnt’s decision making process”. Essentially, reconcile as much as you can of the distinct domains of each source system (ideally, full reconciliation, but it’s almost impossible to get to 100%), keep as much history as you can afford and then apply (a little or a lot, depending on your preferred data modelling approach) denormalization to make it able to support BI tools. If you denormalize a lot, you may end up not needing data marts, but you’ll be applying area-specific needs to what should be a common basis for all areas. Common data modelling approaches for a Data Warehouse are normalised (3NF), detailed star schemas (search about Ralph Kimball if you don’t know) or, a bit more recently, Data Vault;

- Data Mart: an analytical database to support a specific business need - sales reporting, Marketing campaigns, financial reporting, Employee satisfaction, all these applications could have their own data marts. Each data mart should be fed from the Data Warehouse, applying heavy denormalization and calculating specific metrics and reporting structures (i.e., dimensions) to meet that use case’s needs. The most popular data modelling technique for data marts would be dimensional modelling (AKA, star schemas). The names come from the use of two main types of tables - facts and dimensions. A dimension could be a business entity, like departments, customers, products, employees, etc; a fact is a metric (like sales volume, sales $$$, satisfaction, stock) over time and a number of the aforementioned dimensions. When darn, the fact is in the middle and the dimensions around, looking a bit like a star, hence the name.

Analytical processing is nothing more than wading through this data searching for answers to business questions, interesting patterns, outliers or just confirmation that everything is as it should be. It usually means applying filters on the dimension tables, joining with the fact tables, aggregating over hierarchies and possibly time and watching how the resulting time series of metrics behave - the basic job of a data analyst. BI tools expect data to be structured as stars, and will help the analyst do this slicing and dicing over the data, generating the required database queries, possibly doing some part of the computation and then the visualisation rendering. The names “multidimensional” and “hypercube” come from the data being structured across several “dimensions” as I explained above. Some BI tools will even have its own compute/data storage engine, optimised for handling this kind of data. Usually this is called an “OLAP engine” or a “multidimensional database”. It’s a database-like functionality, optimised for filtering, aggregating and generally wading through a large chunk of data. When loaded into a specialised database like this, a “star” is usually referred to as a “cube” or “hypercube”.

And finally, about the “online” mention. All the analysis above is supposed to be ad hoc, interactive and close to instant, so that the analyst’s train of thought (think of it as each question generating additional questions to the data) is not interrupted. The term “online analytical processing” (OLAP) was coined to refer to this, in contrast to the pre-existing term “online transactional processing” (OLTP), which is what most modern systems do - send a large number of small transactions to be processed by a database online (as opposed to batch). OLAP sends a moderate number of very complex queries to be processed by the database online (as opposed to batch).

I hope that made it clearer.

Very thorough answer on the DW and Data Lake points. On the Lakehouse, though, as I wrote above, it’s not just the data storage format (Delta, Iceberg, whatever). The storage is important, because it creates the possibility of using cloud storage as a basis for table-like structures, with ACID (or a subset of it) capabilities. Beyond that, you still need management, performance, security, SQL access and probably some more that I forget.

The goal for a Lakehouse is to create a platform/stack that can effectively do everything a DW does, everything a Data Lake does (so, all your pros above), get rid of most of the cons and also allow mixing and matching of the functionalities. Acknowledging this need to handle all kinds of data is not new; having the technology to deliver on it is, and that’s what the Lakehouse is.

Disclaimer: I work for Databricks.

With that out of the way, the Data Lakehouse name has a Marketing motivation, but the concept is “let’s not have a DW and a Lake as separate structures based on different technical stacks - let’s have one stack that can do both”. That is a very valid proposition, because the information needs tend to span what a DW or a Lake can easily do by themselves. Also, having one common stack makes it much easier to manage all data and generally be productive.

If you look back even at Inmon’s Corporate Information Factory, you’ll see mentions to unstructured data coming in, and the expectation that all data, structure or unstructured, should become useful information for some business purpose. This was back in the 90s, the need was already there, but the technology was not.

The Lakehouse as proposed by Databricks is essentially a combination of technologies (including but not limited to Delta Lake format) to deliver that unified design. It’s a platform where you can process all kinds of data, apply structuring where needed, manage quality, do DataOps and generally deliver information, regardless of the structure or lack of it in the original data.

So, to put it simply, a Lakehouse is a platform (or technical stack, if you want to build your own) where you can implement a DW, a Data Lake and mix capabilities of both to deliver data products more efficiently. It’s more than just the union of DW and Lake, because the combined capabilities allow you to do things that neither could by itself. For instance, parse a large set of documents for sentiment and context to generate metrics on some business topic. Environment, Sustainability and Governance - ESG - is one example area where the results are BI-like metrics for reports and dashboards, but the sources are usually text documents. Social media processing for Marketing purposes is another.

Did you mind checking the availability of people with lisp skills vs most other languages used in research and DS? Or the availability of libraries and pace of evolution? If your solution is to teach lisp massively and have everyone just build their tooling, can you explain why it hasn't happened yet, despite lisp being around much longer than most languages in use in this field - and many attempts at what you're suggesting?

Have you considered that the notebook is an evolution of a repl, with improved visualization and feedback, for for analysis-heavy work? The problem starts when notebooks are used for development and production.

You're thinking it the wrong way. Notebooks don't do well in software development, but they are extremely useful on exploratory data analysis and quick iteration when searching for a suitable modeling approach. These two tasks use code, but for completely different purposes. A DS is working on the data, understanding it and trying to identify what information it may have. Then they try to find a model that will leverage that information to deliver whatever inference solves the business need. This is extremely interactive and iterative, and everything from the actual business problem to the ML approach may change at each iteration. Imposing software development practices at this point is disruptive to the train of thought, which is very burdened already by the level of uncertainty and all the mathematics required to understand the data results. The goal is to find a viable approach, not write production code.

Once this approach is found, a good clean-up/refactor is strongly recommended, to then start a proper software development that will create a live product from the found approach. I call this the switch between research mode and development mode, and it has strong parallels to the way R&D is done in many industries. I believe a lack of understanding of this dual nature of ML is what causes many of the problems in MLOps: plans that don't take into account the research time and risk, mixed teams where engineers don't understand the initial nature of DS work, attempts to put notebooks containing research code in production, etc. Even planning for the refactor doesn't solve it all - what will happen when the next generation of a model has to be created? Will the refactor Ed code be forced on the DS and ruin their research productivity? Will they start from scratch again and not only lose all the refactor/dev cost but also make this a recurring cost? I have been looking for answers for this for years now, and found none so far.

Source: I've been working with data for 27 years, as a data engineer, data architect and data scientist. When I do DE, my code is considered high quality by my peers, but when I'm doing DS research, I know I write bad code - and I won't change that. It's more productive to work this way and do the big refactor (possibly leaving the notebook env behind along the way) than the alternative.