1,444 karma · joined August 17, 2010
hnchat.com:N1jRS58LGuGVd6IotNBO
That said, details will matter in the longer run. The markdown and code rendering on github has gotten such a large amount of attention over the years, that it's hard to compete.
Pachyderm is on my TODO list for a while, so thanks for reminding me, I'll try to implement something real with it soon.
One thing that I missed a bit was automatic task output naming based on the parameters of a task, so I wrote a thin wrapper for that [1]. This helps, but mostly for smaller deployments.
Airflow and luigi seemed to me like two side of the same thing: fixed graphs vs data flow. One fixates the DAG, the other puts more emphasis on composition.
That said, I am excited about the data processing tools to come - I believe this is an exciting space and choosing or writing the right tool can make a real difference between a messy data landscape and an agile part of business and business development.
I do not use FB and I still have friends, family, fun and a happy life.
My current understanding is that a event-based (in a very broad sense) system are hard to "replay" in the case of a failure (error in data or just a bug), unless you build additional machinery, which decreases robustness. In contrast the task of making a batch processing system perform fast is easy and much better defined.
> Recall that in our general definition a feed-forward neural network is a computational graph whose nodes are computing units and whose directed edges transmit numerical information from node to node.
> Each computing unit is capable of evaluating a single primitive function of its input. In fact the network represents a chain of function compositions which transfor m an input to an output vector (called a pattern).
From: https://page.mi.fu-berlin.de/rojas/neural/chapter/K7.pdf (1996)
Another ancestor would be the Data-Flow paradigm:
> .. programming paradigm that internally represents applications as a directed graph, similarly to a dataflow diagram. Applications are represented as a set of nodes (also called blocks) with input and/or output ports in them. These nodes can either be sources, sinks or processing blocks to the information flowing in the system. Nodes are connected by directed edges that define the flow of information between them.
From: https://paginas.fe.up.pt/~prodei/dsie12/papers/paper_17.pdf (2012)
That said, it's not super efficient and the incomes vary greatly between employees with strong lobby groups and laborers covered by legislation only.
It's funny. It is not that capitalism does not tend towards central planning. When companies fuse and get bigger, they want to reap the profits of centralization and unified control. To exaggerate a bit: the ultimate way to deal with the competition is to absorb or to kill it.
As for the zip: We simply "unzip -p" and stream process it carefully (with a custom program reading XML and transforming it). Cuts processing time from hours (extracting the zip and creating all directories, then visiting each file) to minutes (read from a single file).
Me too, for better or for worse.
As for the issues, there are many. Just quickly a few:
* Data provider has an FTP server, most files are automatically generated, some are hand-named (with inconsistencies). How do you handle (without a lot of effort) a list of exceptions along with the regular files?
* Data provider has a good strict XML schema, but the relevant information for a single item is spread across three files, inside a tar archive. Since the there are 500k files inside the archive, you best not want to extract it, but process it on the fly.
* Data provider chooses layout that saves every item in a single XML file, inside 2-3 levels of directories. There are 20M of them. Unzipping the archive alone takes more than a day with default system settings and the usual tools. How do you process these things fast?
There are more subtle issues as well:
* FFFD regularly occurs in natural language strings. Can you correct these strings?
* File has .csv ending, looks like CSV on first glance, but all the standard RFC compliant parsers choke on it.
* XML file that elements, that have RTF tags embedded in it. You need to parse the RTF in the elements, because there is relevant information there, that you need to add to the transformed version.
* Date issues. Inconsistent formats and almost-valid dates.
* Combine data, coming from an API with data fetched from ten different servers to produce a transformed version with a legacy command line application (that might be slow, so you have to split your data first and parallelize the work, combine it and make sure it's complete).
I am thinking about a longer article or even a short book about these kind of data handling and quality questions and what ways there are to address them. Would you read a book like this and what topic would be the most pressing or relevant?
What I found so far: Most tools, that invent a new language or try to cram complex processes into lesser suited syntactical environments are not loved too much.
A few people like XSLT, most seem to dislike it, although it has a nice functional core hidden under a syntax that seems to come from a time, where the answer to everything was XML. There are big data orchestration frameworks, that use an XML as configuration language, which can be ok, if you have clear processing steps.
Every time a tool invents a DSL for data processing, I grab my list of ugly real world use cases and most of the tools fail soon, if not immediately. That's a pity.
Programming languages can be effective as they are, and with the exceptions that unclean data brings, you want to have a programming language at your disposal anyway.
I'll give dgsh a try. The tool reuse approach and the UNIX spirit seems nice. But my initial impression of the "C code metrics" example from the site is mixed: It reminds me of awk, about which one of the authors said, that it's a beautiful language, but if your programs getting longer than hundred lines, you might want to switch to something else.
Two libraries which have a great grip at the plumbing aspect of data processing systems are airflow and luigi. They are python libraries and with it you have a concise syntax and basically all python libraries plus non-python tools with a command line interface at you fingertips.
I am curious, what kind of process orchestration tools people use and can recommend?
Me too.
What I found was that people do not recognize what tool git really is. People do not read the manuals and count on their world knowledge acquired from years of svn or other centralized systems. It's not the best sign, that you have to really dig git to be able to use it properly. But once you understood the core ideas, you can basically derive a lot of the workflows yourself, in a very principled and clear manner. Also: once you get a bit more familiar with the data model, one almost cannot not appreciate its beauty, I believe.
Nobody thinks of that as a career, but 4.5 million people seem to rely on it for income nonetheless.
The point being: You do not need general AI anytime soon, the consequences will still be significant, I would bet.
Quick check whether it is super fast for a simple task like extracting a single field from 1M JSON objects:
$ time rq -j 'at "title"' < fixtures/1m.ldj > /dev/null real 2m16.692s
$ time jq .title < fixtures/1m.ldj > /dev/null real 0m26.040s
So jq is 8 times faster? Anyway, rq functions look useful, so will definitely check it out.
But if I open a random influencer agency website, I can read something like this: "Influencers drive trust". The message here seems to be: Yes, trust is super valuable - and now the price is finally dropping! But is it really?
Same scheme that started AWS.
On a recent hackathon, I wanted to implement instant search and ended up using mostly plain javascript for this reason. Not pretty, but worked: http://i.imgur.com/lLi3noX.gifv
Unfortunately there are some roadblocks, like little information and conversation about the topic and no real mainstream implementation of these ideas.
I've been working on a system, that has is based on an immutable data layer and it is just a wonderful thing to work with, as long as you have enough disk space.
I remember Linus Torvalds mentioning, that while the 386 was a complex CPU, he was able to understand it on a sufficient level. But this time seems to be gone.