I have no experience with NiFi itself.. just that graphical tools are not inherently "non-technical"
It's more that graphical tools don't generally serve their stated purpose of making it possible for non-programmers to get things done because it appears that the main skill behind programming is actually problem decomposition and modelling rather than syntax.
Additionally, graphical tools tend to have a bunch of problems inherent to them, such as being unable to write comments and harder to store in git and work on collaboratively and only having one single editor, which is usually much buggier than a text editor and compiler.
So: graphical "languages" don't make most things much easier, and make other things much harder.
Totally. This is where declarative and intention-oriented systems shine. Take something like SQL, where, in the majority of cases, the end-user needs to know next to nothing about algorithmic complexity and can still achieve excellent performance and correct results.
It'd be neat to see a system that was actually designed toward problem decomposition. What's the state of the art around this kind of stuff?
Mind, though, that this system had made a ton of money for 15 years, so in a way it was a huge win for them! They just eventually painted themselves into a corner and performance started to get worse faster than Moore’s law could save them. Last I heard, the great untangling is still in progress, and I stepped away from that project a few years ago.
They likely have still come out on top when you take the sum of revenue they made from the system, but they incurred a huge unanticipated cost and got backed into a pretty bad corner. Sales people were still selling features that didn’t exist, and the team was desperately trying to build new things while also fixing the ever degrading performance.
We have recently released a GUI tool to do data transformation/ETL ( https://www.easydatatransform.com ). It is aimed more at power Excel users, than programmers or data scientists. We have tried to overcome some of the issues mentioned here:
* It is only intended for the limited domain of data transformation, which limits the complexity.
* No need to remember any syntax/commands for the vast majority of use cases.
* The transformation template is stored in a single XML file, so it can be versioned.
* You can use regexp, Javascript and command line arguments to get extra power if you need it.
* It is written in C++ and is fast.
Obviously it can't do everything you can do with a general purpose programming language, but it hopefully do enough for many use cases and is much easier to learn.
Above a certain complexity they become very hard to organize. If the complexity of your problem is below this level and the user has any self control, all is good. If the complexity of your problem is higher you'd need a programmer who knoes how to handle this anyways.
No. But it's controversial to claim that a graphical tool can do maintainable code.
Everything under source control, defined as code.
* This gives you the ability to trace who did what
* Good code-commit hygiene with prs
* The ability to revert changes
* Ability to have an exact replication and promote across environments
Pragmatically test changes, unit test your flows
* Tie all the above up with a deployment pipeline
NiFi is being used in the critical parts of business data pipelines and all the devops rules for reliable development and deployment go out the window.
If NiFi is used by one person for a minor role the GUI is fine. But if it’s the core data pipeline which is a critical service used by multiple teams it’s not the smartest way to do things.
It's a similar train of thought that leads to a lot of game developers to look down on games made with GUI tools like gamemaker or such.
Programmers have seen these sorts of UIs fail to deliver countless times as described in the root of this thread. Where is the counter example?
I'm honestly all for tools that enable non-programmers to get things done. E.g., Excel, Google Forms, Zapier, and a lot of Salesforce tooling all deliver to some extent on that promise. But there's only so much they can do.
You're right that there's also an elitism problem. But if we're going to get past that, we have to start by understanding what GUI tools are good for. And being honest about their limits.