asavinov

I am curious to see what happens with other apps like airtable, rowshare or zenkit and the whole market. Will they be able to survive as independent companies (like Tableau or Qlik in self-service BI)?

Ingesting such amounts of data is a challenge indeed. But problems will become much more complicated if it is necessary to perform complex analysis during data ingestion. Such analysis (not simply event pre-processing) can arise because of the following reasons:

* It is physically not possible to store this amount of events. For example, assume you collect them from devices and sensors

* It is necessary to make faster decisions, e.g., in mission critical applications

* It can be more efficient to do some analytics before storing data (as opposed to first storing data persistently and then loading it again for analysis)

Such analysis can be done by conventional tools like Spark Streaming (micro batch processing) or Kafka Streams (works only with Kafka). One novel approach is implemented in Bistro Streams [0] (I am an author). It is intended for general-purpose data processing including both batch and stream analytics but it radically differs from MapReduce, SQL and other set-oriented data processing frameworks. It represents data via functions and processes data via column operations rather than having only set operations.

[0] Bistro: https://github.com/asavinov/bistro

It is a general-purpose and highly configurable data processing engine which can be applied to many workloads and scenarios including data integration, data migration, ETL, big data processing etc.

What is new is that this column can use data from other tables - not necessarily this one. And eventually Bistro gets rid of joins and group-bys which are not difficult to understand but frequently their use is inappropriate, for example, in most cases, joins are used where we actually want to link two tables (new relation is not needed).

Note also that we cannot avoid set operations and creating tables. Therefore, Bistro also supports table creation. Yet, it does it in a functional way, that is, by defining a new columns.

I like this idea because it allows for reusing an existing engine and, what is important, integrate it with many different engines (which might have already quite sophisticated mechanisms of data managements). Yet, such an engine has to expose quite low-level operations with data. Also, there has to be support for user-defined functions (lambdas).

* JavaScript data processing framework for in-browser data processing

* Python framework like pandas

* Big data processing framework like Spark

* Database management system

* Data integration system like typical ETL and BI

* Stream analytics like Kafka Streams

* IoT (light weigh) stream processing engine

* Something else?

I would be very thankful for any suggestion from people who know the market and (acute) needs of the customers. What is the best niche for this kind of technology?

* UI (Angular 2): https://github.com/asavinov/sc-web

* REST server: https://github.com/asavinov/sc-rest

* Core engine: https://github.com/asavinov/sc-core - Bistro is a complete rewrite of this project

There is an (old) implementation of this idea in C#:

* UI (WPF): https://bitbucket.org/asavinov/dc-wpf

* Core engine (C#): https://bitbucket.org/asavinov/dce-csharp

And also some old Java implementation the engine in Java:

* https://bitbucket.org/asavinov/dc-core

There are at least three major questions when we want to introduce a new logical data model:

* How we define columns within one table. Conceptually, it is easy, e.g., SELECT x, y, c = a+b FROM T. Yet, even in this simple case we see a controversy: this statement will create a table but our goal is not to create a table - we want to create a column (function). Bistro uses calc operation for that purpose. But it is of course not new. In pandas, for example, one can use df.apply.

* How to connect several tables. RM, map-reduce and other set-oriented approaches use join which produces a new table. Here we have a similar controversy [1]: I do not want to produce a new table, it is not my goal. My goal is to link these two tables and this means creating a new column. Bistro introduces link operation for such columns.

* How to aggregate data. Here a typical operation is group-by. It is an eclectic operation which combines several other operations and such an approach also has some problems [2]. Bistro changes the way data is aggregated by introducing accumulate functions which get one input (not a subset) and return one output value. An accumulate function is called for each element of the group by updating the current aggregate instead of computing the aggregate by getting the whole group.

So linking and aggregation are what distinguishing Bistro from other approaches and frameworks to data processing including pandas, SQL and map-reduce.

[1] https://www.researchgate.net/publication/301764816_Joins_vs_...

[2] https://www.researchgate.net/publication/316551218_From_Grou...

This is what I am trying to understand myself :) Previously I have implemented this approach as a web-app (self-service tool for working with tables where users can define columns as formulas - similar to spreadsheets). Discussed here: https://news.ycombinator.com/item?id=14351461 But it is too difficult to implement (much more resources are needed) so I switched to developing a library.

Now I want to implement a server for in-stream analytics (an alternative to Kafka Streams). Hopefully, in the next version of Bistro. It will have quite significant portions of data processing logic in UDFs (retention policy, when to evaluate, how to add etc. when to do what).

> Does it support time series? How works you do a moving average or pivot table?

I am designing it now as a tool for stream processing (Bistro Streams) and hence it will support time series (each new row will get a time-stamp and the system will "know" how to deal with time).

Current approach to user-defined functions (Evaluator interface) does not support moving average or other rolling aggregations. New API will be defined. This task has high priority since it is very important for stream analytics.

Pivoting is conceptually more difficult because it is not an operation with data - it is an operation with schema (using data). Maybe some kind of ad-hoc solution will work.

It is definitely interesting and important to implement persistence (for example using Parquet or Arrow) as well as other mechanisms like sharding or replication (for big data processing, fault tolerance etc.) Yet, currently this direction has lower priority because the next task I want to focus on is in-stream analytics (an alternative to Kafka Streams).

In general, the whole approach is focused on the logical level of data modeling and processing, that is, the goal is to increase performance and simplicity of development. The general idea (and hypothesis) is that defining how data is being processed using column operations is easier, more intuitive, less error-prone and easier to maintain than using purely set-operations.

In other words, at logical level, it is an alternative to map-reduce, SQL, pandas and other models and frameworks where set-operations are used to process data.

After getting some feedback I decided to position this technology differently. Instead of exposing the functionality via web app, I started implementing a Java library http://github.com/asavinov/bistro - an alternative to map-reduce. In 2018 I am going to develop a server for IoT and stream analytics - an alternative to kafka stream analytics based on Bistro.

1. How data is represented. A database is a collection of column objects. For example, it is easy to create a column or delete a column.

2. How data is being processed (queried). The engine processes columns as objected managed by the system

- https://medium.com/@pechyonkin/understanding-hintons-capsule...

- https://hackernoon.com/what-is-a-capsnet-or-capsule-network-...

It is much more difficult not to add a feature than to add it, and the ability to reject even important feature requests is very important for the survival of a system in long term. This observation is formulated in "Getting Real" (https://gettingreal.37signals.com) as "Forget Feature Requests" and "Start With No".

[1] https://cloudandheat.com

[2] https://www.informationweek.com/cloud/heating-your-house-wit...

  language constructs <--> design patterns

In fact, these design patterns and language constructs come in (anti-phase) waves.

Indeed, major problems of programming languages can hardly be solved in the area of programming languages itself as it is being done now.

I would say that one needs a new programming or computing model so it is not about languages. At least it is my conclusion after 10+ years of research and attempts to develop such a new programming paradigm. And although I have quite significant progress (concept-oriented programming), the more I do and the deeper I go, the more fundamental problems I meet. And these problems are not about programming languages at all. It is more about "how a system works", "what is a system", "what is computing" etc.

The term "paradigm" is too general indeed and therefore can be quite ambiguous in many situations because it covers different aspects like the underlying computing model or language design.

[1] http://www.jetp.ac.ru/cgi-bin/dn/r_043_0605.pdf (in Russian, Abstract in English at the end).

    f: X -> Y

    f: U -> V

    If u <: x then v=f(u) <: y=f(x)

Currently formulas are evaluated only when the user explicitly presses Evaluate button (or using explicit API call).

The logic of automatic evaluations is work in progress and it is needed more for data integration or stream analytics. It is assumed that the formulas will be evaluated under certain conditions on regular intervals (say, one time per second), on certain events (after 10 appends) and so on.

> I would like to try the tool but keep getting a "Server Error" message

Hm, I have just checked and it works. It uses JSESSION so cookies must be enabled or may be some problem with the old session id so try to open new clean browser window.

It is MVP so there defenitely can be some technical problems. Sorry for that and thanks for testing it.

Here is my implementation of this column-oriented approach as a Java library for server-side transformations: https://bitbucket.org/conceptoriented/dc-core

Data Commandr (DC) is a web application for working with tables. Its goal is to make complex operations with data tables as easy as working with classical spreadsheets (but it is not a spreadsheet application) for producing reports, data integration, data migration and other data wrangling tasks. Its main distinguishing feature is that it does not use traditional (set-oriented) queries but rather relies on column formulas to derive new data.

I would like to get your option about Data Commandr from three points of view:

- [As developers] How do you like the current UI and general design? How it can be improved?

- [As users] Can such a tool be useful for working with tables?

- [As investors] What other tasks can be solved by using this approach? For example, I want to pivot in the direction of stream analytics. Or it could be applied to tasks where Spark is normally used (DC might have much higher performance and much simpler data processing language.) Other applications: data dictionary and data catalogs.

Thank you in advance for testing the app and your feedback.

COMPARISON:

- [Spreadsheets] In spreadsheets, a cell is defined as a function of other cells so we get a functional model where the result of each cell (one value) is computed by evaluating other cell formulas. In contrast, DC uses columns instead of cells so that a model is a number of column formulas.

- [Set-oriented approaches] In set-oriented approaches (RM, Hadoop, Spark etc.), new data is derived by applying an operation to input sets and producing an output set. Data transformations are represented as a graph of set operations. In contrast, data transformations in DC are represented as a graph of function operations.

- [Research] Formally, Data Commandr relies on the concept-oriented model of data [2]. In particular, DC does not rely on such difficult to use and understand operations like join and group-by. Instead, it uses link columns [4] and accumulate functions [1], respectively.

- [Competitors] When I started this project [5] I knew only one similar product which is Power BI and its DAX language. Now there exist such relatively new products like airtable, fieldbook and rowshare.

[1] A. Savinov. From Group-By to Accumulation: Data Aggregation Revisited. Proc. IoTBDS 2017, 370-379, 2017. https://www.researchgate.net/publication/316551218_From_Grou...

[2] A. Savinov. Concept-Oriented Model: the Functional View. arXiv:1606.02237 [cs.DB] 2016. https://arxiv.org/abs/1606.02237

[3] A. Savinov. DataCommandr: Column-Oriented Data Integration, Transformation and Analysis. Proc. IoTBD 2016, 339-347. https://www.researchgate.net/publication/301764506_DataComma...

[4] A. Savinov. Joins vs. Links or Relational Join Considered Harmful. Proc. IoTBD 2016, 362-368. https://www.researchgate.net/publication/301764816_Joins_vs_...

[5] A. Savinov. ConceptMix: Self-Service Analytical Data Integration Based on the Concept-Oriented Model, Proc. DATA 2014, 78-84 https://www.researchgate.net/publication/265301356_ConceptMi...

Deep learning is called deep because it is based on multiple levels of features corresponding to a hierarchy of notions. Of course, it can change how generalization as well as other operations are performed but the way generalization is done not a specific feature of deep learning.

What I see similar in object-capability model and COP is that they both try to treat object access as a highly important part of a system behavior. In COP, I used to write that it is more important what happens during access rather than in the object itself. Hence, being able to make these intermediate actions integral part of the program (and develop good PL for that purpose) is very important.

[1] https://arxiv.org/abs/1501.00720 [2] https://www.researchgate.net/profile/Alexandr_Savinov

Disclaimer: I am the author of concept-oriented programming and data model