asavinov

Indirection [0] is a quite big topic and applying it can result in both significantly improving the code and getting more problems than benefits. Therefore, the question is when indirection is appropriate. One opinion [1] is that

    "We can solve any problem by introducing an extra level of indirection" 

Another opinion is expressed in this post (avoid indirection). In the context of this example, indirection is related to the problem of modularization and separation of concerns. In particular, a typical question is whether we want to hide some details or use the functionality directly.

[0] https://en.wikipedia.org/wiki/Indirection

[1] https://en.wikipedia.org/wiki/Fundamental_theorem_of_softwar...

> However, I'm wondering if SQL is the right tool for the task. For instance, the listing 2 seems complex when compared to the query expressed in plain English

For me Listing 2 is more complex and less intuitive than Listing 1 (CQL). Yet, it is a general problem when we try to adopt relational (and SQL) concepts for solving such kind of tasks (windowing, grouping, aggregation etc.) One solution is to switch to some kind of column algebra rather than relational algebra as described in [1] and [2] (it has also been applied to stream processing).

[1] Joins vs. Links or Relational Join Considered Harmful: https://www.researchgate.net/publication/301764816_Joins_vs_...

[2] From Group-By to Accumulation: Data Aggregation Revisited: https://www.researchgate.net/publication/316551218_From_Grou...

And also wonderfully visualized! Does anybody know what tool can be used to produce such figures?

From the conceptual modeling point of view it is important to understand that:

(1) There can be several levels of entity identifiers, that is the same entity exists at several levels where it has different identifiers. Example: a computer has DNS name and IP address (and MAC address); a person may have several identifiers. These (initially independent) spaces can be structured differently:

(1.1) Layered structure like DNS and IP (1.2) Independent, e.g., a person has two passports from different countries

(2) Links (representing relationships) are attributes. Like all attributes they are functions which map input entity identifiers into output (address) space identifiers.

Taking this into account, the problem is that URLs frequently play two roles, that is, one address convention is used at two layers simultaneously which can lead to numerous problems and controversies:

o URLs are used at access (protocol) level where the goal is to identify computers and access paths within some service (and not entity). For example, this layer has no idea what this means: http://service.com/passports/1234 - it is used for HTTP access only.

o URLs are used to identify real entities and here we re-use the previous name convention for higher level purposes by identifying people like http://service.com/passports/1234 although we do not care about protocols or computer names.

In order to avoid serious problems we need to follow these steps:

* Recognize and document that there are two independent address spaces

* Define a mapping between these two spaces. Initially, it can be identity relationship, that is, every entity name (URL) is mapped to the same URL used for access.

* If necessary (and frequently highly desirable) use relative names by assuming that the higher level segments of the name can be restored from the context or configured separately.

These rules are especially important for extensibility and for complex evolving systems where the standards and conventions change in time.

Conceptual and data modeling aspects of this problem are discussed in [1]. It compares links with joins (and foreign keys) by proposing a solution (concept-oriented model) which does not use joins at all but rather relies on links only.

Essentially, a foreign key is viewed as a relational workaround for representing links with some significant drawbacks and the question is why not to use links directly without relational wrapping.

[1] Joins vs. Links or Relational Join Considered Harmful: https://www.researchgate.net/publication/301764816_Joins_vs_...

IMHO (deep) feature engineering is important in these cases:

o the lower the level of representation the more important it is to increase the level of abstraction by learning or defining manually new features

o in the presence of (fine-grained) raster (automated) feature engineering is especially important. Therefore, feature engineering is important in audio analysis (1d raster) and video analysis (2d raster).

Deep feature extraction is important for not only image analysis but also in other areas where specialized tools might be useful such as listed below:

o https://github.com/Featuretools/featuretools - Automated feature engineering with main focus on relational structures and deep feature synthesis

o https://github.com/blue-yonder/tsfresh - Automatic extraction of relevant features from time series

o https://github.com/machinalis/featureforge - creating and testing machine learning features, with a scikit-learn compatible API

o https://github.com/asavinov/lambdo - Feature engineering and machine learning: together at last! The workflow engine allows for integrating feature training and data wrangling tasks with conventional ML

o https://github.com/xiaoganghan/awesome-feature-engineering - other resource related to feature engineering (video, audio, text)

> The relational model is not well suited to uncertain data, as a row in a table is generally interpreted as a true proposition. For statistical data sets, analytical processing may be better served by array/tensor models (which also exhibit uniformity).

Arrays/tensors and (relational) tables can be thought of as alternative ways to represent a set with this major difference:

o [Relational table] Each column is an axis. A row is point.

o [Arrays] Each dimension is an axis. A cell is a point.

This is why it is wrong to interpret a table with a two-dimensional array - their data have completely different semantics.

The success of a (data) model depends on its ability to represent and manipulate relationships in a simple and natural way.

There exist of course other uniform ways to represent data, for example, using functions (and operations with functions). But many of them have been developed under the umbrella of the relational model even though they have little to do with the relational principles.

> Apache Flink, Flume, Storm, Samza, Spark, Apex, and Kafka all do basically the same thing.

Yes, conceptually they are very similar. If youu want something radically new then check out Bistro Streams: https://github.com/asavinov/bistro

Because they are so much similar that it is much easier to implement them as part of one mechanism. For example:

* Apply a (say, SVM) classification model to each object (row) by producing a new column

* Generate a new column as a difference between its values and its average over all rows.

In both cases, you produce a new column (=feature) by applying some transformation. Also, in both cases, you need to find parameters of this transformation from the data. In the first case, by training SVM model. In the second case, by find the average value.

Conclusion: there is no essential difference between defining/training a feature and a ML model.

Lambdo is a workflow engine which simplifies data analysis by combining in one analysis pipeline

* Feature engineering and machine learning: Lambdo does not distinguish them and treats them as data transformations

* Model training and prediction: both feature definitions and ML models can be trained as part of one workflow

* Table population and column evaluation: workflow consists of nodes of these two types. This makes it similar to Bistro: https://github.com/asavinov/bistro

Lambdo is intended for the following use cases:

* Numerous derived features with parameters derived from the data

* Regular re-training is required by using the same features as those to be used during prediction

* Time series analysis because it is where the quality of derived features is especially important

* Customization via user-defined Python functions

The book focuses on classical (statistical) methods of forecasting. In this sense, it provides the fundamental notions needed to deal with practical problems. Real world problems are much more complicated and first of all because of the natural of source data which is not limited by univariate numeric data. In most practical cases the success depends on the ability to extract (manually or automatically) good features from heterogeneous data sources. There exist the following frameworks for that purpose:

o https://github.com/asavinov/lambdo - Combines feature engineering and data mining with strong focus on time series analysis

o https://github.com/blue-yonder/tsfresh - Automatically extract informative features (also from time series)

> Joins are the part that everyone has trouble with.

There is significant problem-solution mismatch in joins and some other SQL constructs which are therefore semantically quite controversial in many use cases:

https://github.com/asavinov/bistro#why-column-orientation

Another project relying on lamdas for data processing https://github.com/asavinov/lambdo yet focused more on feature engineering and ML

It is again a workaround because <authors> is a member of a collection - it is not a tuple attribute. Which suggests that we cannot enforce this separation and an application has to understand itself which element is an object property and which element is a member of a collection.

Yes, you are right. So the question is which model is better: a tree model (with nodes as flat tuples) or JSON model with nodes being either collections or combinations (with arbitrary nesting). It seems that JSON is more general while XML requires you to map the reality into a tree representation. It is somewhat similar to the relationship between the hierarchical data model and the relational data model.

Typing (enforcing constraints) is an important aspect. But even without typing XML has one fundamental flaw. You are not able to (correctly) represent tuples with attributes which are sets. In XML, tuple attributes are properties, for example:

    <book id="1" title="XML"> -- object or object

Now let us assume that I want to have a list of authors as a property:

    <book id="1" title="XML" authors=["Me", "My Friend"]> -- NOT SUPPORTED

Therefore, we use a workaround (a crime actually):

    <book id="1" title="XML"> 
        <author>Me</author>
        <author>My friend</author>
    </book> 

Now a book is a collection where authors are members (IS-IN relationship). This is not what we wanted. Our goal was to have an attribute "authors" which is a collection.

If we ignore syntactic aspects then the success of JSON is due to one fundamental reason. JSON relies on two structural elements [] and {}, which have very clear semantics of sets (collections) and tuples (combinations), respectively. It is simple, general, natural and consistent.

In comparison, XML also provides two structural elements but they do not have such a nice duality and clarity of interpretation. Here is a very typical example which is more than an anti-pattern:

    <books> 
        <book id="1"><title>XML is evil</title></book> 
        <book id="2"><title>XML considered harmful</title></book> 
    </books> 

Here we see that semantically <books> is a collection while <book> is a tuple (where <title> is an attribute). Yet, they use one and the same structural construct. XML does not enforce the dual semantics of sets and tuples, and therefore XML files are frequently a conceptual mess. The above example is essentially wrong but XML does not care.

For stateful processing you need something like:

* https://kafka.apache.org/documentation/streams/ - Kafka Streams (and its KTable)

* https://flink.apache.org/ - Flink

* https://spark.apache.org/streaming/ - Spark Streaming

* https://github.com/asavinov/bistro - Bistro Streams

The focus of Benthod seems to be performance and resilience (implemented without persistence).

There are two general (potential) problems due to the use of (multiple) joins:

* run time: performance disaster

* design time: conceptual chaos

Some of them are analyzed in [1] where it is essentially argued that join considered harmful and a join-free (column-oriented) approach is described. This approach has been implemented in [2] which is a library for batch and stream data processing, and in [3] which is a library for data analysis. Their main common unique feature is that they rely on column operations, that is, data processing is described as a graph of operations with columns as opposed to using set operations like join.

[1] https://www.researchgate.net/publication/301764816_Joins_vs_... Joins vs. Links or Relational Join Considered Harmful

[2] https://github.com/asavinov/bistro - A general-purpose data analysis engine radically changing the way batch and stream data is processed

[3] https://github.com/asavinov/lambdo - A column-oriented approach to feature engineering. Feature engineering and machine learning: together at last!

(Disclaimer: I am an author)

You can check out Bistro [1] which is an alternative to SQL-like languages and to set-oriented approaches in general. It focuses on column operations (formally, functions) as opposed to having only set operations. It is pricesely why it works well for time series and it is why it has been used for stream processing.

[1] https://github.com/asavinov/bistro

Another problem of anomaly detection is that they do not provide any (domain specific) explanation for why the system thinks it is an anomaly. The system also does not say what to do in this situation, which means that such anomalies are not actionable findings. Therefore I think anomaly detection should be used as a pre-processing step which generates input for some other other components of the system.

Here is another generic approach to anomaly detection from event data which has been used for analyzing logs received from automatic lawn mowers:

https://www.researchgate.net/publication/323971244_Detecting...

It allows for using different algorithms like one class SVM or MDS (including custom algorithms). It also allows for defining custom domain specific features as integral part of its analysis engine. In particular, for log analysis, frequencies of various event types have been generated.

> I'd like a query language that more expressive, expression-based syntax where the results of each expression can be "piped" into the next step to perform transformations, joins, reductions and so on.

I do not think this can be easily fixed in SQL because the problems are not in the syntax but rather lie deeper at the level of the underlying paradigm and principles.

> However, the underlying design philosophy doesn’t fit very well into a classical OOP world

These problems indeed belong to a non-OOP view of programming. In OOP, the behavior is concentrated in objects while here a significant part of the behavior is concentrated in references.

I have been working on one possible generic solution which is called concept-oriented programming [1] and where references are first-class elements of the program where a great deal of the program behavior is performed during object access using these references.

One of the main ideas is that each object class is accompanied with a reference class. Such a couple is referred to as a concept (hence the name of the model). If reference class is empty then we get normal OOP. If object class is empty then we can model normal values (which are passed by copy). Of course, we can build a hierarchy by inheriting the behavior of references and objects (which is different from the classical inheritance but generalizes it in case we have only object classes).

COP is still on early stage and cannot be used just because there are other quite serious unsolved problems. But I see that the problems described in this post are not exceptions - describing how objects are represented and how they are accessed is an important piece of custom functionality in any complex system and it is precisely COP tries to formalize (along with some other tricky problems).

[1] http://conceptoriented.org Links on concept-oriented programming (COP)

It is based on a quite general (and novel) technology so it can be applied to many different data processing tasks like data integration, data migration, extract-transform-load (ETL), big data processing or stream processing.

But I would like to narrow this list down with the purpose to increase probability of success (degree of adoption). Therefore, currently I focus on edge computing and IoT. But maybe I am wrong and therefore I would like to ask the community about potential uses of this kind of technology – fail early fail often :)

For simplicity, it can be viewed as Kafka Streams but implemented on completely different principles at all levels of the architecture.

Bistro Streams [1] is similar to Kafka Streams in how it is supposed to be used - it is implemented as a library which can be part of an application (including IoT analytics) or run at the edge, for example, on gateways or devices. However, Bistro Streams is based on different principles and has the following major distinguishing features:

o [Column-oriented logical model] Bistro Streams describes its data processing logic using mainly column operations as opposed to set operations [2]. It is a unique feature of this system: no joins, no group-by, no map-reduce.

o [Column-oriented physical model] Data within the system is represented in columns (in-memory). It is apparently not new and widely used in column stores but it is new for stream processing. In the case of long histories (which is needed for complex analysis) and complex analytic workflows it can provide higher performance. It is also important for running on edge devices with limited resources.

o [Separate injection, processing, retention] Bistro Streams separates the logic of (1) triggering the processes for appending, evaluating, and deleting data from the logic of (2) what to do during evaluation (data processing itself). In particular, the frequency and conditions for starting evaluations are specified using a separate API. The same for retention policy where deletion time is determined not by the windows used during its processing (say, for moving average) but separately.

I am the author of Bistro Streams [1] and the underlying Bistro Engine [2]. I will be glad to answer questions and any feedback is welcome. In particular, what are possible application areas for this system.

[1] Bistro Streams: https://github.com/asavinov/bistro/tree/master/server

[2] Bistro Engine: https://github.com/asavinov/bistro/blob/master/core

The question whether we need distributed stream processing can hardly be answered unambiguously. In general, everything depends on what features we actually need: scalability, fault tolerance, high availability, performance (latency or throughput) etc.

In any case, there are two ways to distribute stream processing tasks:

* [~Heavyweight] Use a cluster with multiple runners like [Spark Streaming](https://spark.apache.org/streaming/) or [Flink](https://flink.apache.org/)

* [~Lightweight] Do stream processing in applications, at the edge, on gateways or devices like [Bistro Streams](https://github.com/asavinov/bistro/tree/master/server) or [Kafka Streams](https://kafka.apache.org/documentation/streams/)

Normally, distributed stream processing requires also partitioning the data, e.g., by user or session ids, so that these isolated streams can be processed independently at different nodes.

> it seems like it's almost a hybrid between a message queue and a database?

If Kafka supported data processing (queries or whatever) then it would be much closer to databases. Also, databases are normally aware of the structure of the data (for example, columns).

Therefore, Kafka can hardly be viewed as a DBMS because it explicitly separates two major concerns:

* data management - how to represent data (Kafka)

* data processing - how to derive/infer new data (Kafka Streams - a separate library)

Theoretically, if they could combine these two layers of functionality in one system then it would be a database.

Formally, null can be defined as empty tuple NULL=⧼⧽ as opposed to normal values which are non-empty tuples like ⧼5.0⧽. It is analogous to having empty set ∅={} and non-empty sets {5.0, 6.0, ...}.

    TUPLES       SETS
    NULL=⧼⧽       ∅={}
    combinations collections

Obviously, there is duality between tuples and sets but currently there is no theory which would be based on this property by treating them as equal constructs (tuples are defined as ordered sets by simply ignoring their duality to sets).

Since variables need to allocate some memory to store their values, it is necessary to introduce a mechanism to deal with empty values and then somehow interpret them.

There was another discussion of nulls some time ago: https://news.ycombinator.com/item?id=17028878 (The Logical Disaster of Null)