loa_observer

What's the unique of HowdyGo, seems there existed lots of similar services.

Applied Perception (visualization) at coursera of University of New York.

It can be time consuming learning some lib in developing. Those libs has lots of APIs and it might be of low efficiency to learn them in docs.

Before ChatGPT, I learn those by searching examples and real repos which is more efficiency that follow tutorials or searching in docs. But sometimes it's hard to find examples you need. But with ChatGPT, it generates exactly sample codes for me to learn stuffs, and I even do not need to spend lots of time searching for resources.

The target is to handle common size csv/excel files on the client-side. Supporting to 1GB may cover most cases in daily work.

For larger datasets, I agree to handle them on server-side.

I have been working on an open-source project called RATH, an automated visualization tool for data exploration. However, I have encountered some performance issues when working with large datasets (> 1 million rows).

1. High memory usage: When loading large datasets, the memory usage can become quite high. (50MB csv will use 700MB memory in RATH)

2. Slow computation tasks: Group-by, filter, bin, or even Cube operations can be slow and sometimes block the main thread.

3. Slow chart rendering: Chart rendering can also be slow and sometimes block the main thread. (Currently using VegaLite)

I have implemented some solutions to address these issues:

1. For high memory usage, I am storing large raw data in indexedDB and reading it as needed. This reduces memory usage but can still consume a lot of memory when the data is loaded into the main thread.

2. To improve the performance of computation tasks, I am using web workers for some data computations (such as group-by, bin, transform, and filters). I am also testing duckDB-wasm, but lack of some knowledge of its best practice.

3. For slow chart rendering, I have tried using offscreen canvas to render the chart in a web worker. However, this approach creates a static canvas without any interactive features (such as tooltips, zooming, or callbacks for data selections). I am looking for methods on how to make the chart rendered in the web worker interactive.

Any suggestions or experiences shared would be greatly appreciated.

RATH Github Repo: https://github.com/Kanaries/Rath

RATH basic background: RATH is beyond an open-source alternative to Data Analysis and Visualization tools such as Tableau. It automates your Exploratory Data Analysis workflow with an Augmented Analytic engine by discovering patterns, insights, causals and presents those insights with powerful auto-generated multi-dimensional data visualization.

Besides automated data profiling, RATH focuses more on EDA, especially good at high-dimensional cases. It can generate visualization with insights about complex relations of multi variables (for example, 6 vars).

You can find the difference of RATH in mega/semi-automation module(based on predictive interaction), and in data painter, and causal analysis as well.

It's a cool project, thanks for sharing. graphic-walker focuses more on a Grammar of Graphics based graphic system, not a chart-based system.

But for performance in the browser, I am also researching ways to handle large datasets in the browser, such as duckdb-wasm.

Another project I am working on, called RATH is facing more challenging performance issues because it is designed to automate the exploration of data to find insights and causal relations by machine. It currently is using web worker and indexedDB to implement a lite cube service in the browser, but it is not enough.

https://github.com/kanaries/rath

Thanks for your advice, I will continue to improve the both computation and render performance in browser.

RATH is beyond an open-source alternative to Data Analysis and Visualization tools such as Tableau. It automates your Exploratory Data Analysis workflow with an Augmented Analytic engine by discovering patterns, insights, causals and presents those insights with powerful auto-generated multi-dimensional data visualization.

To get started with RATH, you can: - [Try RATH now!](https://rath.kanaries.net) - [Read RATH Docs](https://docs.kanaries.net) - [Video introducing RATH's data painter](https://youtu.be/djqePNyhz7w)