Show HN: Marple – Visualising millions of datapoints for engineers | Hacker News Reader

Show HN: Marple – Visualising millions of datapoints for engineers

getmarple.io

116 pointsNeroVanbierv45 comments

Hi HN, co-founder here! We designed Marple for engineers to make visual analysis intuitive and powerful. Most of our users are control systems engineers or test engineers (but not the kind that tests software). They often have to draw conclusions from millions of time series data points. As a quick example: logging data from a drone control system for 15 minutes at 1.5 kHz is already more than 1 million samples.

This creates some interesting technical challenges. We implemented a custom visualisation engine that efficiently displays only the data that fits the screen. In the same way, we also support applying functions such as low-pass filters, integrals, derivatives, etc.

Having a visual approach to analysing data is in our opinion still highly underrated. Common engineering tools such as Matlab or python+matplotlib scripts require you to reduce the number of samples yourself. And you end up with a plot that is not interactive: zooming, panning highlighting, adding a different signal are all cumbersome or impossible. Marple treats interactivity as a first class citizen.

We recently launched a beta version and we're currently looking for test users to try it out. If you sometimes need to visualise some data from a .csv file, python script, matlab script, simulation, ... you should give it a spin!

Cheers! Nero

Homepage and pitch looking awesome!

Some questions:

- Is there druid integration? Is this a replacement for druid? - How does schema inference work? - Can I do SQL queries? - How is this better than superset?

There is no integration for any db setup at the moment, but this is one of the next things we will tackle for sure! Druid was not on our radar, seems like it is used a lot in automotive/aerospace. Never tested superset, I'll give it a try. But it looks to be more like a dashboarding tool. Our goal is to provide an investigative tool that allows the user to detect problems and in a very interactive way change the visualisation.

I downloaded and looked at it. Very cool. Superset has the broad alchemy integration, which is pretty nice.

Integrating into a larger enterprise data dictionary would be nice. Eg, amazon glue

This is really a big data visualization tool. There's a whole gigantic rich market there to capture. You seem to be looking at it from the perspective of csv files, which is a fresh take, I guess.

It's also unfortunately a very competitive space being worked on by folks with insanely deep pockets and hyper smart engineers: https://www.newgenapps.com/blog/10-big-data-visualization-to...

Still, a lot of what's out there is ungainly bloat ware meant for business logic data. If you target specific engineering/science verticals with clear use cases, it would compete.

(other co-founder) Thanks a lot! Great to hear these positive reactions. You make some very valid points. We are indeed trying to focus on a certain use case, which we encountered our self in the past.

This looks great, will definitely try it out. May I ask what tech stack are you using?

Python + React in electron

Audio data is typically sampled at 44.1kHz and up, so audio visualization tools also have to handle millions of samples for filtering and display. I'm curious, is there any overlap in the techniques you used vs. those that an audio tool would use?

A very good suggestion and something we need to look into. Thinking about it, the techniques will probably be similar.

Congrats for your tool, seems impressive.

Yet another great project supported by imec!

How much does this actually cost? I don't see pricing anywhere on the site. I assume that info comes with the trial email but TBH I don't want to waste my time with the trial if it's going to be out of my price range.

This is a beta version, the goal of the trial is to gather feedback and find interested parties. Pricing and pricing model is not set in stone yet and will depend on what we see is feasible. Feel free to contact us in private to discuss more in detail! :)

Well that's just odd because I happen to live in Marple the town https://en.wikipedia.org/wiki/Marple,_Greater_Manchester

Looks amazing! I’ve been using KST for years https://kst-plot.kde.org/ for interactive plotting but marple looks to have some features kst does not

(other co-founder) Cool! Let us know how you like it compared to KST :) We focused a bit more on UX and interactiveness

Ooooooh interesting! Does it integrate with streaming or realtime services / db solutions? What's the max load you've tried out so far? Would you offer on-prem like Metabase?

Realtime and db are among the things that we will work on next. We still need to figure at a way to do this efficiently for us so that we don't need to integrate ourselves with every kind of db. The maximum was about 400 million datapoints in one visualisation, and that worked fine! We're also definitely considering a Metabase/Grafana approach where you can deploy it yourself.

Hello. Nice product, but some small problems:

- on MacBook 13", when editing calculated signals the dialog is bigger than the screen and the dialog cannot be closed (and maybe some other parts of the interface are hidden) - maybe add a max width/height on the dialog and a scroll

- the license does not work on windows (but works on mac build)

- on mac the feedback button the application splits in 2 identical views, with the mouse cursor movements mirrored (and an error about creating a folder .../bugs)

Sorry for the huge delay, I missed your comment. Thanks for the feedback! - just to confirm: this means your screen is has a height < 800px? - do you remember what exactly went wrong on windows? - sounds very weird behaviour, do you have a screenshot?

Thanks!

I guess I will take the opportunity to congratulate Marple and say (self promotion), that you have an open source alternative, that is equally able to deal with gigabytes of data and millions of points.

https://github.com/facontidavide/PlotJuggler

It looks likes this is a chromium/proton wrapper around a webapp. Is that correct? The plot lib used seems very similar to plotly.

I'd be very curious to hear about some of the tech stack design choices. I'd be also interested to understand who's your target user.

Very close, it uses Electron! Front-end is written in ReactJs. The plot library is react-vis currently, but we're replacing that soon. All the zoom/pan logic we implemented ourselves. Backend is a completely separate process. We've chosen for this architecture as it gives us the flexibility to easily change from desktop product to a web/server product, if we feel like that would be a better fit :) Target users can be very broad and is something we are still figuring out, but at the moment we see that engineers that have to deal with data from tests or simulations are a good fit for Marple.

Do you plan to commit back to the community your extra graph UX/UI. They look very neat and I don't think it will jeopardize the success of your startup. The reason why I asked about the usage of Electron (or similar) is because they are well known to be resource hungry. Slack went down that route and found themself in a pickle. Considering you are also doing a lot of data manipulation in memory (I assume) this might be very expensive on a React Stack. Anyway, something to keep in mind, don't worry about it right now.

If you are going after individual contributor, I'd spend quite some time figuring out good integration points (ie.: Notebooks), so you are better integrated in their workflow. I'd avoid going enterprise, because date warehouse and data visualization are well solved problem (both in SaaS and PaaS flavor). Source: I led data organization at mid/big companies.

(Mbaert his comment, but his account is throttled) We were worried about the performance of Electron as well. All data intensive work is done in a separate process, once in react/electron no data manipulation is needed anymore. So far it works quite ok, performance upgrades we find in electron usually don't have anything to do with the data.

Thank you for the info, we're still finding our position in the market. Figuring out the perfect use case and integration in a certain workflow will be key indeed! Feel free to contact us in private if you like to discuss more in detail about your experiences, we'd love to hear more :)

For some reason, your Website is broken on Safari: The images only show as a gray vertical line, about 5px high. Right-click -> "open image in new tab" works, so I'm guessing it's a CSS issue?

Works for me in Safari (Mac Catalina and iOS).

Yeah, it worked for me as well after reloading. Then it didn't work again on the next reload. And once I opened developer tools, it always worked.

(other co-founder here) Thanks for the heads-up! Might be slightly overloaded at the moment

Nice! Having developped a tool similar in features at my current job, I'm about to start on my own a similar projects... Happy to see that this area get some traction.

Tried it out, looks nice :) Is there a quick screenshot feature? That would be really helpful for sharing analysis and storing important views.

Thanks :D We're working on an export feature to quickly share or paste in a ppt etc.. But for now you'll have to rely on your printscreen or snipping tool skills.

This looks great, congrats on the launch! Are you taking advantage of any hardware acceleration (eg GPU or CPU vectorization)?

Thanks, cool question. The insane performance gains we have when visualising are mostly because of conceptual decisions we made. One such thing is that we don't try to plot more datapoints than there are pixels on the users screen. But when a user does calculations on data (e.g. a low pass filter), we really have to do all the math. So in that case we are mostly using numpy (python) operations, which are optimised in C already. But I don't know if this includes vectorisation

Is this a system that runs on a laptop or is it on cloud?

On your own laptop for now. This has the advantage that your data is already there on your local disk - no need for uploading. But with some minor changes we can run it in the cloud too

Best guess they use an integral images to do the zooming

Good guess! We do some cool subsampling (certainly on big files) to be able to show a lot of datapoints.

Maybe the financial industry is interested in it, they have time series too

Site is broken (broken images). Fix it before you launch.

Please don't be a jerk on HN, especially not in Show HN threads, which have special rules: https://news.ycombinator.com/showhn.html.

Jerk might be a strong word, I think he/she might have been trying to help, but it comes across a bit terse.

It's true, I felt it was a bit of a strong word even when posting it. But naked imperatives ("Fix it before you launch") come across as aggressive in casual English nowadays.

sorry for the tight tone. I was clicking on the link on the first load and the images are all broken. It worked after several refreshes. Thanks for heads up!

Which ones are broken?

Reply on news.ycombinator.com