HNHacker News
TopNewBestAskShowJobs

chaotic-good

182 karma · joined August 12, 2013

submissionscomments
chaotic-good··on Measuring execution performance of C++ exceptions vs. error codes
Error checks are almost free and can be considered zero cost because branch predictor works really well with error checks (because errors are rare). Branch predictor uses some space in L1 cache to store tables so it's not 100% free but very close to that. Exceptions aren't zero cost either but you're paying for them only when an error occurs most of the time.
chaotic-good··on Online databases dropping like flies, with 10,000 falling to ransomware
>> No(Auth)SQL.

Absence or presence of auth is irrelevant. You're database servers, message queues and other infrastructure shouldn't be accessible from the internet. No auth protocol can protect you from this.

chaotic-good··on UDP vs TCP
>> In most genres people would rather have the simulation pause when there is packet loss, and then run fast to catch up rather than have unreliable data about player locations.

I thought that modern games use interpolation to compensate for this, because network lags are always present in WAN.

chaotic-good··on Gorilla: A Fast, Scalable, In-Memory Time Series Database (2015) [pdf]
I'm working on my own solution for a while (about 3 years) - https://github.com/akumuli/Akumuli. One of the greatest things about it is that there is no need for preconfigured rollups. The storage is based on B+Tree and some aggregates are stored inside inner nodes of the tree.
chaotic-good··on Gorilla: A Fast, Scalable, In-Memory Time Series Database (2015) [pdf]
https://en.wikipedia.org/wiki/Unevenly_spaced_time_series You can call that _logging_ if you want, but people will ask for it anyway.
chaotic-good··on Gorilla: A Fast, Scalable, In-Memory Time Series Database (2015) [pdf]
> I do not understand the general obsession with specialized TS stores

Specialized TS stores exist for decades (process historians, tickerplants, and so on).

chaotic-good··on Modern garbage collection
Calling malloc is not that expensive if you're using jemalloc or any other modern allocator. And with generational GC allocation is not just a register increment because Java has mandatory object initialization and all objects will be pushed to 2nd gen heap eventually. std::shared_ptr is quite fast when been used correctly (no excessive sharing between threads and no abusing).
chaotic-good··on Show HN: SummitDB – In-Memory NoSQL DB
I've found several submitted issues that described this problem. All this issues are closed now so maybe I was wrong about Bolt.
chaotic-good··on Show HN: SummitDB – In-Memory NoSQL DB
BoltDB can be corrupted on crash (kill -9 or hard reset) at least this was the case when I gave it a try. Any database can but in that case that was easy to achieve.
chaotic-good··on Show HN: SummitDB – In-Memory NoSQL DB
You can try LMBD - https://symas.com/products/lightning-memory-mapped-database/ In my opinion there is exactly two production ready embedded databases - SQLite and LMDB. At least this two databases have decent crash recovery mechanisms (I've tested both). I haven't tested all of them, maybe ForestDB and BerkleyDB are good as well.
chaotic-good··on Cap’n Proto
I'm glad that I was wrong about it.
chaotic-good··on A Review of Time Series Databases
It is usually a good idea to discard late writes. It should be possible to write data with some time-stamp skew but if skew is too large data should be discarded as erroneous. It's totally depends on an application so.
chaotic-good··on Cap’n Proto
I think that compression is a must for the serialization library. Protobuf uses almost twice less memory than Cap'n Proto. Using an external compression is not an option in some cases. E.g. consider building tcp-server that communicates with thousands of clients simultaneously. Each client connection will have its own LZ4 context that should be heap allocated. I believe it's about 16KB per connection + buffers. This results in large memory consumption and a lot of random memory access and TLB misses.
chaotic-good··on Websocket Shootout: Clojure, C++, Elixir, Go, Node.js, and Ruby
With C++ one can go far further than that. It's the matter of time spent on the solution. With all that other languages there is a performance ceiling that can't be breached.
chaotic-good··on TimeSeriesDB
I'm working on my own time-series database server here - https://github.com/akumuli/Akumuli The most limiting factor is not the query language or network its a storage engine itself (and not the HDD/SSD write speed). I've discovered that write speed in akumuli is limited mostly by compression algorithm speed and parallelization. Solution above is bad because they're have only time-based index. It's impossible to read only one time-series without decompressing entire chunks that contains a lot of data. This solution is write optimized but it will introduce a lot of read amplification. It's not suitable for interactive applications. I've done the same mistake too but moved away from this design.
chaotic-good··on Qt Creator 4.0.0 released
I find QtCreator superior to CLion. It's faster and more polished, has better vi-emulation and clang ecosystem integration.
chaotic-good··on InfluxDB v0.10 GA with hundreds of thousands writes/sec, 98% better compression
Even simple delta+RLE can squeeze increasing unix timestamps up to few bytes. I'm talking about floating point data compression. Even the specialized state of the art algorithms can't guarantee you 2.2 bytes per float in general.
chaotic-good··on InfluxDB v0.10 GA with hundreds of thousands writes/sec, 98% better compression
It depends greatly on the shape of your data, but with regularly spaced timestamps at second level precision and float64s, we’ve seen compression reduce each point down to around 2.2 bytes per point.

AFAIK you're using float64 compression scheme from "Gorilla" paper. 2.2 bytes per data point is possible with it but only on data that doesn't utilize full double precision (example: small integers converted to float64). You should compare compression algorithm used by TSM storage engine with zlib or any other general purpose compression algorithm, otherwise this number will be meaningless.

chaotic-good··on OpenTSDB – A Distributed, Scalable Time Series Database
OpenTSDB is really nice but it has some drawbacks:

1. Limited timestamp precision (because of storage schema).

2. Dependency on HBase.

chaotic-good··on Have Static Languages Won?
No, they didn't. This article is narrow minded and prejudiced. Of cause if we're talking about programming in general - static typing languages are often a better option. But programming doesn't ends there, there is technical computing, statistical computing, data analysis and so on. How about beating Python + numpy + pandas in data crunching or maybe R in statistics? In this fields underlying data types is almost always very simple, and because of this you simply can't benefit much from static typing.

This is not about expressiveness (the article author is wrong here IMO). It's all about flexibility and (more important) interactivity. I can do useful things in ipython console or in ipython notebook in the browser. It's like a tiny lab for small and moderately sized things, data munging, analysis and visualization, prototyping and all this stuff.

There is dynamic languages like Python, R and even MATLAB that's not going to disappear anytime soon. There is a new dynamic language called Julia that's only starts getting traction but IMO it's already more interesting then Rust, Go and Nim combined together.

chaotic-good··on Elasticsearch as a Time Series Data Store
If you can predict next value in the sequence you can achieve better compression (xor predicted value and actual, not previous and next - http://users.ices.utexas.edu/~burtscher/papers/dcc06.pdf), but I don't think that Prometheus developers are using this.

And BTW, I don't believe in 2.5-3.5 bytes per data-point somewhere outside synthetic benchmarks.

chaotic-good··on Time series database Graphite seems to be falling into disfavor
> It can still be pointing to the raw data.

Yep. Each SAX word should be mapped to the list of seriesid:timestamp pairs. This list is often referred as postings list in information retrieval. The resulting data-structure is an inverted index. SAX and iSAX papers describes inverted index variant (really bad one) based on folder structure but one can use convenient IR tools for this.

chaotic-good··on Time series database Graphite seems to be falling into disfavor
You can't retrieve original time-series data from SAX storage because of normalization. To query time-series data by content (approximate 1-NN, motif discovery, etc) you need inverted index.
chaotic-good··on Time series database Graphite seems to be falling into disfavor
Can't agree more. Time-series data analysis should be considered the most important, not storage. Storage is a solved problem. Let's imagine that we're getting ECG data from many patients. What most time-series databases is able to do with this data? They can draw a nice graph. OK. They can resample this data. Well.. this is mostly meaningless for ECG. Can graphite/opentsdb/whatsoever find specific patterns indicating a disease (for example T-wave inversion)? No, but this is OK, because they're not designed for this. We should build new tools to solve new problems, not the old ones (like monitoring in DevOps).
chaotic-good··on Time series database Graphite seems to be falling into disfavor
SAX requires different type of storage. You can store SAXified time series in ElasticSearch, or Solr but time-series database doesn't fit for this. Time-series databases should be able to generate SAX representation.
chaotic-good··on Time series database Graphite seems to be falling into disfavor
I'm building my own time-series database for some time.

https://github.com/akumuli/Akumuli

It's a standalone solution without dependencies on other services or databases. It can handle more than a million data points per second and uses fixed amount of memory to store data on disk but without Graphite's shortcomings (compression is used for everything and timestamps is stored with nanosecond precision).

I'm focused mostly on realtime time-series analysis. Akumuli has built-in anomaly detector: EWMA, Holt-Winters and sketch based methods. It can generate different time-series representations, like SAX and PAA. DTW and correlation search is in progress now.

chaotic-good··on Time Structured Merge Tree: From LSM Tree to B+Tree and Back Again
SCADA systems usually handles small amount of data (because data comes from real sensors, this sensors is expensive and you simply can't have millions of them).
chaotic-good··on Time Structured Merge Tree: From LSM Tree to B+Tree and Back Again
You can't compare this solution to InfluxDB or other time-series databases out there because you're loosing information here. You can't store timestamps with arbitrary precision, and all the data is resampled, there is no way to store raw data. This is not acceptable in many cases.
chaotic-good··on Ask HN: Who is hiring? (August 2015)
Are you considering remote candidates?
chaotic-good··on Time-Series Database patented by GE
We have used such systems in industrial control systems for decades. TSDB is integral part of any modern SCADA system. This patent is a nonsense.
← PreviousPage 2 of 3Next →