killme2008

Thanks for pointing it out! The footer has been updated.

Agree. Leveraging capabilities provided by cloud vendors is always a good idea. However, as the scale grows, cost inevitably becomes an issue. Third-party solutions often offer cost advantages because they support multi-cloud deployments and are optimized for specific scenarios.

Thanks for your question. GreptimeDB, like MongoDB, is schemaless. When ingesting data via OTEL or its gRPC SDKs, it automatically creates tables by inferring the schema and dynamically adds new columns as needed.

Secondly, I prefer wide tables to consolidate all sources for easy management and scalability. With GreptimeDB's columnar storage based on Parquet, unused columns don't incur storage costs.

Interesting idea. Edge AI for initial anomaly detection before sending data to the central system makes sense. However, how do we handle global-level anomalies that require a broader perspective?

That's a good question.

First of all, Kafka is still an event streaming platform and lacks database capabilities such as indexing and query optimization. Although ksql/Kafka Streams can perform computations based on consuming data, they require repeatedly pulling data, and there are no technologies like indexing to accelerate queries.

Secondly, dashboards and alerts in monitoring scenarios require a large number of views—these are the “known unknowns”. When dealing with “unknown unknowns” during exploration, it’s necessary to create views dynamically, which may result in a significant increase in the number of views. I’m not sure whether Kafka can handle such situations. Because monitoring requires greater real-time performance, it’s difficult to tolerate delays.

Yes, that's the LGTM(Loki, Grafana, Tempo, and Mimir) stack.

First, the main issue with this stack is maintenance: managing multiple storage clusters increases complexity and resource consumption. Consolidating resources can improve utilization.

Second, differences in APIs (such as query languages) and data models across these systems increase adoption costs for monitoring applications. While Grafana manages these differences, custom applications do not.

Agree. The new wide event pipeline should fully utilize cheaper storage options-object storage like S3. Includes both cold and hot data and maintains performance.

The extraction of raw data is the cooking or processing, and the results are ingested back into the same database. I think it's the approach described in this article.

This is something that has never happened before. When you need open source for promotion and fundraising, you embrace open source; when you want to make money, you kick open source to the curb. That's what we see.

Thank you for considering GreptimeDB! Feel free to give it a try next time, and we warmly welcome you to join our Slack community: https://greptime.com/slack

This architecture tries to address the key challenge with a disaggregated compute/storage architecture using object storage.

Delta Lake and Iceberg are great for managing offline, big data workloads like batch analytics and ETL on data lakes. They're not built for real-time queries or high-throughput writes.

GreptimeDB, on the other hand, is optimized for real-time, time-series data like IoT and observability metrics, but it's not suited for offline batch processing.

Choose Delta Lake or Iceberg for offline analytics at scale. Use GreptimeDB for fast, real-time data ingestion and queries. Together, they can complement each other: GreptimeDB handles real-time data, while Delta Lake/Iceberg manages offline analytics.

In the past, I also had doubts about whether a Rust database built on open-source components would be performance-limited, but this evaluation has dispelled our concerns. Apache DataFusion + Arrow + Parquet + OpenDAL, as a new data stack, have proven their potential.

AI has accelerated McLaren's analysis and application of massive data, in deep collaboration with Dell Technologies. Through custom hardware and efficient algorithms, the team has significantly enhanced its real-time decision-making capabilities. Hundreds of onboard sensors collect data in real time, with AI quickly filtering and providing insights. This not only aids instant strategy adjustments on the track but also speeds up car development iterations, driving a new revolution in F1 data and performance.

This manuscript gives a big-picture, up-to-date overview of the field of (deep) reinforcement learning and sequential decision making, covering value-based RL, policy-gradient methods, model-based methods, and various other topics (including a very brief discussion of RL+LLMs).

The three types of observability data(metrics, logs and trances) still follow the traditional approach. However, based on my hands-on experience with large-scale monitoring systems, these three types can be unified in terms of data models (AKA events), storage engines, and query languages (SQL). This unification would significantly improve both development and analysis efficiency while substantially reducing operational and maintenance costs

Have you heard of GreptimeDB? I am the creator of GreptimeDB. It's open-source and compatible with InfluxDB Line Protocol but only supports SQL as the query language. And it wants to support all types of time-series data, including metrics, logs, and events.

InfluxDB is a great project, but as you said, the v3 migration is painful and is still WIP. About the performance, We already did some benchmark GreptimeDB vs. InfluxDB v2, you can find it in our blog.

I believe the primary issue with using full-text search in PostgreSQL is its performance and scalability limitations.

Elasticsearch/OpenSearch offers more than just search functionality; they come with extensive ecosystems, including ELK for logging and processors for ETL pipelines. These platforms are powerful and provide out-of-the-box solutions for developers. However, as the article mentioned, scaling them up or out can be costly.

ParadeDB looks like is based on tantivy(https://github.com/quickwit-oss/tantivy) which is an impressive project. We leverage it to implement full-text indexing for GreptimeDB too (https://github.com/GreptimeTeam/greptimedb). Nevertheless, building full-text indexes with Tantivy is still resource-intensive. This is why Grafana Loki, which only indexes tags instead of building full-text indexes, is popular in the observability space. In GreptimeDB, we offer users the flexibility to choose whether to build full-text indexes for text fields, while always creating inverted indexes for tags.

Experience organizing variant Error types in a complex system.Topics including: (1) How to build a cheaper yet more accurate error stack to replace system backtrace; (2) How to organize errors in large projects; (3) How to print errors in different schemes to log and end users.

It's interesting and cool! Looks like the software stack is still running on the local server.

Looking for Prometheus and Grafana on the cloud? Check out GreptimeCloud! https://greptime.com/