vadman97

Our JS SDK provides a strict privacy mode that obfuscates the entire DOM for this use-case (see more here: https://www.highlight.io/docs/getting-started/client-sdk/rep...). Plus, our open-source nature means that companies with stricter data control requirements can self-host highlight and manage how the data is retained.

The open-source infra we use allows us to have a highly-available, scale-able system.

Kafka - buffers data ingest so that we never lose recordings. Redis - acts as a cache, optimized for write-heavy processing in our app. Postgres - stores metadata for the data we record. ClickHouse - stores logs and other time-series data that need to be aggregated.

We're looking to simplify the system and replacing InfluxDB + OpenSearch in the near future with ClickHouse as a more performance and cost-effective solution that has proven simpler to maintain.

For the cloud version (app.highlight.io), we currently don't support plugging into your own infrastructure. We also don't give you direct access to the database we use. For the self-hosted version however, you can use any infrastructure you want.

Yes, our self-hosted option can plug into existing ClickHouse servers and will set up the necessary schema for highlight to function. The infrastructure configuration is driven by the docker compose file. More info in our docs: https://www.highlight.io/docs/getting-started/self-host/self...

That's one way to summarize it :)

We do quite a bit more though with session replay and error monitoring for your full-stack web apps.

Haven't heard of it unfortunately. What kind of time series data are you thinking of storing? If you're building an application on top of it, you're better off using an OLAP DB like ClickHouse, InfluxDB, etc. to have access to lower-level constructs.

Interesting, thanks for sharing. Do you implement your own storage with Quickwit or can it be backed by a cloud storage solution like S3?

Our session replay is similar to OpenReplay but we've focused a lot of effort on making a cohesive backend debugging experience. Highlight sessions give you backend error monitoring and logging out of the box to make it easy to get to the root cause of a bug.

What does you ingest setup look like with BigQuery? Are you using something like fluentd to pipe logs over?

You get structured attribute search without a schema with Highlight. More in our docs: https://www.highlight.io/docs/general/product-features/loggi...

Highlight is a full-stack observability platform, so our recordings start from a user's frontend session and associate server-side errors and logs to provide debugging context. We've been building quite a lot to bolster the backend observability use case, including releasing the new logging product. We can't offer feature parity for backend traces and metrics yet, but we're planning to get there by the end of the year.

The new feature we're announcing is the server-side + browser-side logs ingest, query interface, and alerting engine. Our web session recording product is something we've had around but now closely integrates with logs to help you debug.

We have had folks in our Discord successfully run highlight on a Raspberry Pi with 4GB RAM, so our recommendation is definitely on the safe side. We're running multiple infra services in the docker stack (postgres, opensearch, clickhouse, influxdb, kafka, redis) that we would look to consolidate in the future to help with running on leaner instances.

Apologies for the inconsistency here. Our Docker resource requirement recommendations were out of date after some recent improvements (https://github.com/highlight/highlight/pull/5074 and https://github.com/highlight/highlight/pull/4993).

Just updated this: 8GB of RAM, 4 CPUs, and 64 GB of disk space.

For us, a significant reason was the ClickHouse cloud-hosted offering, rather than having to manage a cluster ourselves. Their use of S3 as the backing storage medium means that large-scale data retention is quite affordable.

A good comparison we've referenced: https://leventov.medium.com/comparison-of-the-open-source-ol...

Apologies for the confusion. We meant to link to our main readme: https://github.com/highlight/highlight/blob/main/README.md

highlight.io: https://github.com/highlight#our-values

Absolutely, makes sense. We'll make sure to make all security features we ship a first-class citizen of the open source version and available to all customers.

With the highlight.io client, we use an opt-in strategy to be privacy-first (since network request bodies + headers are often sensitive). We're not capturing network requests/responses unless you enable it and allow deeply configuring what should be captured [0].

[0]: https://www.highlight.io/docs/getting-started/client-sdk/rep...

Yup, planning to have a Ruby/Rails SDK shipped next week. We're also working on docs on our opentelemetry usage to make it easy for folks to contribute to future ones.

Here's the github issue if you'd like to add any considerations: https://github.com/highlight/highlight/issues/3999

Performance and ease-of-use of ClickHouse have been great so far! Looking forward to be leaning more in on ClickHouse for the logging product (https://www.highlight.io/docs/general/product-features/loggi...) we're building, and potentially consolidating other infra with ClickHouse.

That's great feedback; understood. What features would you consider "enterprise" in that case? Right now, nothing is paywalled in our self-hosted version, and we're still very much in the brainstorming phase here.

That's a great point! Just opened a PR to fix this: https://github.com/highlight/highlight/pull/4362

Switching this to https too since the our rrweb fork is now public (ssh reference was previously necessary for cloning a private fork).

Following up regarding the other points I missed. Compared to OpenReplay, we have similar functionality in our session replay, but we've focused a lot of effort on making a cohesive error monitoring (backend and frontend) experience. We closely link sessions with errors (stacktraces and associated metadata) and vice versa to make it easy to get to the root cause of a bug.

To demo your own sessions on highlight, you can easily spin up our next.js example app that has highlight installed: https://github.com/highlight/nextjs-13-sample. All you need to do is make an account on highlight.io and copy your project ID over into the sample app!

Hey! Sure thing. Our frontend session replay happens in two parts:

1. The HTML DOM Recording using rrweb (https://github.com/rrweb-io/rrweb) 2. The monkey patching of fetch, xhr, console methods, combined with data from `window.performance`

The rrweb repo has a good in-depth explanation of how the DOM recording works. In short, it captures the full HTML of a page when a user first loads it, then tracks DOM changes via the `MutationObserver` api. From our client, these are all shipped to our backend as serialized events. We process the events per 'user session', storing them temporarily in redis, then compressing a permanent payload once the session no longer is sending new events in the local filesystem or S3.

Monkey patching network and console methods allows us to capture request/response payloads, headers, status code, etc. We then combine that data with the `window.performance` api's notion of network requests to ensure we capture all requests (even ones that happened before our monkey-patch had time to apply), as well as to get precision timing data.

Happy to give more detail about parts of our stack if you're interested!

We've put a lot of effort already to make the OSS setup a single `docker compose` command despite the extensive infra we use under the hood. Longer term, we're planning to support other deployment strategies such as kubernetes, cloud templates, and openshift templates. Appreciate any feedback on the self-hosting end here or on our GitHub issues!

Data ingest challenges mounted quickly with a synchronous pipeline. Check out how we added Apache Kafka to help relieve our woes!