There are designs for untethered structures in orbit that could function with current technology, e.g. https://en.wikipedia.org/wiki/Skyhook_(structure)
468 karma · joined March 18, 2010
The current approaches to creating "golden datasets" for LLM testing are either infrastructure-heavy (observability-based) or time-consuming (manual creation). Many teams end up relying on intuition-based development, which leaves crucial questions unanswered about model selection, edge cases, and performance optimization.
Our solution generates comprehensive, realistic test datasets tailored to your specific use cases. You provide context (existing examples, domain information, or system prompts), and it creates robust test data that helps you:
- Balance distribution across your test suite including edge cases - Evaluate prompt effectiveness across various scenarios - Compare and optimize model selection - Identify and handle edge cases systematically - Assess performance across diverse use cases - Support rapid R&D experimentation with different data shapes
We're launching this as the first data studio focused on giving AI engineers both speed and precision in their development workflow. If you're building AI products, we'd love your feedback on our approach to making LLM testing more reliable and efficient.
Check out what we’re building at https://www.withcoherence.com/
Coolify and cnc are very different technical solutions. Coolify is a server you deploy to a VM that then can schedule workloads onto that VM, managing features like ingress and updates. cnc is a client-side CLI that schedules workloads into managed cloud services like lambda, cloud run, ECS, or Kubernetes. It orchestrates public cloud provided services instead of providing them itself (e.g. RDS vs. MySQL in a docker container on a VM). The trade-offs here are too big for a comment and both are a great fit for different use cases. We dive in a bit deeper with our POV here: https://www.withcoherence.com/post/the-2024-web-hosting-repo...
Coolify is awesome software, and alongside similar tools like Caprover, Dokku, and Cloud66, it has its role. But for business use-cases I believe that giving up managed cloud services is too big a leap to make sense, and that a middle-ground approach will win in the long term.
You can learn more in our docs:
AWS Lambda - https://docs.withcoherence.com/reference-architectures/aws/l... GCP Run Lite - https://docs.withcoherence.com/reference-architectures/gcp/r...
The tagline clearly doesn't say what we want here, but the idea is that we are a framework that sits on top of IaC tools like the ones listed above, and makes them easier to use. Under the hood, cnc uses terraform/opentofu to actually manage infra, the cnc layer adds new concepts like environments onto those tools as well as providing a much simpler, higher level configuration with cnc.yml, while still enabling you to edit the templates directly so you're not limited in the infra you can deploy for your app. In this way, we're the first tool of that kind, sort of like a "PaaS in your own cloud" but we expose the plumbing as a CLI.
Thanks again!
There are some obvious "tooling" considerations that impact developer experience. For the most part, these are related to how much devs can "self-service" the various parts of the SDLC, and how good the user experience of that service flow is. For teams that want to get a great experience without reinventing the wheel, internal developer platforms like what we're building at Coherence (withcoherence.com) are a great answer.
We've also got a free 1-minute PR velocity scorecard that you can try: https://www.withcoherence.com/post/measure-your-pr-efficienc...
At Coherence (withcoherence.com) - I'm a cofounder - we deliver a developer platform that lets use use either cloud in a developer-friendly way. It also gives you some protection against lock-in to either cloud.
We generally see a ton of velocity increase in dev from teams that adopt the Cloud IDE alongside the rest, as exhibited by lots of comments. Internally, we dogfood them 100% and the team would never go back to the old "local-first" ways. So we are using them and love them. The key qualities we love are ephemerality, parallelism, and accessibility. But in general there is a lot of resistance for various reasons, as seen in many comments here.
We wrote up some of our POV in our docs here: https://docs.withcoherence.com/docs/explanations/remote-deve...
[I'm a cofounder of Coherence]
Would be cool to try using this extension on top of RDS or Cloud SQL to offer the same experience without the overhead of another SaaS tool!
Immutable or chain-of-trust based host OS (e.g. nix or iOS)
Minimal software installed (including docker, which itself is heavy and full of vulns or opens the door to them)
Do everything on ephemeral remote environments where the configuration is stored in reviewable tools (e.g. GitHub) and the state can be wiped at will. This means you reduce your surface area for persistent malware to supply chain and network attacks, which require careful practices to avoid but which are well-understood
Remote envs are preferred to local virtualization (e.g. quebes) because they lend themselves to team use and sharing more, and so are more likely to be widely adopted and collectively improved. Also easier to create different hardware configurations as needed (when you need a bigger GPU temporarily), as well as different environment types - e.g. always-on previews for QA testing. Also eliminates persistent paths in the local OS for malware storage
I consider the "best-in-breed" tools with this shape to break into the following categories:
"Infra Abstractions": DuploCloud, Massdriver, Stacktape "PaaS in your own cloud": Nullstone, Zeet, Quovery, Porter, Architect.io, FlightControl "SDLC Platform for Teams": Coherence (withcoherence.com) - the key difference here is in how CI/CD (incl integration tests) are handled, how development environments are configured alongside other environments, and how production is segregated from other environments.
This a somehow both a crowded space and an emerging one. But we believe that even with the diversity of choice that there are clear reasons that most teams will by (vs. build) their internal dev platform in the future: namely the cost to buy vs. to build/maintain, and the productivity from better tools.
Other products to check out: codesphere, stackblitz, Replit