same thing happened to me. was on codex for about a month, felt that exact shift, ended up going back to claude max. not sure if it's routing or tuning but something changed and prompting around it didn't help.
6 karma · joined May 9, 2021
I was spending over 5 hours manually testing my Agentic AI application before every patch and release. While automating my API and backend tests was straightforward, testing the actual chat UI was a massive bottleneck. I had to sit there, type out prompts, wait for the AI to respond, read the output, and ask follow-up questions. As the app grew, releases started taking longer just because of manual QA.
To solve this, I built Mantis. It’s an automated UI testing tool designed specifically to evaluate LLM and Agentic AI applications right from the browser.
Here is how it works under the hood:
Define Cases: You define the use cases and specific test cases you want to evaluate for your LLM app.
Browser Automation: A Chrome agent takes control of your application's UI in a tab.
Execution: It simulates a real user by typing the test questions into the chat UI and clicking send.
Evaluation: It waits for the response, analyzes the LLM's output, and can even ask context-aware follow-up questions if the test case requires it.
Reporting: Once a sequence is complete, it moves to the next test case. Everything is logged and aggregated into a dashboard report.
The biggest win for me is that I can now just kick off a test run in a background Chrome tab and get back to writing code while Mantis handles the tedious chat testing.
I’d love to hear your thoughts. How are you all handling end-to-end UI testing for your chat apps and AI agents? Any feedback or questions on the approach are welcome!
Generate a Blueprint Creates a basic structure and suggests necessary components.
Firebase Integration: Likely designed for seamless integration with Firebase services (though not deeply explored in the review)
Public Link Sharing: Enables sharing a public link to the prototyped application
So, I built WorkToDoToday.
Imagine typing or saying:
"Remind me tomorrow to send a follow-up email to Customer XYZ regarding our PoC next month."
And instantly, the app:
* Creates a task with a due date of tomorrow * Tags it to Customer XYZ with a clear title and description * Lets me see all related tasks, notes, and past interactions in one place
Why It’s Different:
* No forms or complex workflows—just type naturally, and the app organizes everything * Manage tasks through chat or UI, whichever works best for you * See all interactions tied to a contact for a full view of your relationships
It started as a personal tool, but after sharing it, I realized a lot of founders and busy professionals needed this too. If you struggle with managing follow-ups like I did, check it out: https://www.worktodo.today Would love to hear your thoughts!
About a month ago, I was lucky enough to participate in the Falcon LLM Hackathon, a crazy 5-day sprint to build something innovative with a cutting-edge language model. My team and I had one goal in mind: to make it easy for anyone to build conversational AI apps using their own data.
After countless late nights, a lot of debugging, and a few coffee-fueled “aha” moments, we built the first version of RAGGENIE. To our surprise, we won first prize! That win gave us the confidence to take RAGGENIE to the next level – an open-source low-code platform for building retrieval-augmented generation (RAG) applications.
Why RAGGENIE?
During the hackathon, we noticed how hard it was to integrate large language models with custom data sources, like databases or documents. Most existing tools required too much manual setup or technical expertise, making it hard for smaller teams to experiment with RAG tech. So we set out to simplify that process.
RAGGENIE enables teams to quickly spin up custom Copilets (our take on bots) that can chat with your data – without the heavy lifting. It’s perfect for small businesses or teams who want to get into GenAI without dealing with complex pipelines.
The Open-Source Decision
We’ve always believed in the power of open-source software. After the hackathon, we decided that RAGGENIE should be available to everyone, especially smaller teams who need flexibility and transparency. By making it open-source, we’re hoping to build a community of developers and businesses who can push this technology forward.
What You Can Do with RAGGENIE:
• Create low-code conversational AI apps that work with your own data (databases, documents, etc.) • Easily deploy retrieval-augmented generation tools without heavy technical setup • Customize it to fit your specific use case, whether it’s internal tools or customer-facing bots
The Road Ahead
We’re just getting started, but I’d love to hear from you! If you’re interested in the tech, have feedback, or want to contribute, check out our repo: github.com/sirocco-ventures/raggenie.
I’m also happy to chat about our experience at the hackathon, the challenges we faced, or how you can use RAGGENIE for your own projects. Feel free to ask anything!
The criteria we discussed were:
- Is the problem statement a good fit for LLM? - Why hasn't this been solved before without LLM? - Is there a better solution than LLM for this? - What's the industry, and what's the potential for expansion?
After this consideration, we shortlisted some use cases, but none were unique—we found they'd either already been solved or weren't necessary. So, we decided to return to our comfort zone: building a dev tool.
We chose to build a low-code platform where users could create all the use cases we'd discussed without coding. We found a few alternatives, but none quite met our needs.
We started creating wireframes while some of us began building the backend. In one day, we completed the full wireframe. Then, while discussing how to present the demo, we chose one of our previously discussed use cases: an insurance chatbot.
To expedite the demo, we decided to use Airtable as the destination and built agents for Airtable and PDF to fulfill this use case. This took us a day. By the next day, we had also developed the front end.
On day 5, we began integrating in the morning and finished before evening, then started recording the demo. We weren't entirely satisfied with some aspects, but then we learned the deadline had been extended by a day.
We worked for another 24 hours to improve it. Finally, we received an email announcing we were finalists. That's when we decided to take this project further.
After the final demo, we won first prize, which was incredibly exciting. We're now using the prize money to develop this project further and plan to open-source it once it's complete.
We called it RAGGENIE ( inspired by the magical Genie from Disney who could create things with a snap) , and with just five days on the clock, we knew we had to move fast. The result? A fully functional prototype that not only worked but also impressed the judges enough to win us first place!
RAGGENIE is designed to simplify the process of building RAGs, providing a low-code environment that makes it accessible even to those without deep technical expertise. The goal is to empower more teams to leverage AI in ways that are tailored to their unique needs.
But we’re not stopping here. We're now turning RAGGENIE into a full-fledged MVP, and in the spirit of collaboration and innovation, we plan to open-source it within the next two weeks.
Video : https://www.veed.io/view/16531375-a106-4523-82d5-bc1f318461d... Website : https://www.raggenie.com/
Onepane is a CloudOps tool designed to reduce the challenges faced by cloud users, whether they're small startups or large enterprises.
Cloud adoption starts small and grows with the organization. Initially, it's all about getting things up and running quickly. However, this approach often leads to non-standardized setups and a tangled web of cloud resources that become a pain to manage as your business scales.
We've lived through the struggles that different teams encounter when following their own cloud best practices, often chosen by individual engineers rather than being standardized across the organization. With multiple tools for Git, CICD, Monitoring, APM, and more, the absence of a centralized view complicates matters even further. Cloud's dynamic nature means changes happen frequently and without proper tracking, making it difficult to maintain a clear overview of resources, changes, and alerts along with their respective ownerships and impacted applications.
So, summarizing the challenges faced by cloud users:
- Divergent Best Practices: Different teams within an organization often adopt cloud best practices, which may not align with organizational standards.
- Tool Overload: There are many cloud tools out there for tasks like version control, CI/CD, monitoring, APM, and more. However, the data tends to be scattered across these tools, making it challenging to see the big picture.
- Dynamic Changes: The cloud is dynamic, and changes happen frequently. Keeping track of these changes and correlating them can be a nightmare.
- Lack of Visibility: The absence of a centralized tool for resource management, change tracking, and alerts with clear ownership and application context can lead to operational chaos.
In response to these issues, we've been developing Onepane, that leverages a Well-Architected Framework. While existing frameworks introduced by public cloud providers lack automation and real-time updates, we aim to bridge this gap with Onepane.
Onepane is designed to assist cloud customers in systematically achieving a well-architected infrastructure. Our first step involves addressing visibility challenges, gradually building towards an Automated Well-Architected Framework for Multi-Cloud environments.
From a product perspective, we are solving a problem for companies with limited technical resources to implement multiple tooling and policies for a well-architected and easy-to-govern cloud. We solve this by fixing the basics like naming and tagging and doing discovery in an assisted and automated fashion to give users visibility of changes, alerts, and accountability of cloud resources.
I am doing an AMA. Shoot your questions.
Also, I tried the tool from the demo, but it took too long to provide the results. However, I checked the examples and they seem good.
When I clicked on the "view demo" button, I was expecting a demo video. So, I think it would be better to have a "try it out" button there.
Most of the examples seem to be prompts for image generation. I was wondering if the tool is also capable of generating promt for contents. I will test it again later once the performance improves.
This entirely depends on what your website is built on and what your technical skills are. If you can provide more information on that, I would be happy to give you some good suggestions.