HNHacker News
TopNewBestAskShowJobs

Vasco2006

17 karma · joined May 7, 2020

submissionscomments
Vasco2006··on Show HN: Deepmark AI- LLM assessment tool for task-specific metrics on your data
Hey Everybody ,

We are excited to open source an amazing tool that we've been using internally for some time and which helped us a lot in most of our AI projects - Deepmark AI! Deepmark AI is a benchmarking tool for GenAI builders that enables assessment of several large language models (LLM) on various extrinsic (task-specific) metrics (e.g. accuracy, relevance, failure rate, latency, etc) on your own data, so your AI applications have predictable and reliable performance.

Based on IngestAI Labs' 40'000 users and dozens of AI projects we completed, we clearly see that GenAI builders and organizations need to be able to assess large languag models (LLM) on their own data to deliver predictable and reliable results that balance accuracy, precision, recall (the model’s ability to correctly identify positive cases within a given dataset), as LLM models can produce different answers to the same prompts, impeding the user’s ability to assess the accuracy of outputs.

To address this challenge of reliability, we (IngestAI Labs) have developed and open sourced Deepmark AI - a benchmarking tool, that enables assessment of several large language models (LLM) on various extrinsic (task-specific) metrics on your own data. It has pre-built integration with leading Generative AI APIs such as GPT-4, Anthropic, GPT-3.5 Turbo, Cohere, AI21, and others.

DeepMark AI is a tool specifically designed for AI builders.This solution focuses on iterative assessment of extrinsic metrics to identify predictable, reliable, and cost-effective Generative AI models based on the unique needs of a particular use case and data. Deepmark AI offers comprehensive assessment of various important performance metrics such as:

Question answering accuracy Text classification accuracy PII recognition accuracy Named entity recognition (NER) accuracy Summarization quality (Relevance) Sentiment analysis accuracy Cost analysis Failure rate Accuracy Latency

Deepmark AI empowers organizations to make informed decisions when navigating through the most important performance metrics of Large Language Models.

So what are you waiting for? SignUp to IngestAI today and take your customer support to the next level!

Follow our roadmap on Github : Star us / watch us / fork us: https://github.com/IngestAI/deepmark

And join our community at : Discord : https://discord.gg/kMpbueJMtQ Twitter : https://twitter.com/ingestaiio

Hope you loved it.

Team IngestAI Labs / Deepmark AI

Vasco2006··on Show HN: [Embedditor] - the Open-source Microsoft Word for Vector Embeddings
We are thrilled to open source Embedditor today.

Embedditor is the open-source MS Word equivalent for embeddings that helps you get the most out of your vector search, while saving up to 30% on vectorisation and storage.

This solution is inspired by the experiences of over 30,000 IngestAI users. Our insights revealed a common bottleneck in AI and LLM-related applications, one that goes beyond LLM hallucinations or token limits, which are far easier to resolve. The prevailing issue lies in the GIGO (garbage in, garbage out) principle. With no one-size-fits-all approach to chunking and embedding, certain models excel with individual sentences, while others thrive on chunks of 250 to 500 tokens. Blindly splitting chunks by the quantity of characters or tokens, and embedding content without normalization and with up to 40% of redundant noise (such as punctuations, stop-words, and low-relevance frequent terms) often leads to suboptimal vector search results and low-performing LLM-related applications using semantic or generative search. The issue was consisting in trying to enhance vector search using existing technologies, which proved to be as challenging for our users, as creating an outstanding document using a basic .txt format. We decided to address the root problem, so we developed Embedditor - the Microsoft Word equivalent for embedding pre-processing, enabling with no background in data science or technical skills to improve performance of their vector search capabilities while saving up to 40% on embedding and storage. We've made Embedditor open-source and accessible to all because we genuinely believe that by improving vector search performance and boosting cost-efficiency simultaneously, Embedditor may have significant impact on current NLP and LLM industry.

Hope you loved it. and we would love to hear your usecases & feedbacks.