warangal

I recently came across https://www.kernel.org/pub/linux/kernel/people/paulmck/perfb... , it draws from author's experience about applying optimizations in linux codebase .

I also like this very readable paper https://xnning.github.io/papers/perceus.pdf, about reducing the costs of reference counting in context of functional languages.

VITS is such a cool model (and paper), fast, minimal, trainable. Meta took it to extreme for about 1000 languges.

It seems like you have been working on this application for sometime, i will go through your code , but could you provide some context about upgradations/changes you have made, or some post describing your efforts.

Cool nonetheless!

I may be wrong here, but blog-post seems AI written, with repetition of sequences like "the inference pipeline was rebuilt using architecture-aware fused kernels, optimized scheduling, and dis-aggregated serving". I don't know what that means without some code and proper context.

Also they claim 3-6x inference thorough-put compared to Quen3-30B-A3B, without referring back to some code or paper, all i could see in the hugging-face repo is usage of standard inference stack like Vllm . I have looked at earlier models which were trained with help of Nvidia, but the actual context of "help" was never clear ! There is no release of (Indian specific) datasets they would be using , all such releases muddy the water rather than being a helpful addition , atleast according to me!

pretty cool work! Though it leaves me wondering if coreboot/bios code can directly interface with thermal-management and battery controller , shouldn't it be feasible to improve upon battery life by exposing some interface to OS, like apple laptops ?

I was also reading through lobsters Memory management, which (i think) currently implements "borrow first" semantics, to do away with a lot of run-time reference counting logic, which i think is a very practical approach. Also i have doubts if reference counting overhead ever becomes too much for some languages to never consider RC ?

Tangentially, i was experimenting with a runtime library to expose such "borrow-first" semantics, such "lents" can be easily copied on a new thread stack to access shared memory, and are not involved in RC . Race-conditions detection helps to share memory without any explicit move to a new thread. It seems to work well for simpler data-structures like sequence/vectors/strings/dictionary, but have not figured a proper way to handle recursive/dynamic data-structures!

How do you market it, through social-media or are there dedicated channels for sharing awareness for such Mac Apps, if you don't mind sharing?

I work on an image search engine[0], main idea has been to preserve all the original meta-data and directory structure while allowing semantic and meta-data search from a single interface. All meta-data is stored in a single json file, with original Path and filenames, in case ever to create backups. Instead of uploading photos to a server, you could host it on a cheap VPS with enough space, and instead Index there. (by default it a local app). It is an engine though and don't provide any Auth or specific features like sharing albums!

[0] https://github.com/eagledot/hachi

Currently (Semantic) ML model is the weakest (minorly fine-tuned) ViT B/32 variant, and more like acting as a placeholder i.e very easy to swap with a desired model. (DINO models have been pretty great, being trained on much cleaner and larger Dataset, CLIP was one of first of Image-text type models !).

For point about "girl drinking water", "girl" is the person/tagged name , "drinking water" is just re-ranking all of "girl"s photos ! (Rather than finding all photos of a (generic) girl drinking water) .

I have been more focussed on making indexing pipeline more peformant by reducing copies, speeding up bottleneck portions by writing in Nim. Fusion of semantic features with meta-data is more interesting and challenging part, in comparison to choosing an embedding model !

Hi, Author here!

I have been working on this project for quite some time now. Even though for such search engines, basic ideas remain the same i.e extracting meta-data or semantic info, and providing an interface to query it. Lots of effort have gone into making those modules performant while keeping dependencies minimal. Current version is down to only 3 dependencies i.e numpy, markupsafe, ftfy and a python installation with no hard dependence on any version. A lot of code is written from scratch including a meta-indexing engine and minimal vector database. Being able to index any personal data from multiple devices or service without duplicating has been the main them of the project so far!

We (My friend) have already tested it on around 180gb of Pexels dataset and upto 500k of flickr 10M dataset. Machine learning models are powered by a framework completely written in Nim (which is currently not open-source) and has ONEDNN as only dependency (which has to be do away to make it run on ARM machines!)

I have been mainly looking for feedback to improve upon some rough edges, but it has been worthwhile to work upon this project and includes code written in assembly to html !

Using `zig-cc (clang)` to set a particular `LibC` version is one of the best decisions i have made, and saved me from those meaningless libC mismatch errors!

I know it may be not what you are looking for, but most of such models generate multiple-scale image features through an image encoder, and those can be very easily fine-tuned for a particular task, like some polygon prediction for your use case. I understand the main benefit of such promptable models to reduce/remove this kind of work in the first place, but could be worth and much more accurate if you have a specific high-load task !

Pretty cool project!, I have been also trying to do something similar with very limited (abstract) OPs akin to fundamental computer instructions. Just using the numpy backend for now to test theory, but neat thing is that most of complexity lies in the abstract space like deciding which memory accesses could be coalesced even before generating the final code for a specific backend! As far as i know most of DL compilers struggle to generate optimum code, as model starts getting bigger and bigger . Halide project was/is a very cool project that speed up many kernels just by finding better cache/memory access pattern. If you happen to share more insights about your projects through blog-posts or whitepaper that would be really helpful.

A bit tangential statement, about parakeet and other Nvidia Nemo models, i never found actual architecture implementations as pytorch/tf code, seems like all such models, are instant-ized from a binary blob making it difficult to experiment! Maybe i missed something, does anyone here have more experience with .nemo models to shed some more light onto this?

Nice write up! It is about speeding up underlying hash-table for a `cuckoo filter` , as in false-positives are allowed, unlike data-structure like dictionary, which makes sure that false-positive rate is 0.

Whenever i need to use dictionary/hash-table in a hot loop where maximum possible iterations are known, i use a custom hash-table initialized with that value, and in some cases, standard library implementation may store `hash` along with `key` and `value` in case it needs to `resize`, in my case i don't store the `hash` which leads to speed up about 15-20% due to less cache misses, as there would be no resize possible!

Also `cuckoo filters` seems to use `robinhood hashing` like strategy, using in a hot-loop it would lead to a higher insert cost ?

Thanks for explaining it, given you are writing it from scratch it gives you a lot of control in modelling a particular feature!

I did bookmark this project a few months ago but couldn't spend time to understand more about it. I wasn't aware of documentation, which should now make it easy to start with. Thanks for putting a lot of work in documentation!

Are you trying to do this message passing using Nim channels ? For my use-cases, i always had to resort to just passing `pointers`, to prevent any copying, most of time i just divide the problem to use independent memory locations to write to avoid using locks, but that is not general pattern for sure. For read only purposes i just use `cursor` while casting to appropriate type. If you find a useful pattern, please share.

for images readme is at: https://github.com/eagledot/hachi/tree/main/images/readme.md with more than enough details! It is supposed to be a search engine for all modalities, for now `images` are supported !

For demo, i don't have much resources to host it, would a video showcasing features help ?

Also for windows there is a portable app at https://github.com/eagledot/hachi/releases/download/v1.3/hac...

Disclaimer: I work on such a project[0]

I think a combination of CLIP and some face-recognition may solve your issues! It just takes a path to your directory, and can index all the images while preserving your folder hierarchy along with a high quality face-clustering. Each image indexing takes about 100ms on a cpu. Every combination can then be mixed and matched, from a single interface, It doesn't take much to try as dependencies are very minimal . There is a self contained app for windows too. I have been looking for feedback, so just plugging it here in case some one has a use case.

[0] https://github.com/eagledot/hachi

Yes, trigram mainly but also bigram and/or combination of both are used generally to implement fuzzy search, zoekt also uses trigram index. But such indices depend heavily on the content being indexed, for example if ever encounter a rare "trigram" during querying not indexed, they would fail to return relevant results! LSH implementations on the other hand employ a more diverse collection of stats depending upon the number of buckets and N(-gram)/window-size used, to compare better with unseen content/bytes during querying. But it is not cheap as each hash is around 30 bytes, even more than the string/text being indexed most of the time ! But its leads to fixed size hashes independent of size of content indexed and acts as an "auxiliary" index which can be queried independently of original index! Comparison of hashes can be optimized leading to a quite fast fuzzy search .

I myself have been working on a personal search engine for sometime, and one problem i faced was to have an effective fuzzy-search for all the diverse filenames/directories. All approaches i could find were based on Levenshtein distance , which would have led to storing of original strings/text content in the index, and neither would be practical for larger strings' comparison nor would be generic enough to handle all knowledge domains. This led me to start looking at (Local sensitive hashes) LSH approaches to measure difference b/w any two strings in constant time. After some work i finally managed to complete an experimental fuzzy search engine (keyword search is a just a special case!).

In my analysis of 1 Million hacker news stories, it worked much better than algolia search while running on a single core ! More details are provided in this post: https://eagledot.xyz/malhar.md.html . I tried to submit it here to gather more feedback but didn't work i guess!

Embeddings capture a lot of semantic information based on the training data and objective function, and can be used independently for a lot of useful tasks.

I used to use embeddings from the text-encoder of CLIP model, to augment the prompt to better match corresponding images. For example given a word "building" in prompt, i would find the nearest neighbor in the embedding matrix like "concrete", "underground" etc. and substitute/append those after the corresponding word. This lead to a higher recall for most of the queries in my limited experiments!

It is a small DSL written using macros at https://github.com/mratsim/Arraymancer/blob/master/src/array....

Nim has pretty great meta-programming capabilities and arraymancer employs some cool features like emitting cuda-kernels on the fly using standard templates depending on backend !

It is quite possible B variant is not enough for some scenarios, earlier version also included the videos search, frames used for indexing were sometimes blur (not having fine-details) and these frames generally would have higher score for naive Natural language queries. I only tested with B variant.

But i resolved that problem upto a point by adding a Linear layer trained to discard such frames, and it was less costly than running a bigger variant for my use case.

I keep forgetting to put a benchmark for a standard flickr30k like dataset! But a ballpark figure should be about 100ms per image on a quad-core CPU, i also generate an ETA during indexing and provide some meta-information to make it easy to get information about data being indexed.

I think image-encoder from CLIP (even smallest variant ViT B/32) is good enough to capture a lot of semantic information to allow natural language query once images are indexed. A lot of work actually goes into integrating with existing meta-data like local-directory, date-time to augment NL query and re-ranking the results.

I work on such a tool[0] to enable end to end indexing of user's personal photos and recently added functionality to index Google Photos too!

[0] https://github.com/eagledot/hachi

Thanks for this resource! Even though i have been able to collect enough resources for Neon/simd programming but having a proper guide is indispensable and arm official documentation doesn't make it easy in any way!

Just wanted to say thanks for this and all other such detailed posts!. Found your blog a few weeks ago and have been going through posts running NuttX RTOS on these SBCs, but admittedly feel overwhelmed with sheer amount of information packed together !

This is a very nice writeup, and clearly dictates performance benefits to be gained from using underlying hardware properly.

Couple of things i noticed. ``` # calculating dot product

a = np.random.randn(1,1536).astype("float32") b = np.random.randn(1536, 1).astype("float32")

result = np.matmul(a * b) ~1.1 microsecond ```

My system configurations are: Intel(R) Core(TM) i5-8300H CPU @ 2.30GHz 2.30 GHz, (quad core) AVX256 instructions are supported.

problem lies in calculating the magnitude of vectors, which i guess scipy is doing as ``np.sum(np.square(a)``.Any numpy routine like ``matmul`` using Blas library would be very very fast, but all other routines are just simple C for loops. Magnitude calculation by passing numpy buffer to a C extension takes about ``1.4`` microseconds. using SIMD instructions for magnitude calculation is very straightforward and i get a speed up of about 8x.

For critical calculations you are almost always better off by passing numpy buffer to C code and fusing operations there, and optionally speed up code using SIMD instructions based on hardware you want to target.

It should be ``scipy.spatial.distance.cosine``, not ``scipy.distance.spatial.cosine`` in your blog !

I sometimes find GUIs equivalent to comfort-food as you can just stroll through a gui interface without lot of cognitive overload that may come from a rarely used CLI.

A bit tangential to current discussion, but i came across CUDATEXT editor a few months ago here which provides a single file python API to let me use arbitrary GUI elements like MENU, INPUTS etc which editor is itself using. I currently generate my blog from editor itself with configuration done through these simple GUI elements.

Indeed it was a great read about what goes behind the scenes for maintaining such packages and a problem generally every developer thinks about when such an extension has to be shipped along .

I still wonder if targeting Python's LIMITED C API wouldn't help in this case. I use a tool (for Nim) which seems to target that Limited API and solves the problem of specific python version mismatching at-least for my specific code. I never had to upgrade a pure C/Nim extension due to python version !

Using zig-cc along with a fixed GLIBC (2.27) and generic CPU flag also made it possible to target linux ecosystem in case user is not a developer and just use the compiled extension shipped with the package.