Thanks for these comments! Seems their measurement of Blender is off and we need some more in-depth benchmarks.
853 karma · joined June 4, 2017
This could possibly enable higher quality instant render previews for 3D designers in web or native apps using on-device transformer models.
Note the timings above were on an A100 with an unoptimized PyTorch version of the model. Obviously the average user's GPU is much less powerful, and for 3D designers it might be still powerful enough to see significant speedups over traditional rendering. Or for a web-based system it could even connect to A100s on the backend and stream the images to the browser.
Limitations are that it's not fully accurate especially as scene complexity scales, e.g. with shadows of complex shapes (plus I imagine particles or strands), so the final renders will probably still be done traditionally to avoid any of the nasty visual artifacts common in many AI-generated images/videos today. But who knows, it might be "good enough" and bring enough of a speed increase to justify use by big animation studios who need to render full movie-length previews to use for music, story review, etc etc.
I've got a few questions:
Could I ignore all flow runs that have an estimated cost above a certain threshold, so that the overall cost of optimization is less? Suppose I choose an acceptable level of accuracy and then skip some costlier exploration stuff. Is there a risk it doesn't find certain optimal configurations even under my cost limit?
How does the system deal with long context-length documents that some models can handle and others can't? Does this approach work for custom models?
Suppose I want to create and optimize for my own LLM-as-a-judge metrics like https://mastra.ai/en/docs/evals/textual-evals#available-metr..., how can I do this?
Are you going to flesh out the docs? Looks like the folder only has two markdown files right now.
Any recommendations for creating the initial QA dataset for benchmarking? Maybe creating a basic RAG system, using those search results and generations as the baseline and then having humans check and edit them to be more comprehensive and accurate. Any chance this is on the roadmap?
Cool stuff, I'm hoping this approach is more widely adopted!
The job of any context retrieval system is to retrieve the relevant info for the task so the LLM doesn't hallucinate. Maybe build a benchmark based on less-known external libraries with test cases that can check the output is correct (or with a mocking layer to know that the LLM-generated code calls roughly the correct functions).
To me this indicates that in many energy markets where renewables are sufficiently built out, the only factor for why we aren't using them more is the storage capacity and grid infrastructure to handle their variability -- and instead just running stable but dirty energy systems.
There are a lot of cool mechanical grid systems like gravity batteries (e.g. weights on a pulley system) [2] or compressed or liquid air storage systems [3] which provide cheaper storage at higher capacities and durations than electric batteries
1. https://reneweconomy.com.au/a-near-100-per-cent-renewable-gr... 2. https://www.youtube.com/watch?v=trA5s2iGj2A 3. https://www.youtube.com/watch?v=fjERw-Ol-_s
Would you ever extend your app to search the web or specialized databases for law, finance, science etc?
> (d) With respect to information dissemination, each agency shall— (3) provide adequate notice when initiating, substantially modifying, or terminating significant information dissemination products; (4) not, except where specifically authorized by statute— (A) establish an exclusive, restricted, or other distribution arrangement that interferes with timely and equitable availability of public information to the public; (B) restrict or regulate the use, resale, or redissemination of public information by the public
If these datasets were actually permanently deleted then the incident should be investigated by NARA [1]. The people responsible could be charged with a crime under 18 U.S.C. § 2071: > (b) Whoever, having the custody of any such record, proceeding, map, book, document, paper, or other thing, willfully and unlawfully conceals, removes, mutilates, obliterates, falsifies, or destroys the same, shall be fined under this title or imprisoned not more than three years, or both; and shall forfeit his office and be disqualified from holding any office under the United States. As used in this subsection, the term “office” does not include the office held by any person as a retired officer of the Armed Forces of the United States.
1. https://www.archives.gov/records-mgmt/resources/unauthorized...
Describes how Maurice Hilleman invented 40 vaccines, including for eight of the most common diseases in the US, over a 36 year career at the Walter Reed Army Medical Center and Merck & Co. His vaccines are estimated to save 8 million lives each year.
k8s, and the go runtime and network stack have been heavily optimized by armies of engineers at Google and big tech, so I am very suspicious of these claims without evidence. Show me the resource usage from k8s component overhead, and the 15 minute to 3 minute deploys and then I'll believe you. And the 200 file YAML or Helm charts so I can understand why in gods name you're doing it that way.
This post just needs a lot more details. What are the typical services/workloads running on k8s? What's the end user application?
I taught myself k8s in the first month of my first job, and it felt like having super powers. The core concepts are very beautiful, like processes on Linux or JSON APIs over HTTP. And its not too hard to build a CustomResourceDefinition or dive into the various high performance disk and network IO components if you need to.
I do hate Helm to some degree but there are alternatives like Kustomize, Jsonnet/Tanka, or Cue https://github.com/cue-labs/cue-by-example/tree/main/003_kub.... You can even manage k8s resources via Terraform or Pulumi
It was fascinating digging into this to find their dataset weights defined in a declarative YAML file [2]. 70% is from FineWeb/Commoncrawl but filtered using a classifier trained on Llama-70b's rating from 0-5 of the educational content of the text [3]. This is something we know small models like Phi-3 have been doing for a while, but it's great to see a fully open reproduction of it that beats their benchmarks. Definitely supports the idea you can get even better reasoning at smaller model sizes by carefully filtering and curating your training data (and generating good synthetic data from/distilling bigger models).
You can see the 450k Llama educational value scores here: https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu-ll... It's interesting, I think the text with 3 scores is really good, but the 5 scores pick content that is not very reasoning or information-heavy but just mentions education or a worksheet. For SmolLM they just took the documents with scores >= 3 so it doesn't matter a ton.
2. https://github.com/huggingface/smollm/blob/9efce803bc7e37727... 3. https://huggingface.co/HuggingFaceFW/fineweb-edu-classifier
https://www.businesswire.com/news/home/20230522005289/en/Int...
This is a project by the Allen Institute for Artificial Intelligence. I have no affiliation.
The link above is to an active demo. I'm not sure how scalable the backend is so far, so it may crash if there is significant load.
1. Demand removal of analytics software 2. If no action, fork and re-publish.
Obviously folks who aren't technical/didn't see these threads wouldn't get the benefit of an update.
This is something where an explicitly pro-opensource and anti-tracking (or at least minimal tracking) policy by the browser extension stores would be valuable. The store itself could recommend the no-tracking community version instead. Of course this would have to happen on an individual basis and be carefully managed as so not to be abused.