paradite

Opus 4 beats all other models in my personal eval set for coding and writing.

Sonnet 4 also beats most models.

A great day for progress.

https://x.com/paradite_/status/1925638145195876511

I don't think it's a black and white distinction between agentic and non-agentic tools. Not to mention tools are constantly evolving and changing.

For example, Cursor a year ago was not agentic at all. GitHub Copilot only recently added agentic features.

I also think the end game for an agentic tool would not an IDE, because IDE was designed for human workflows, not agents.

I wrote about this topic a while ago and made a classification that's probably a bit outdated, but still relevant: https://prompt.16x.engineer/blog/ai-coding-l1-l5

Missing OpenAI Codex cli

Also missing a class of non-IDE desktop apps like 16x Prompt and Repo Prompt.

There are actually a lot of tools that do this:

https://prompt.16x.engineer/cli-tools

I built a tool that helps with copy pasting code into chat UI called 16x Prompt.

https://prompt.16x.engineer/

It is used by quite a lot of people. So the problem is definitely there.

The app supports API integration as well. From usage stats, still there are more people using the copy pasting flow instead of API flow.

I suspect it is because people are already have a subscription so they can basically use that for free, versus via API where they have to pay.

With that said, Cursor's $20/month unlimited usage is really too good to miss. I will wait for it to end soon.

I'm getting so much sci-fi vibes from this post.

I've read so many sci-fi stories where big tech corporations have similar control over people as countries. Now we are actually heading there.

I'm both excited and a bit worried about the future.

It's kind of interesting if you view this as part of RLHF:

By processing the system prompt in the model and collecting model responses as well as user signals, Anthropic can then use the collected data to perform RLHF to actually "internalize" the system prompt (behaviour) within the model without the need of explicitly specifying it in the future.

Overtime as the model gets better at following its "internal system prompt" embedded in the weights/activation space, we can reduce the amount of explicit system prompts.

I wonder what if this is just a decoy to get the more sophisticated candidate in.

I can't belive Ollama haven't fix the context window limits yet.

I wrote a step-by-step guide on how to setup Ollama with larger context length a while ago: https://prompt.16x.engineer/guide/ollama

TLDR

  ollama run deepseek-r1:14b
  /set parameter num_ctx 8192
  /save deepseek-r1:14b-8k
  ollama serve

Thinking takes way too long for it to be useful in practice.

It takes 5 minutes to generate first non-thinking token in my testing for a slightly complex task via Parasail and Deepinfra on OpenRouter.

https://x.com/paradite_/status/1917067106564379070

Update:

Finally got it work after waiting for 10 minutes.

Published my eval result, surprisingly non-thinking version did slightly better on visualization task: https://x.com/paradite_/status/1917087894071873698

https://eval.16x.engineer/ - 16x Eval: A desktop GUI app to evaluate prompts and models

With 16x Eval, you can manage your prompts, contexts, and models in one place, locally on your machine, and test out different combinations and use cases with a few clicks.

Seems like the story of Stackoverflow.

If you want to evaluate your personal prompts against different models quickly on your local machine, check out the simple desktop app I built for this purpose: https://eval.16x.engineer/

I think you should try more tools and use cases.

Yes some of the current AI coding tools will fail at some use cases and tasks, but some others tools might give you good results. For example, Devin is pretty bad at some trivial frontend tasks in my testing, but cursor is way better.

I have good success with web development and JavaScript in particular, on several large codebases.

I also built my own AI coding and eval tools that suits my needs, though cursor has largely replaced the AI coding tool that I built.

Ah I see what you mean. I was trying to convey that this is a limitation, hence not a tick symbol.

But I guess it could be interpreted differently like you said.

Have you tried using a tool like 16x Prompt to send only relevant code to the model?

This helps the model to focus on a subset of codebase thst is relevant to the current task.

https://prompt.16x.engineer/

(I built it)

I forgot to mention that OpenAI also invented PPO, which is the default algorithm that everyone uses for RL since 2017: https://en.wikipedia.org/wiki/Proximal_policy_optimization

DeepSeek's GRPO is also just a minor variant of PPO.

You can bypass this problem by embedding relevant source code files directly in the prompt itself.

I built a desktop GUI tool called 16x Prompt that help you do it: https://prompt.16x.engineer/

The author mentioned AlphaGo and Alpha Zero without mentioning OpenAI gym and OpenAI Five.

Those products show OpenAI was innovating and leading in RL at that stage around 2017 to 2019.

https://github.com/openai/gym

https://en.wikipedia.org/wiki/OpenAI_Five

Here's a longer blog post I wrote on the same topic, with new updates daily:

https://prompt.16x.engineer/blog/quasar-alpha-openai-stealth...

My understanding is that they have a base model checkpoint for Behemoth from pre-training.

This base model is not instruction-tuned so you can't use it like a normal instruction-tuned model for chatbots.

However, the base model can be distilled, and then the distilled model is post-trained to be instruction tuned, which can be released as a model for chatbots.

I might be missing the point of the author, but what's the difference between Steam Deck and a normal Windows gaming laptop in terms of software freedom?

Wouldn't you have more software freedom on Windows? Because you can run both Windows and Linux software (via WSL2).

I use macOS, Windows and Linux daily. They are all pretty open to installing and running your own software. And all of them have some sort of security measures that prevent you from running arbitrary apps unless you close some scary warnings or bypass it with some flags.

This is not a good comparison for real world coding tasks.

Based on my own experience and anectodes, it's worse than Claude 3.5 and 3.7 Sonnet for actual coding tasks on existing projects. It is very difficult to control the model behavior.

I will probably make a blog post on real world usage.

Maybe it is. Maybe it's not.

I prefer to state facts and leave interpretations to others.

You can turn it off. This is just a poorly worded FAQ.

At the beginning, there is an FAQ that says you can't turn off ads in scenic mode.

  Q: Can I turn Scenic Mode ads off?
  A: No, not at this time... 

But in fact you can turn the Scenic mode off completely, but it buried later in the content of an FAQ about settings.

  Q: Can I change any settings for Scenic Mode?
  A: Yes. To use Scenic Mode with or without volume, navigate to the left side menu on the VIZIO Home screen, click Settings > Extras > Home Page Settings > Scenic Mode. Once you’re in the Scenic Mode section of Settings, scroll down and select either Volume On or Volume Off. You will see a checkmark appear, noting that you’ve correctly selected your setting. If you want to turn off Scenic Mode completely, follow the same instructions to get to the Scenic Mode section of Settings and select Disable.

For anyone not in LLM field, OpenAI API is the de facto standard for almost all AI companies and labs nowadays.

DeepSeek docs just point users to use OpenAI SDK: https://api-docs.deepseek.com/

Anthropic recently did some something similar: https://docs.anthropic.com/en/api/openai-sdk

Ok, so the certificate used to sign the package is generated by Apple, why can't I just use that to prove my identity for notarization?

Or maybe simpler, why can't Apple just do code sign and notarization with one single cli call, with one set of credentials?

Google Play does this under the hook, I don't even think about it. iOS is similar, Transponder app does everything in one go.

It's true. One example I can give is how Gmail used to automatically recognise flights and hotel bookings and add them to calendar.

It was suddenly completely broken and stopped working a few years ago. I tried every setting to try to get it working but couldn't.

I feel like a stone age caveman having to manually type everything into my Google calendar.

There are a lot of people raising the same issue in Google forums, but it's not fixed yet.

Ironically they are adding new Gemini AI features into Gmail, which can't do this as well.

As someone who actually signs, notorizes and distributes desktop apps for macOS, I can safely say their documentation is less than ideal.

Maybe because I'm using Electron framework which makes things more complicated, but I don't really understand why there's is a difference between different types of certificates (Developer ID, Apple distribution, macOS distribution) and I had to guess which one to use everytime I set it up.

Also why is notorization a completely different process from code signing, and requires completely different set of credentials from it. Seems odd to me.

I think this is an interesting pivot, but Cursor's project level rules and custom modes will probably quickly evolve to cover all the aspects listed on your hub. (Maybe it already does)

This also allows developers to switch between projects quickly while simultaneously also switching the setup for the AI agent.