albertan017

We find several work on refining Ghidra decompilation results with GPTs, that could be another interesting directions!

Yes, GPT4 is very impressive, as it's not directly trained on the decompilation. We're working on improving our model, please keep watching updates!

Thanks. We acknowledge that an LLM cannot completely replace human expertise in decompilation, much like GPT-4 has not achieved true human-like intelligence. However, the aim of our llm4decompile project is to do something like GPT-4, and offer assistance and enhance productivity in the decompilation process.

As for test suites, it's one of our project's main challenges—figuring out which functions satisfy the expectations of reverse engineers, how to autonomously produce high-coverage test suites, and how to objectively qualify decompilation outcomes without relying solely on human judgment. Looking forward to your advices!

Thanks! The concern is how to uniformly uplift binary code from various architectures and configurations to the same IR like RzIL? Is there a method to automate the disassembly process reliably across these different systems?

Ideally, with a substantial dataset of obfuscated JavaScript and corresponding raw code, a language model could potentially make good predictions. The first key difficulty, however, is collecting a large-scale dataset and setting up a system for automatic compilation and segment out the binary-source pairs.

Thanks! We're working on Ghidra/IDA pro. The problem we face is the right kind of data to test with and how to evaluate it. It's like there's no "standard" benchmark/metrics that everyone uses for decompilation.

Thanks! But people want an all-in-one solution for decompilation. Given the vast array of architectures and compilation settings, and the fact that these information are usually not predetermined, finding a way to effectively navigate this complexity is quite difficult.

Thanks! The model is trained only for O0-3, not support for obfuscation. There's still a long way for llm to go.

We're interested in the toolchain, could you share the link or reference to it? GPT4 does an amazing work, we're also very surprised that it can work.

Thanks! We're working on Ghidra/IDA pro. The problem we face is the right kind of data to test with and how to evaluate it. It's like there's no "standard" benchmark/metrics that everyone uses for decompilation.

Yes, it's not easy to train a 33B model. An interesting point is, naive fine-tuning, which means if one followed the standard way to fine-tune the model. Training a larger model is tricky, not only the data amount matters, everything like data cleaning, learning rate, and decays will affect the final performance.

Thanks! Our initial experiments indicate that for simple cases, such as short snippets (tens of lines) of code without external dependencies, the LLM can decompile very well. However, for more complicated examples, it tends to offer speculative solutions, and the utility of these results is challenging to assess. The determination of whether the decompiled output is correct or useful is subjective and lacks a universal standard. One approach we're considering is utilizing GPT-4 as a benchmark to evaluate other models' performance. We're open to further suggestions to refine our evaluation methods.

I found lots of decompilation work are conducted on C. It seems not much python projects are compiled into binaries.

Thanks! Training a language model from scratch is data-intensive; Llama2 was developed using 2 trillion tokens, while our dataset is around 4 billion.

The appropriate size of the model is not straightforward to determine. In our experiments, a 7 billion parameter model achieved 21% executability compared to just 10% for a 1 billion parameter model. However, their re-compilability rates are quite similar.

To run a 1 billion parameter model, a minimum of 2GB GPU memory is necessary, which is feasible on most GPUs. A 7 billion parameter model needs 14GB, suitable for GPUs like the 3090/4090 series. For running a 33 billion parameter model, an A100 GPU (80G) would be the single card option, although technically a MacBook could work, but you won't really want to use it.

Thanks! Processing raw binary data directly would be inefficient for the language model, as it's not designed to interpret strings of zeros and ones but for understanding higher-level instructions (like code and natural language).