Cot Decoding – Eliciting Reasoning from LLMs

A recent talk (https://dennyzhou.github.io/LLM-Reasoning-Berkeley.pdf) from Denny Zhou covered a number of techniques that improve LLM reasoning. In the talk he a recent paper from Google Deepmind on "Chain-of-Thought Reasoning without Prompting" (https://arxiv.org/abs/2402.10200).

The key idea in the paper is that existing models are capable of doing CoT style step-by-step reasoning via a new decoding strategy. I implemented their approach in optillm - https://github.com/codelion/optillm/blob/main/optillm/cot_decoding.py as I couldn't find any decent open-source implementation.

I have also replicated their core idea with the recent open source Qwen 2.5 (0.5B) model. I ran the GSM8K benchmark with cot decoding and found over +9.55 points improvement (from 22.82 to 32.37). Thus, cot decoding is an interesting approach that can elicit reasoning from existing LLMs without explicit prompting.

Remember in optillm you cannot use cot decoding with the proxy as the technique cannot work with just the LLM API, you need to have access to the model. You can test it with any model from HF with this Google colab notebook - https://colab.research.google.com/drive/1SpuUb8d9xAoTh32M-9wJsB50AOH54EaH?usp=sharing