Have you been tracking calorie intake, both before and after the diet change? That's the deciding factor for your weight. Metabolism, wheat/yeast sensitivity or even normal exercise only has a minor impact.
I would think dropping something as common as wheat would have significant impact on your diet, or?