How are you optimizing over textures? Do you have a differentiable renderer? Or are you not calculating those gradients via backpropagation?
It's a reasonable amount of effort to disentangle it from the rest of our code and build a nice API, so it will be several weeks till we get around to this.