And in your particular case, where we want to also tell it is big, we do: The big building red (Le grand bâtiment rouge)
But we can't do the following: The red building big (Le rouge bâtiment grand)
Different language, different grammar :) French is one of the most difficult language for these I think, a bunch of exceptions. It makes the language more "lovely" to hear but less "effective". Here we say English is made for business, it is one of those languages that just gets to the point fast and clear overall :)