Little endian does appear strange at first, but if you consider the motivation it makes a lot of sense.

Little endian's most valuable property is that an integer stored at an address has common layout no matter the width of the integer. If I store an i32 at 0x100, and then load an i16 at 0x100, that's the same as casting (with wrapping) an i32 to an i16 because the "ones digit" (more accurately the "ones byte") is stored at the same place for both integers.

Since bits aren't addressable, they don't really have an order in memory. The only way to access bits is by loading them into a register, and registers don't meaningfully have an endianness.