int intermediate = byte_or_word << N; // N chosen to put the five bits in the highest positions in the int.

int value = intermediate >> bits_per_int - 5;

Instead of masking rely on arithmetic shift right to keep the sign bit.