Yes for each color. That means two pawns for each column can be in position to be captured en passant. And also two pawns can be in capturing position.
Edit: Column combined with whose turn it is will work, but not just column.
Part of a chess position is whose turn to move. That has to be encoded anyways.