Back in Maxwells time the idea of a machine that could make "intelligent" decisions was not very developed and neither was the miniaturisation of technology that we know today. So to not let his thought experiment get sidetracked by questions of "how would you actually build such a machine" he just called it a demon and could then focus on the thermodynamic properties of the system.

These days we would probably call it a quantum device or something.

> 2. A neutral supernatural spirit.

> An hypothetical entity with special abilities postulated for the sake of a thought experiment in philosophy or physics.

https://en.wiktionary.org/wiki/demon#Noun