Northern Europe usually refers to Norway, Sweden, Finland, and sometimes Denmark. But they are further north than most of the other countries people think of as "eastern", that is true.
CIA indeed defines Northern Europe as you describe, including Denmark, plus Iceland. [1] However the United Nations has a different definition, which also includes the Baltics, UK, and Ireland. [2]
[1] https://en.wikipedia.org/wiki/Northern_Europe#CIA_World_Fact...
[2] https://en.wikipedia.org/wiki/Northern_Europe#UN_geoscheme_c...
In the UN geoscheme, Baltic countries are part of Northern Europe. According to EuroVoc, Baltic countries are part of Northern Europe.