Unicode is not an encoding, despite MS Notepad calling some encoding "Unicode".
However you'd probably like to store something more compact than, say, JSON arrays of integers. So there are also a bunch of encodings which turn the integers into bytes. These encodings would work for any integers, but they make most sense to encode Unicode's code points. UTF-8 turns each code point into 1-4 bytes, a pair of UTF-16 encodings turns them into one or two "code units" each of 2 bytes either little or big endian. And UTF-32 just encodes them as native 32-bit integers but again either little or big endian.
> A: Unicode is the universal character encoding, maintained by the Unicode Consortium. This encoding standard provides the basis for processing, storage and interchange of text data in any language in all modern software and information technology protocols.
1. Map a character to a unique number in a character set (in Unicode: called codepoint)
2. Map a number that represents a character in a character set to a bit pattern for storage (transiently or persistently, internally or externally). Unicode code points can be bit-encoded in various ways: UTF8, UCS2 and UCS4/UTF32.
The original code points permit the same character to be represented in various ways, which makes equality checks non-trivial: for instance a character like "ä" can be represented as a single character or alternatively as a composition of "a" + umlaut accent (2 characters).
So far, this is all about plain text, so we are not talking about font families or character properties (bold, italics, underlined) or orientation (super-script, sup-script).
Ken Lunde's opus magnum is the standard book on representing text in various languages other than English, with a focus on Asian languages: https://www.oreilly.com/library/view/cjkv-information-proces...
It’s like a catalogue of scripts. We have to extend it when we encounter new scripts that are not catalogued yet (or when we create new emojis)
Converting a byte sequence to a Unicode code point sequence and vice-versa is called transformation format (or more generally an encoding form, but then might not be deterministic) by Unicode (see <https://www.unicode.org/faq/utf_bom.html#gen2>). Unicode specifies UTF-8, -16 and -32. We do not have to change these formats unless the catalogue hit the limits of 32 bits (not a big problem for UTF-8 but for the other two formats). These formats are already able to encode code points that are not assigned yet.
And the confusion now is that a lot of people call what Unicode calls transformation format (i.e. the byte to code point mapping) encoding as well. The term charset is also used sometimes.
PS: Note that a goal of Unicode is to be able to accommodate legacy encoding/charsets by having a broad enough catalogue. This is so that these legacy encoding which may come with their own catalogue can be mapped to the Unicode catalogue. So we have control codes (even though not part of any “proper” human script), precomposed letters (there is a code point for à although it could be represented by a + combining `), things like the Greek terminal form of sigma separately encoded, although that could be done in font-rendering (like generally done for Arabic), and a lot more to aid with mapping and roundtrips.
Edit: by which I mean that I’ve only seen the non terminal form so far in maths but it’s hard to write an algorithm that distinguishes between a word ending in sigma and some juxtaposition of Greek variables.