The Minimum Every Software Developer Absolutely Must Know About Unicode (2003)
joelonsoftware.com
joelonsoftware.com
Unfortunately, Java and Windows both adopted 16-bit Unicode as their internal representation. So did the Python of the time. This led to a hokey scheme where two 16-bit Unicode characters are used to represent one longer character. Current language status is that Go and Rust are all 100% Unicode, Java still has the 16-bit hack, Python 3 is 100% Unicode but Python 2.7 has 16-bit or 32-bit Unicode builds, and C and C++ still require a lot of explicit work by the application programmer. PHP has some Unicode support, but doesn't really understand string encodings.
On the web, HTML is now almost entirely UTF-8. At least that's been straightened out. Mostly. There's still too much Latin-1 (ISO8859-1) around. Browsers still mostly support the old encodings, although Microsoft Edge apparently doesn't do so very well. Note that HTTP is not Unicode-aware; URLs, domain names, and HTTP data often have strings with other encodings.
The major databases understand Unicode, but it may be necessary to specify an encoding. MySQL went through a 24-bit Unicode period ("utf-8" is 24-bit) but now supports 32-bit Unicode. This requires specifying "utf8mb4" as the encoding. For new work, use that for everything. Postgres uses "utf-8" for 32-bit Unicode. MongoDB has no idea what character set it's using; it only knows raw bytes, and the application has to handle encoding and decoding.
The current big question is whether programs should use UTF-8 encoding internally, and whether the encoding should be visible to the programmer. Rust uses UTF-8 internally, it's totally visible to the programmer, and you have to use special libraries to iterate over a string properly. Go tries to hide it a little; if you iterate over a Go string using "range", you get 32-bit "runes". Python hides the internal representation completely; if you iterate over a Python string, you get strings which contain one code point, and strings are subscriptable by code point.
[1] http://unicode.org/emoji/charts-beta/full-emoji-list.html
What's the reason for the quotes around utf-8? To my knowledge it's bog standard utf-8, with the exception that NULL is disallowed in many types.
It's often easier to use that for-loop with a break to look at the first character of such a string, as in...
package main
import "fmt"
func main(){
str:= "我是一个人"
for _, r:= range str {
fmt.Printf("%c\n", r)
break
}
}
import "unicode/utf8"
...
r, _:= utf8.DecodeRuneInString(str)[1] http://www.lsi.upc.edu/~robert/teaching/master/material/p5-g...