to see how much the runtime contributes to the size of the binary use an empty main. for hello world you bring in the fmt package which is a relative heavyweight:
$ cat > t.go
package main
func main() {}
$ go build t.go
$ wc -c t
623280 t
$ nm t | wc -l
1029
$ cat > t.go
package main
import "fmt"
func main() { fmt.Println("hello world") }
$ go build t.go
$ nm t | wc -l
2541