Socialization is the main point of elementary school. Missing it is a gigantic drawback.
And anyway, unless you live in some sort of jerk-run school district, the public school system welcomes home school kids for band, athletics or any other after-school program.