The server is a proxy between a client library and a browser (or a browser driver like ChromeDriver). Like all good proxies, it can add some cool features to your workflow, but is not required for basic usage.

For example, you can download the Selenium client library for Python and drive Chome (via the ChromeDriver executable provided by Google). No Java required.

Removing the Java requirement for basic point-to-point (e.g. Python to Firefox, Ruby to Chrome, etc.) automation was a key feature of the Selenium 2 (aka WebDriver) architecture when Simon Stewart first released it back in 2011.

Selenium Server is only need if you want to run a remote Selenium WebDriver. If you want to drive a local browser you can just use the language bindings you wish.