Hacker News Reader

Comment by Spooky23 | Hacker News Reader

What is it about elastisearch that dopey people stuff them with information and leave them on the open internet, all of the time?

View on HN

Elasticsearch started life as a free product and security was a paid addon to that product via the X-Pack, now Elastic Co has made the security stuff free but people still don't implement it. Elasticsearch is insecure out of the box and it takes extra steps to get it secured, and most people don't do those steps even though its pretty well documented right here:

https://www.elastic.co/guide/en/elasticsearch/reference/curr...

Security features in Elastic still require paid subscription. The link you pasted even says that. You can use the xpack features for free on a trial basis but for production use you're required to buy a license.

Thank you for the correction. I thought I heard rumblings about their X Pack being free, but I had probably just heard about the trial.

You can use the Amazon OpenDistro provided plugins on top of oss ES to enable security features.

This is not true, you can use some xpack security features such as basic auth, client TLS and node-to-node TLS for free. We use basic auth (with Vault integration) at my company using just the basic license. https://www.elastic.co/subscriptions has details on the subscription levels.

Thanks, I did not know about basic license. Although if you wanted to use OSS license my point still stands. (Not sure what restrictions the basic license brings so for some the only option might be the OSS license.)

You're right, and it is only recently that this stuff became free anyways, sometime last year. I think Amazon's OpenDistro stuff put some pressure on them. Even TLS was gated, which was the biggest offense in my opinion!

In a word: devops.

Developers are not operators and operators are not developers. The whole idea that we can do away with this specialization and and relegate operations to the people that create software because it is now possible to script infrastructure and to install complex packages with a few mouseclicks does not make it true. Operations and the complexity that goes with it is a job in its own right, no competent operator would have left this situation as it came out of the box.

A combination of businesses' desire to spend less on labor and your average developer's inherent sense of superiority mean this trend is unlikely to go away any time soon though.

I believe you can be both competent operator and a reasonable developer at the same time. The skills complement each other nicely. It is a lot of work to be these things though.

It's the default configuration. Unless you go out of your way to ensure security, you'll get this result.

I believe Elasticsearch does not have authentication built into the open source version.

This is no longer true, but ONLY as of May 2019 in response to pressure from the containerization world.

https://www.elastic.co/blog/security-for-elasticsearch-is-no...

It’s not default insecure like Mongo was - this was far far worse. You couldn’t even prototype in a secure way even if you wanted to, without a massive contract. One of the most frustrating things in software - IMO they deserved to have AWS commoditize their stack.

Perhaps it's a controversial opinion, but I feel like it's just flat out unethical to relegate basic security to the paid/enterprise version of your product.

Of course it's unethical to use said product to store real user data too, but the road goes both ways.

Just deploy it on your local network. No need to expose it to the internet. Sure, authentication is a nice bonus, but a simple firewall goes a long way.

I don't get why it ever needs to be on the internet even when it does have authentication. Surely the public/private subnet split is a common practice.

The way I see it, password auth for db servers is the last-resort protection mechanism.

If you get to the point where it’s what’s protecting your data, you’re already fundamentally screwed.

You should definitely have it too, though.

Unless the private subnet is airgapped from the Internet, it's not a good enough separation.

Hell, even if it is airgapped, it can still be compromised by viruses on USB sticks and such.

You should never be leaving sensitive systems wide open, period, regardless of how secure you might think that network is. Thousands of data breaches have been caused because networks didn't end up being as secure or as separated as hoped for.

Defense in depth is important. Lots of data breaches have been caused because things that should have just been viewable from a local network, weren't, or the network was compromised. Unless you think every single employee is invulnerable to spear-phishing (which is impossible), you should never be leaving anything sensitive wide open on your local network.

The module is now free, but it's not open source, it is licensed under the proprietary Elastic license. The source is available but it is not licensed to be used with anything except the Elastic licensed version of Elasticsearch (not even the Apache licensed version of Elasticsearch)

However, Amazon has thankfully released a free and open source security module for Elasticsearch as part of their Open Distro project. It is based on another project called Search Guard. See: https://opendistro.github.io/

ES is very poorly engineered but unexplainably popular. The interesting question is why it's popular.

Fast free full-text search over arbitrary documents. It solves a problem lots of people need.

Iirc it doesn't have very good support for document formats (eg pdf, office). It's also a huge resource hog so not particularly fast in the bang for the buck sense.

Honestly, some of the most preventable and dumbest outages and failures in my career have involved ElasticSearch. Most of the time it's deployed and managed by a dev team with no operational oversight, and therefore nobody to think about or catch these types of issues. It is compounded by the fact that all the security features in ES were paywalled for a very long time and most technologists don't understand basic networking anymore.

As many other answers to your query have stated, this is caused by a broken understanding of the devops methodology among organizational management forcing developers who are not competent in systems administration to be responsible for these systems.

We don't have sysadmins anymore "because of the cloud"

"Sysadmins? No, we just have devops now!"

Many managers use "devops" as an excuse to put a lot of burden on a small team, then this team is doing their best to automate managing a large number of machines but it's physically impossible to delve deeper into details and polish things, hence mishaps are bound to happen. And don't get me started on on what is happening inside containers.

Sorry but I'd like to get you started on what is happening inside containers ;P

Specifically can you go into more details about what worries you with containers. Is it insecure images with out of date software, or risky applications inside the containers? Something else?

Let's imagine your JIRA is insecure, someone owns it and obtains RCE, then does a privilege escalation on the host, whoops suddenly all services are accessible whereas that would have required more steps and owning in the old one-vm/bare metal-server-per-service model.

Escaping properly namespaced/pivot_root'ed environment and owning a host is non-trivial too.

Equally important, we don't have network admins. It would be physically impossible to expose our search database like this to the open internet. Extra layers of protection are great.

It reminds me of companies I've worked with before that accidentally had a production site pointed to a dev database. Why the hell is that even physically possible with your network setup?

Welcome to the wonderful world of Kubernetes (or, for that matter, any Docker orchestration solution, such as DC/OS).

Anything can reach anything, provided you know the naming schema... and there's no easy way to fix it on anything that is not AWS/Azure/GCP, not without losing all the benefits of a self hosted k8s cluster in the first place.

Openstack at least provides ways to isolate machines, but that's VM-level only and truly an ultimate PITA to set up.

Not entirely true, there are network security policies (on select few CNI providers) and other means of segregation using good old iptables (although probably need to update alternatives for iptables to point to iptables-legacy for them to work).

Ew. iptables (or any other way of messing around with the black magic that Docker and the orchestrators do to provide intra-container networking) is one thing only and that is a nice way to shoot yourself in the head while aiming at your legs.

To be entirely honest, if you know what you're doing and how lets say kube-proxy works in essence, things get pretty easy and simple. If you start every configuration of firewall with iptables -F, you're gonna have an interesting time. However, if you spend some time around these beasts, they are pretty well and logically built and it is trivial to coexist and modify your chains without touching those managed by docker/k8s. There is no black magic/and or wrong with the way they manage the rules. I'm much more angry at proper iptables being moved to iptables-legacy and systemd messing around with my resolv.conf :)

Amazon does provide a mechanism to define private networks disconnected from the internet.

https://aws.amazon.com/vpc/

Reply on news.ycombinator.com