MongoDB security notice | Hacker News Reader

MongoDB security notice

mongodb.com

269 pointsciudilo169 comments

We are completely locked out of our Atlas account and the support portal right now. We Okta-auth with Mongo and all attempts to auth right now are failing with "The request contained invalid data." displayed on their login screen.

Of course, the support portal requires you to auth to use it...to get help with auth failing.

Anyone else seeing issues getting in to their dashboard?

Edit: Auth started working for us and dashboard access became available for us around 5:15 pm ET.

upstream request timeout when trying to sign in

On our side, Okta is saying the auth is good.

I'm trying my personal account as well and it's telling me MFA isn't set up (it is) and it's making me go through the MFA setup flow again. All attempts to setup another 2FA code in 1Password or to get even an SMS code sent to my phone are failing.

Edit: Personal account with a TOTP 2FA is working again now as well.

This is feeling worse than they're letting on to.

You really should not be using SMS for 2FA.

You really aren't following along closely enough: all other options were failing for me.

But you have setup SMS 2FA enabled, which is convenient this time but a big security hole. You should consider disabling it once the situation comes back to normal.

> But you have setup SMS 2FA enabled

No. I did not. Nor do I now.

I had a TOTP setup in 1Password and Mongo was telling me MFA _wasn't_ set up and sending me through the MFA setup flow again.

All options, SMS included, were failing in that MFA setup flow they pushed me in to.

They're back now and my existing TOTP token is generating one time use passwords that work now.

I bet that's because different parts of their stack disagreed. Obviously a two factor setup should not be acceptable when one is already in place-- if the frontend thought it wasn't but the backend/auth services thought it was, it could explain that.

For my own knowledge, if the options were between using SMS for 2FA or not having 2FA at all then what is better? I've heard mixed things about this.

SMS 2FA is better than no MFA at all, despite the very valid concerns about SMS. It at least protects against credential stuffing and similar automated attacks.

I guess I've always cynically assumed that companies want my phone number to make the data they gather more valuable by making it easier to link with a unique index like a phone number.

Two things can be true at the same time.

Well a simswap attack requires the account password, since otherwise you would not be able to receive an SMS message for the two factor part.

But without two factor, only your account credentials are needed.

So yeah, it's definitely better than nothing, you are effectively forcing your opponent to social engineer your carrier, and doing that generally requires knowing the full number and usually at least your name, if not more identifying information that's harder to get, like social security number or equivalent.

Sure, TOTP or other two factor mechanisms are better because they require access to one of your authenticated devices (assuming the TOTP isn't done by a secure enclave), but SMS two factor is definitely better than disabling two factor.

Same here with Google SSO

We regained dashboard access around 5:15 pm ET.

Seeing that reminded me of this recent onoz, which one should fold into their threat model: https://trufflesecurity.com/blog/google-oauth-is-broken-sort... <https://news.ycombinator.com/item?id=38670644>

MongoDB employee posting:

The login issues are unrelated to the security incident. We notified all of our customers and users concurrently resulting in a spike in login attempts. Please try again in a few minutes if you are still having trouble logging in.

Please continue to monitor our alerts page: https://www.mongodb.com/alerts

I mean that totally sounds related (hah!) although I guess we all know what they mean

That’s a funny point, I guess I never really though of whether “related” was more correlation or causation.

Nice and to the point, makes it clear that this is early, explains the current scope, tells us to expect a follow up as the information makes its way to them.

I like this tbh and I hope people won't punish them for not including more info when this is clearly in the early days of investigation.

It was only DETECTED on the 13th, and they suspect had been going on 'for some time'. And basically not sure if user data was touched but they suspect or haven't provided it yet buly saying'NOT'.

I want answers.

Yes, usually breaches take time to detect, and usually the attackers are around for a while first.

I'm sure they want answers too, but they're working on it, and this is what they have right now.

How does it feel to want? They are doing their due diligence currently.

Your options are: (A) Vendor waits until all the facts are in place and the investigation is finished or (B) Vendor tells customers as early as practical so they can take their own mitigation steps.

You do not have the option of (C) Vendor should tell me about a breach they don't yet know about.

Give them some time.

Agreed.

For all the shit MongoDB gets, this is something that people should take a step back and recognize as very high in integrity, transparency, and trust.

Other businesses should follow their lead here.

I'm more inclined to do business with MongoDB because they've demonstrated these principles first-hand.

I don't use Atlas but I do use self hosted MongoDB, and have been pretty happy with that product. I have the impression that a lot of the dirt slung at Mongo was about unreliability and data loss of the core product early on, which (knock on wood) hasn't been a problem for me on the small to medium scale use cases I've deployed it on. Seems reliability has taken a lot of positive strides over the years.

> […] regularly rotate their MongoDB Atlas passwords

Is there some context I’m missing, or is this a modern security team recommending password rotation?

Regularly rotating secrets for applications is good. Forcing users to regularly rotate their passwords is not so good.

Correct! NIST recommends against forcing password expiry unless the password is known to be compromised.

https://pages.nist.gov/800-63-FAQ/#q-b05

Thankfully all of my users are extreme statistical aberrations who do not re-use the same memorized password (or a variation on it) for more than one thing, ever, at all OR they diligently watch every single possible place they have ever re-used any of their memorized passwords, with the globally mandated and complied with reporting, so they can know if a password they once re-used at grandmas-cookies.blog.example.com has been compromised.

The fact that all websites, servers, systems (etc.) check to see if passwords are known to be compromised (since NIST says verifiers will do that) makes things a lot easier, too.

If we are impacted, how would you go about monitoring your systems for odd behaviors? Looking at the logs just doesn't seem adequate.

This highlights risks of extreme consolidation - even if Atlas customers were not affected it is natural for them to be concerned after announcement overwhelming web site or support channels.

More independent MongoDB DBaaS providers is what would offer true redundancy in this case, though it is highly restricted due to SSPL license change.

Hopefully FerretDB will be successful building feasible alternative

“Extreme consolidation” - wait till us-East-1 goes down

Happened in 2012 after a big thunderstorm and took most of the internet down

Yep. Though to be fair AWS provides options for multi region availability.

What did not happen (yet) is complete AWS meltdown

Amazon sells it cheap compared to other regions-- it's economically incentivized for us-east-1 to take out half of the Internet.

Friends don’t let friends run on us-east-1. Consolidate on us-west-1 or us-west-2 instead.

us-west-1 wouldn’t be my first choice either for <other_reasons>

us-west-1 seems small. They routinely run out of the instance size we request and the spot market seems.. unruly

Virginia is for lovers. Ohio is for uptime.

Just got email alert

Received this security notice today:

Hi Redacted,

MongoDB is investigating a security incident involving unauthorized access to certain MongoDB corporate systems. This includes exposure of customer account metadata and contact information. At this time, we are NOT aware of any exposure to the data that customers store in MongoDB Atlas.

We detected suspicious activity on Wednesday (Dec. 13th, 2023) evening US Eastern Standard Time and immediately activated our incident response process. We are still conducting an active investigation and believe that this unauthorized access has been going on for some period of time before discovery. We have also started notifying relevant authorities.

What should you do next? Since we are aware that some customer account metadata and contact information was accessed, please be vigilant for social engineering and phishing attacks. If not already implemented, we encourage all customers to activate phishing-resistant multi-factor authentication (MFA) and regularly rotate passwords. MongoDB will continue to update mongodb.com/alerts with additional information as we continue to investigate the matter.

Sincerely, Lena Smart MongoDB CISO

Yeah I received the same email. Luckily I don’t actually use mongodb atlas

“Your data is safe, because we’ve never written it to disk.”

/dev/null is web scale

https://youtube.com/watch?v=b2F-DItXtZs

Luckily already using https://devnull-as-a-service.com/

Does /dev/null support sharding?

Mongo has made huge improvements tbf, but this is funny

Asked 11 years ago and still going strong... "To what extent are 'lost data' criticisms still valid of MongoDB?" - https://stackoverflow.com/questions/10560834/to-what-extent-...

HN users would rather meme than read.

Isn’t mongodb data losses commonly referred related to their use of fsync()? From what I vaguely remember they call fsync() every 100ms or so and just assume everything went fine, resulting in potential data loss.

No. The default write concern as of version 5 is "majority" which "Requests acknowledgment that write operations have been durably committed to the calculated majority of the data-bearing voting members" (as long as writeConcernMajorityJournalDefault is true which is the default).

So fsync is called on every write.

It all depends on your writeConcern setting. This option is super critical to understand when deploying an MongoDb. Older versions had defaults that were more optimized for availability than consistency.

Even so, not flushing each write is not as bad as it sounds, if you have a 3-node replicaset and your w-parameter is set to majority (default), it means at least 2 servers have the write in memory. It would take both of them crashing at the same time for the unflushed write to be lost.

The idea is that MongoDB allows you to choose which corner of the CAP triangle you want, if you chose AP, that’s your decision. The defaults can of course be argued and I believe it’s been gradually moving over to more and more C for each version. Nowadays the journal does get flushed as next comment described.

Yep, Jepsen has not only confirmed but later reconfirmed that MongoDB has for many years been on par with other NoSQL databases when it comes to transaction guarantees and data consistency.

Er...no....

"Jepsen Disputes MongoDB’s Data Consistency Claims" - https://www.infoq.com/news/2020/05/Jepsen-MongoDB-4-2-6/

Also, "2020-05-26: MongoDB identified a bug in the transaction retry mechanism [2] which they believe was responsible for the anomalies observed in this report; a patch is scheduled for 4.2.8." [1]

[1] https://jepsen.io/analyses/mongodb-4.2.6

[2] https://jira.mongodb.org/browse/SERVER-48307

Those same issues exist in most NoSQL databases. If you're going to avoid MongoDB for data safety reasons, you should just go straight to SQL.

That is 3 years old.

I never used/tried MongoDB, what are the reasons people choose MongoDB over other DBs?

It's pretty easy to start with. MQL is also pretty easy to understand + MongoDB kinda makes it fun.

Note: I work at MongoDB

What's the update internally here? How long this been going on for? Any juice?

And Atlas has a free tier! It's been powering my site (https://bongo.to) for years for a total price of $0. Incredible!

Easy, flexible scheme nosql, plenty of baked in features. Has it's place, and many times when it would not be a good choice too.

It was an early player when everyone thought NoSQL document databases solved every problem.

They did solve many problems, and then they caused many more problems...

At first at least, haven't checked in on that in awhile

And now everyone knows RDBMS solves every problem. Especially when it's SQLite.

It's a really great alternative to firebase for mobile apps. Works pretty nicely with Realm so you get offline first db with powerful syncing. All the benefits of realm on the edge device with the power of the mongo platform. I dismissed mongo atlas for years because "mongo", until I finally gave it a chance. Overall been pretty pleased.

> It's a really great alternative to firebase for mobile apps.

Isn't firebase built on top of mongoDb?

I use it on-prem (well, on a VPS). It stores JSON documents, and it's easy to work with. If your data looks like a tree, it works pretty well, also for large documents. If you depend on relations between documents, you're better off with an SQL database, but note that for many cases --I'd say practically all mundane cases-- there's really no need for relations the SQL way. MongoDB also does relations, but a bit more convoluted.

Postgres does all that plus 100x.

But the horizontal scaling is easier with Mongo than Citus.

But you don't need to horizontally scale if you use a beefy Postgres instance. Horizontal scaling is a problem created by using an inefficient DB in the first place.

I have DB clusters with masters with a terrabyte of ram. Going beyond that is a PITA with warming up cache, backups, and so on. There is a reason there's a huge industry for shared databases.

Also, query for query, Mongo isn't going to be that much slower than PG, and faster for some usage patterns...

*sharded

I've also tried SQLite, but they didn't do large documents that well. And the project was started before I was aware Postgresql had json support.

But what's the advantage of Postgresql in this case?

Mongo is the main nosql choice. Mongo is great if you think a flexible schema is good. Mongo is not great if you think a flexible schema is bad. That sums it up

This is too reductive, you can essentially have flexible schemas with most modern relational databases and without the downsides of document-based DBs.

In 99% of the cases, even if you need a flexible schema, PostgreSQL will remain the best choice.

Unless it's not.

it probably is though

Enterprise support, lots of documentation, does what it says it does.

- Highly-flexible. Because you're not developing against a schema, you can, for example, retool a feature and its data quickly without having to stress about migrations. A big advantage for a startup looking to move fast.

- Queries look more like application code so you're not wasting mental cycles and time trying to translate an idea into a SQL query. From experience, this leads to less-fragile queries.

- Little to no concern over injection attacks (you have to go out of your way to create potential for them).

- Easier to write non-trivial queries than with SQL (IMO).

- Type-casting data can be done in code as opposed to with SQL you have to use inline, platform-specific functions like field_name::timestamp.

- A single source of truth for how to query and develop with it (with SQL, you're almost always developing against a flavor of it).

- Scales reasonably well (and easily) for a majority of use cases.

- No room for dogmatic fervor/confusion around a specific variety of MongoDB as there's only one variety.

> Because you're not developing against a schema, you can, for example, retool a feature and its data quickly without having to stress about migrations. A big advantage for a startup looking to move fast.

Why wouldn’t you need to worry about migrations without developing against a schema? You’ll need to worry more about migrations because your data will be more messy.

It could be rephrased as "worry about them later" :P gotta get those returns on the VC money and pawn those issues onto the next team in 7yrs :)

Not necessarily. If you're utterly careless, sure. A MongoDB migration is far less of a headache than a SQL one (you're just writing code to map/replace values). With SQL, you have to be frustratingly surgical about everything which can slow momentum to a crawl (read: punishment for mistakes in the migration is far worse than with MongoDB in my experience).

I completely agree! Kind of preaching to the choir :)

> Why wouldn’t you need to worry about migrations without developing against a schema? You’ll need to worry more about migrations because your data will be more messy.

If you're careless with your data, yes. "With great power comes great responsibility."

Being careful with my data is exactly why I use a strongly typed RDBMS in the first place.

So basically nothing Django and a postures db can't do.

We use MongoDB in conjunction with RealmDB to build an offline first mobile app. For this specifically it works very well. You basically define which parts of your document collection should be synced to which device, for example based on a query that contains the user id. MongoDB takes care of syncing the right data to the right device when internet connection is available. On the device, the data is stored locally in RealmDB, which represents the other side of the sync.

This is not easy to do with PostgreSQL, which we use for all other scenarios requiring a DB.

Why are people still choosing Mongo over Postgres these days? If there's something I'm missing, I'm genuinely curious as I'm not against json data and frequency use jsonb tables in Postgres.

People use MongoDB because it’s easy to get started. It does “db stuff” and “authentication”. I’ve given up trying to fight the trend. I just recognize immediately when it is used early on that the devs are still operating with training wheels on.

Sometimes this is the case but not always... It's nice to just work with objects in some languages... for some projects. That's engineering - picking trade offs :)

Except you can have objects in Postgres via jsonb; there's no trade off and you're both future and vendor proofed.

Yeah, there is. PG has its own warts that Mongo does not have. Also the JSONB query language sucks. See appending an array in PG [0] vs Mongo [1]. Also PG does not provide a way to atomically append to the array, you have to lock the row via SELECT ... FOR UPDATE. Mongo's $push, $set, etc, are atomic.

[0] https://pastebin.com/v2MiV8PE

[1] https://pastebin.com/RmLuGzAY

Okay, so use a server side language to do it.

that only solves half the problems I mentioned... :)

Are they doing well? Seems like the hype has kind of died down.

They're apparently still growing quite rapidly, though the company is not yet profitable.

Recent 7.0.0 version has dropped old and introduced quite broken new query planner. Caused a lot of our queries to miss. We’ve had the displeasure to work with the support on multiple related issues.

Maybe they're moving into the Apple category of "don't use .0 releases"

precisely, don't use .0 releases and don't rush to update immediately. Let others do that work.

Yeah, as a company they pretty much dominate the NoSQL space. 1B+/year in revenue and that market is still growing at like 30ish% YoY or so.

The hype died but the tool is almost mature now and quite useful, plenty of paying enterprise customers who fly low but have fat wallets. I’d give it a few more years to be a primary data store, but it’ll do if you want a horizontally scalable, capable and performant document database to give some breathing room to the primary SQL cluster.

For now, there are way too many critical issues found late in the release cycle.

Irrelevant but curious if MongoDB is still being picked up for Greenfield projects given it's licensing.

What’s wrong with licensing?

https://www.mongodb.com/licensing/server-side-public-license...

I am not sure really.

"It should be noted that the new license maintains all of the same freedoms the community has always had with MongoDB under AGPL - they are free to use, review, modify, and redistribute the source code. The only changes are additional terms that make explicit the conditions for offering a publicly available MongoDB as a service.

Obviously, this new license helps our business, but it is also important for the MongoDB community. MongoDB has invested over $300M in R&D over the past decade to offer an open database for everyone, and with this change, MongoDB will continue to be able to aggressively invest in R&D to drive further innovation and value for the community."

https://thenewstack.io/the-case-against-the-server-side-publ...

Encryption at rest is not supported in the community/free version of MongoDB.

We built an email service (IMAP support added a month ago) and wrote a WebSocket to SQLite layer to solve our encryption at rest needs for storage.

See our deep dive at https://forwardemail.net/blog/docs/best-quantum-safe-encrypt... for insight.

Really? How many open source databases do you offer? Some may say it’s not right for randos to complain when you give something away and they complain that it’s missing basics. I just happy someone else wrote most of what I need and I can extend it if needed.

I wonder, why would you want DB-managed encryption instead of just putting its storage directory in a LUKS-encrypted volume?

Note Percona Server for MongoDB is drop-in replacement for MongoDB and supports Data at Rest Encryption, on SSPL version

https://docs.percona.com/percona-server-for-mongodb/5.0/data...

MongoDB’s SSPL is neither an open source license[1] nor, most likely, a free software one[2]. Its definition of offering the licensed software as a service is so broad most Linux distributions[3–6] flat out refuse to ship MongoDB (not even in a nonfree repository or the equivalent) so as to (among other things) avoid placing the operators of their package mirrors in legal jeopardy.

[1] https://blog.opensource.org/the-sspl-is-not-an-open-source-l...

[2] https://opensource.stackexchange.com/q/13888

[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=915537

[4] https://fedoraproject.org/wiki/Changes/MongoDB_Removal

[5] https://bugzilla.opensuse.org/show_bug.cgi?id=1122267

[6] https://lists.archlinux.org/archives/list/arch-dev-public@li...

Their license "is to require that enhancements to MongoDB be released to the community."

I think it only hurts people who want to freeride the project and extend it for selfish personal gains. That's OK by me.

That's reply to Amazon abuse of MongoDB (DocumentDB)

It is the opposite. Amazon would have released MongoDB as a service, same as they do for PostgreSQL or MySQL. As MongoDB changed license they implemented DocumentDB instead.

Note AWS significantly contributes to PostgreSQL and MySQL communities (though you could always want even more) but of course does not to MongoDB. While this is fine for MongoDB Inc I think it is not great for MongoDB community at large

Not really. It really designed to prevent anyone else having MongoDB DBaaS without having license from MongoDB, which it does rather successfully.

Anyone can provide Postgres, MySQL or other open source databases as a service.

For this reason, there are many providers to choose from, and there is a healthy amount of innovation and competition in the space. Prices are set by market and demand, as it should be.

And then there is MongoDB where only a handful of providers could negotiate a license, and the price is set by MongoDB Inc.

In my opinion this is by no means "fine" from a user perspective as we are talking about database software.

If anyone did freeriding, it is MongoDB Inc. who chose to freeride on the open source community for marketing purposes, before switching to SSPL.

Nothing wrong with picking mongo if it's a good fit for your use case.

Exactly

And what's a good use case over Postgres jsonb?

Postgress is single master which is a huge limitation.

How so?

How do you scale a single master out of the box?

What's nice about Postgres is there's a ton of Postgres compatible products that do scale for the 10% who actually need it. And it's still all just Postgres / SQL.

Again it's not out of the box and your so called "tons" vary in quality and features.

When one doesn't want SQL for one.

Nosql is a fun target to beat up on of late. But there are good, even infamous, reasons to avoid SQL. Particular if you want to accomplish flexible record queries from untrusted clients.

I’ll ask again, what’s a good use case over Postgres jsonb.

All you do is poop all over the story about postgres. I'm convinced that no use cases will convince you of anything. I'm not really looking to involve myself in a database holy war.

Is jsonb in Postgres not flexible enough? I dump external json in there all the time (like large API responses). The jsonb operators work well. And there's an escape hatch that lets you easily convert json to a table. And importantly, you get indexes with Postgres.

I apologize for being short with you in my previous response. I need to be better about assuming good faith.

Your response seems to betray a certain ignorance of mongo. Don't you realize that mongo has indexes?

For handling documents, this blog post covers the topic in some depth.

https://jimb-cc.medium.com/postgres-jsonb-meets-mongodb-part...

Might I suggest you actually use mongo on a project before engaging further on the topic. It clearly has had a lot more thought put into it in the context of json documents.

If postgres works great for your use cases, great, go be happy with the tool.

https://www.dbvis.com/thetable/postgresql-jsonpath/

https://github.com/obartunov/sqljsondoc/blob/master/jsonpath...

Definitely a better solution than the brick wall he hit. Complicated beastie. I can see the appeal if you just love, eat, and breath SQL.

But a query language within a query language feels bolted onto something alien to do something pretty basic. Pass. If I'm teaching this to new engineers, pass also.

Oh yes. By people who do not or does not care about Open Source... and these are many

Wow lucky I moved our data out not too long ago. Trying to login to MongoDB, I'm just getting "server error" now.

"The login issues are unrelated to the security incident."

Almost decided to use MongoDB in a project for the first time.

Kind of makes me unsure if it’s going to be the right choice.

No reason to use mongo imo

Agreed. It's an over optimization and most problems can be solved by traditional relational databases.

Mongo is never the right choice. Postgres is nearly always the right choice, however.

Legitimate question, please don't downvote.

Are you basing this opinion on:

- popular HN opinion

- issues that Mongo experienced in its infancy

- mis-modelling highly relational data on a non-relational DB, and blaming the DB for ensuing problems

Or are you basing it on extensive experience with wide range of use cases?

Much like saying “no offense” doesn’t make something not offensive, saying “honest question” doesn’t make a question not disingenuous. If you find yourself typing “don’t downvote” you should consider rephrasing your question to not be worthy of downvotes.

This post could just be “can you explain your experiences that have lead you to this conclusion” and we’d all be better off.

We started our startup on Mongo. Hit some pretty hard performance problems and eventually did a multi month long migration to Postgres (Aurora on AWS).

MongoDB is only a valid choice if all you're doing is story key document pairs. The moment you need joins or any sort of aggregations like count/sum e.t.c - Mongo perf is horrendous. Postgres runs circles around Mongo in every way.

With jsonb columns, not much is lost. SQL is a huuuuge bonus. Mongo query language is a giant pain for everyone on team to learn and manage.

I'm of a similar opinion. I experienced all the issues of MongoDB's infancy; it wasn't a good time, but the mmap functionality seemed worth it at the time if you had enough memory or constrained your access patterns appropriately.

Nowadays, PostgreSQL has JSON/JSONB types, a full suite of extensions like pgvector and PostGIS, and I can scale with Citus or use it in any of the big managed clouds.

From a functionality perspective, MS SQL Server makes a more compelling alternative to me simply by way of its native graph database support.

Mostly that Postgres is amazing, runs anywhere and supports jsonb (which I use a lot). And I don't want to worry too much about mis-modeling; that's the entire point of a relational database with a type system.

Plus all the Mongo horror stories from people who I know and hold in high esteem. And recently talking to a data person explaining what a pita getting data out of their Mongo monstrosity into a format adequate for analysis.

The problem here is with the hosted service, not the database itself or its performance.

If you’re trying to solve a problem and think “I’ll use mongodb”, well now you’ve got two problems.

Just pick postgres. If you have unstructured data as input, either put in the effort to create some kind of schema for it if you can or just use jsonb if you can’t.

Reply on news.ycombinator.com