facetube

Yep. totally safe, someone on the internet said we should.

One of the many practical vulnerabilities.

The claim from daimenkatz was that transactions and MVCC are supported internally, but not externally. It's unclear to me if any of this is accurate. Here's the information that was offered: https://github.com/couchbaselabs/couchstore/wiki/Format. Let me know if you see MVCC or transactional behavior in there.

Multiple people. Please do your research before posting.

You made a claim. Support it or withdraw it.

So you have nothing to refute the available evidence. Thanks for the informative response. Don't make claims in public you can't support.

That's why you'd use a cut-out: plausible deniability. Russia has attempted and/or carried out multiple hits on NATO soil in this exact manner, including one with a novel nerve agent.

You have no evidence to support this.

You have absolutely no evidence of this, and there have been multiple independent medical examinations and diagnoses.

Easy enough to convert on the way out, using a mature database system without these performance flaws. The app got done and performed well, and we weren't at the mercy of an unresponsive community that leaves seven year old dependencies in critical paths.

Keep going; it'll bite you eventually. Don't say you weren't warned.

One of the stated goals on their Spidermonkey replacement issue (N.B. it still won't fix the view server protocol performance, but at least it'll be a modern engine) is "Get my C skills back out of the closet", and will require https://github.com/jquery/esprima.

It's depressing.

> I‘ll give you no cite, sorry, because we discovered the effect during internal tests

We discovered the issue in internal tests and reported it upstream where it was confirmed; there's nothing to discuss here.

> BTW they may be patched without upgrade, with 5LOC long design document in Erlang.

I don't have to hand-patch other database systems. Further, as of now there are no functioning packages for multiple versions of Ubuntu. Multiple competitors do not have any of these problems.

> Last CVEs has nothing to do with Spidermonkey

It was a full RCE that didn't have vendor packages ready in time.

Sorry, but using seven year old language runtimes is daft. It might be fine for you, but it's not appropriate in environments where you care about security and performance, or care about the overhead of making your team reason about these things unnecessarily.

> We sometimes measure number of docs with M postfix

Yeah, we did this in 2003 on commodity hardware, and even it built indexes faster than CouchDB builds map/reduce indexes. Fix the external view server protocol or be honest with people and kill it off – the status quo is unacceptable.

Finally, here's a five year old issue that's still open admitting what I just explained: https://issues.apache.org/jira/browse/COUCHDB-1743

What's your definition of huge?

I suspect the bug bounties are much more about deterring the sale of exploits to bad actors than about recruiting employees.

Correction: the library is too old for that specific CVE to be exploitable. There may be others, there may not be. I don't know – and that's actually the problem.

The chief complaint here is that it's very hard to reason about the security of seven year old software that distros are pulling out of their package repositories citing security issues (e.g. Alpine Linux). Advisories tend to say "versions before X", and you're left to either wonder or read tickets/code. Not everyone wants to sign on for that, and it's work that isn't going toward writing your app.

Also for balance: it's unclear if there are CVEs. There may not be any that are exploitable due to the age of the library (i.e. the bugs landed post-release). However, it's also very tricky to rule these out one by one, and the dependency is slated for removal from some distros.

There may be no material security issue. But the age of the software does not instill confidence, and newer engines are likely far more performant.

Thanks a lot; this is super-helpful.

Thanks for the clarification. Can I ask how the MVCC is implemented (e.g. WAL, MVTO, MVRC, etc.), or if there's documentation around that might give some additional insight?

> Except CouchDB entire REST API, where nearly every endpoint streams result row by row

I'm talking about the internals of couchjs.

> As of security: wrong guess. Go compare number of critical CouchDB CVE

I'm talking about Spidermonkey 1.8.5, which is seven years old. Multiple CVEs have been reported including a full RCE, which was patched in later versions. If you believe these are patched in libmozjs185, I'd love to see a link to the source package showing where the fix was applied. Because I can't find it.

Further: https://www.cvedetails.com/vulnerability-list/vendor_id-45/p.... Another full RCE in CouchDB.

> Performance in this area depends much more on how branchy is the json doc being processed

Citation needed. There's an issue specifically about this in the CouchDB issue tracker if you'd like to read more.

> As for query server: indexing is slow for the reasons you pointed out.

Right. It is not performant. That's my point.

Curious what kind of database sizes you're working with too; you won't see this stuff with a few hundred documents.

> Or you can use a query language mostly similar to what MongoDB uses

Yes, provided you started on 2.0, as I specifically stated in a reply.

> That's a feature. If they don't usually change together put them in a separate design documents.

Unnecessary computation and I/O is not a feature. They should hash the views individually. A database that makes me do extra work to keep it from doing unnecessary work is not helpful.

> So managing documents yourself is kind of what you do. Isn't it?

This doesn't have anything to do with the unnecessary index rebuilds. CouchDB has all of the information it needs to avoid this unnecessary computation and I/O.

> So have more shards

More shards spending time serializing and deserializing JSON instead of doing useful work. Throwing shards at inefficient code might help, but it doesn't address the underlying problem.

> Shared memory IPC doesn't magically solve problems but it is a minefield of bugs.

Lots of database systems manage to use it just fine. You probably use one of them. And it does solve a problem: it avoids serializing and deserializing all of your data down a pipe.

> Large blocks of allocated memory use malloc instead of brk so memory allocation uses mmap.

Yes, with MAP_ANONYMOUS. I'm not talking about anonymous mappings.

> Hmm, seeing kswapd being active is not something I look forward on a production server.

For swap? Sure. For file-backed mmap pageout guided by madvise? I actually do. It means the VMM is being used for its intended purpose.

> bulk docks is not a transactional feature

It was an example of the one way to achieve an atomic write of multiple documents, which comes in very handy for any nontrivial application.

I'd honestly be fine with them just deprecating the entire external view server protocol – but as it stands now, it's a supported feature, the performance documentation in 5.2.5 of the manual is nonsensical, and it performs very very poorly due to excessive serialization/deserialization and inefficient IPC.

Exponential population increase would certainly drive increased demand for water.

From the point of view of a single cluster or node (ignoring replication for a moment), it just doesn't have any strong notion of transactions. There's no write-ahead log, no rollback, and no situation where you'd be able to operate in a mode equivalent to something like `SERIALIZABLE` on a relational database.

For balance: there is a newer query/index system called Mango in Apache CouchDB 2.0+, that IIRC is internal and doesn't rely on any external view server. It wasn't in 1.7.1, though, so if you're coming from there, it's very much a "switch query APIs to get tolerable performance" situation.

He was right to abandon CouchDB. Here's my rationale: https://news.ycombinator.com/item?id=17116306

Sorry, wasn't clear if you were talking about MVCC in a replication context or in a node-local context. It does the former, it definitely does not do the latter.

It doesn't use real node-local/shard-local MVCC AFAICT. It's O_APPEND on flat files.

God, where to start.

Map/reduce views ("secondary indexes") are currently built by serializing JSON down a pipe to an external process called couchjs that links against a seven year old version of Mozilla Spidermonkey (1.8.5, released on March 31st 2011, and the only version it currently works with). This external process then pipes the map results back, again as JSON. Forget zerocopy; we're serializing and deserializing twice per document. Almost all of this "external view server" (the only kind AFAICT) is actually written in Javascript on that seven year old engine. Nothing is streaming; your map function is sandbox-evaled in that process, the results of a single call are stuffed into a Javascript array via .push, then stringified and printed on standard output as a single line. The entire internal "view server" protocol is based on a pipe of single-line JSON commands and data payloads. Performance (and likely security, due to age of the engine) in this subsystem is extremely poor IME.

You can write map/reduce views in Erlang itself as an alternative, but that engine is disabled by default due to security concerns – it's "not sandboxed". Performance in this area also appeared to be proportional to the code size of the map function, which sounded a lot like it was being serialized down a pipe with no caching.

If you make a change to a single view in a design document, the database system will rebuild all indexes in that design document unconditionally, even if the other view code hasn't changed. The only way to prevent this is to store a single view per design document, and manage those documents yourself.

Once you've endured your multi-day index rebuild on a couple million documents, you'll also have to issue an extra "cleanup" command to delete the old (now essentially useless) indexes. It's a full rebuild, so you need 2x the space at a minimum to pull it off.

The data itself is stored in a single file per shard, opened in O_APPEND mode, effectively serializing all writes on the shard. You also will not see kswapd active, ever, because the OS VMM is not being used effectively – no shared memory for IPC, and no use of mmap AFAICT.

Server-side changes feed filters appear to be extremely slow when compared to just filtering the results in the client.

Authorization: limiting read access will be difficult at best. Transactional behavior: no, aside from a single bulk write feature (make sure to use "all or nothing" mode though).

Logging: signal to noise ratio is poor. Giant stack traces for minor events. You may even see strings emitted as UTF-32 arrays of integers in base 10. Hope you brought an ASCII table at least.

Documentation: The following passage from section 5.2.5 of the CouchDB v2.1.1 manual just about says it all: "Views with the JavaScript query server are extremely slow to generate when there are a non-trivial number of documents to process. The generation process won’t even saturate a single CPU let alone your I/O. The cause is the latency involved in the CouchDB server and separate couchjs query server, dramatically indicating how important it is to take latency out of your implementation." I can't say anything nice here about that last sentence, so I'm just going to keep my mouth shut.

But worst of all: every time it starts up, it tells me "it's time to relax". Nothing about this is relaxing.

(These are my own private individually-held views, and do not reflect the views of any employer or organization)

> as of LAST NIGHT, there was a mention on 1password.com of using DropBox and other mechanisms for syncing vaults

Below is exactly what the cited page said on February 1st – as far as I can tell, this post offers absolutely no support for this claim.

> Dropbox requires:

> 1Password 4 on Windows [1]

[1] https://web.archive.org/web/20180201210656/https://support.1...

You wouldn't be fined. You'd be arrested and charged criminally, and likely spend the rest of your life in jail. The public does not understand and would happily hang you for being one of those evil hackers. The person who tells them you belong there would be a $1000/hour professional witness.

Infinite-term copyright builds dynasties. They do not take kindly to competition. Stay safe.

Until the cops kick your door in on copyright infringement charges to "make an example" of someone. Don't think it can't happen.