rzezeski

> I apply for jobs because I want money. Not to reflect or learn about myself.

We are not the same.

I wrote a detailed walk-through of the illumos syscall handler back in 2016. It's missing the updates around KPTI (meltdown mitigation), but other than that the mechanisms should be unchanged.

https://zinascii.com/2016/the-illumos-syscall-handler.html

Probably more to do with the fact that everything is moving towards virtualization. Oftentimes these NICs dole out VFs directly to VMs via SR-IOV, in which case I imagine the NIC controller has some safeguards to keep the host and the rest of the guest's safe from denial-of-service and other attacks from a malicious guest driver.

> During those tests, we noticed the machines were randomly freezing after some time, so we decided to upgrade the firmware of the network cards,

Reminds me of the various i40e Tx freezes I debugged while at Joyent. Granted, this is the illumos driver, not Intel's, but basically there were issues with the programming guide that I had to figure out the hard way. The 700-series controllers have not been the easiest to work with.

https://smartos.org/bugview/OS-7492 [Tx freeze when b_cont chain exceeds 8 descriptors]

https://smartos.org/bugview/OS-7457 [i40e Tx freezes on zero descriptors]

You just figured it out.

That's a good question. I should have mentioned that in the post perhaps. The reason I initially ran vmstat was just to make sure my dev machine was "zeroed out". Recently my kernel builds had taken twice the time as normal and I wasn't sure why. That's when I noticed this. It turns out this had no significant effect on my build times but was rather an effect of a different change I made weeks prior. I also noticed no ill effects on my laptop that I recall.

According to POSIX.1b RT signals are used for other features defined in .1b. An excerpt from "POSIX.4 Programmers Guide":

> So far, we've talked about real-time signals sent by the sigqueue function. However, most POSIX.4 signals are sent as a result of three other facilities in POSIX.4. We'll cover these facilities in detail in their own chapters, but for now, you need to know that realtime signals can be generated as a result of expiration of a POSIX.4 timer, completion of asynchronous I/O, and by arrival of a message on an empty message queue.

Linux reserves 2-3 RT signals for use in its threading library:

http://man7.org/linux/man-pages/man7/signal.7.html

All kernels do not use the same libc. In fact, Linux is different in the fact that it doesn't ship a libc with the kernel. Illumos, the OS I investigated, goes by the philosophy that you _have_ to ship libc with your kernel, and thus they have their own libc.

My diagnosis for Illumos may not hold up on other OSs, they may invert priority for other reasons. In any case the spec is fairly clear that a signal is considered delivered when the "appropriate action" is taken. In this case the appropriate action is to invoke the user's handler, which is above libc on the stack.

Finally, in case you missed it, I witnessed 4 unique orderings across the systems I tested. They were not the same across the board.

Author here, I won't get to it today but I would like to try the method you describe. I realize that benchmarking is a very tricky thing having followed Brendan Gregg's work the last couple of years. This is why I also ran my baseline benchmark which simply performed a ret. I learned years ago from tuning cars that absolute numbers are less interesting than relative. That said, I am a total newb at low-level stuff like this and I would not be surprised if I messed something up. I'll be curious to use your method and see how it compares.

Hi, author here. My align_2 implementation is indeed very naive. Part of that was to show just how much overhead a naive implementation could add, in this case 3 orders of magnitude. I'm also new to systems programming. This article should serve as a reminder to experienced hackers that idioms are often only obvious _after_ they've been explained. My last 4 years were spent working on a distributed database; I could bring up "obvious" idioms from that world that might perplex a kernel hacker.

But you are right, this is indeed a very obvious line of code once you understand it. Thank you for your comments.

Not only that, why make every user reimplement the same resolution strategies? Or make Basho dev's reimplement them for each official client? You'll increase the chance for bugs or subtle semantic differences. Also, many users don't know or want to understand conflict resolution. They want a data structure that just "does the right thing". This is most easily achieved by baking it into the database itself.

Disclaimer: Former Basho developer with a lot of pride in the 2.0 release.

There may be more to the situation. I wonder if Normalcy Bias also played a role in Air France 447.

Replicas (or as you put it, ring replication) is critical, and Riak very much has replicas. What it doesn't have in it's open source version is multi-ring replication (cross DC) which is a separate concern.

In the Dynamo paper the ring spans DCs but they also have a very different network than most that allows them to do that. In Riak it is recommended that each ring is contained in a single DC. If you want ring-to-ring replication from Basho then you can pay for Riak EDS. You could also build it yourself as others have mentioned (Kresten Krab Thorup has done something like this in Riak Mobile [1]).

Nothing is stopping you from running a single ring (cluster) across DCs, and it might even be okay for certain apps, but it's not a choice that should be taken lightly. In general, if you don't understand the tradeoffs you're making in that regard then it's best to stick to one ring, one DC.

[1]: http://www.erlang-factory.com/upload/presentations/413/Erlan...

I agree in general with the sentiment but don't like rule of an absolute # of slides. To this day one of the best presentations I've ever seen had 500+ slides and is an example of what a good presentation looks/feels like.

http://www.infoq.com/presentations/archaeopteryx-bowkett

I work at Basho. Specifically I've been doing a lot of work on Riak Search lately. I absolutely agree that the Search documentation is underwhelming and could use some major love. I take it as a personal challenge to improve the situation before the 1.0 release comes out.

That aside, you might like to know that for the 1.0 release Search is bundled along with the standard Riak package (you just need to set a flag to enable it). You can try using the latest pre-release or even build from source on master or the 1.0 branch. If you don't actually need full-text search then you could also checkout Riak's new support for secondary indexes. Please drop a line on the mailing list or IRC if you have questions.

If we are talking about performing range queries on an index then Riak already has it in the form of Riak Search. In 1.0 this is also supported by secondary indices.

If we are talking about performing a range operation on the primary key which returns the matching objects, then no, Riak doesn't currently offer that. However, given it's support for an ordered data store such as leveldb in 1.0 it should only be a matter of time before that is possible.

Just to try it out I already implemented this for fun on my fork.

https://github.com/rzezeski/riak_kv/tree/native-range

https://github.com/rzezeski/riak-erlang-client/tree/native-r...

I know nothing about Switft so I'll keep my comments strictly related to Luwak.

Luwak preforms really well as a large object store...up to a point. I've personally seen latencies of < 4s when writing/reading a 700+MB CSV across 3 physical nodes (keep in mind, in the case of write, that doesn't mean all the data was strongly consistent yet...it's all async). Luwak has a really cool feature that also acts as a double edged sword -- it's a persistent data structure [1]. When you insert data it's chunked into blocks which are then keyed by their hash, i.e. a Merkle Tree [2]. If 2 (or more) blocks of data are the same only one instance will ever be created which acts as a general form of compression. The flip side of this is that you can't just willy nilly delete a file. You must perform garbage collection (via reference counting). Currently this is not implemented in the main line but I have a branch with a prototype implementation that uses Map-Reduce under the covers [3]. It scaled for me up to about 20+ GB of data and then I started hitting timeouts. I had plans to take this further but went a different direction for my purposes (which wouldn't relate to your problem at all so I'm not going to bother stating them).

[1]: http://en.wikipedia.org/wiki/Persistent_data_structure

[2]: http://en.wikipedia.org/wiki/Hash_tree

[3]: https://github.com/rzezeski/luwak/tree/delete2-1.0

You are performing a GET for the same key each time?

Don't quote me on this but I took a quick glance at the code and it looks like all gets are [currently] serialized at the vnode level. Since your key will always hash to the same vnodes (increasing machine count doesn't help) it means that request will pile up on the vnodes' mailbox.

I also see an asynchronous get defined in the vnode but don't see anyway to use it.

That said, if this isn't an artificial use case then I'd probably throw a short-lived cache in front of Riak for situations like this. I mean, if this were a more traditional database, you wouldn't run the same query 5000 times if you know the answer isn't likely to change.