838 karma · joined April 2, 2010
javascript:window.location="http://"+window.location.search.match(/id=(\S+)/)[1]+".com"
* Javascript can be used to program these things and is quite capable of handling low-level kind of stuff
* Since it's easier to write js code than c/c++ code (for me), I would personally prefer it; assuming that the performance degradation is within allowable limits
I saw you mention in another comment that memsql stores almost 30x the (rows) data as mysql by the end of the experiment. I would be curious to know why this is so, and what that data is.
Also, from what I understand after reading the paper, the lock-free skip-list node is pretty heavy (more than the 3~4 pointers that a naive implementation would use). What is the # of bytes it occupies in memory. Assume that the data field is just a pointer (8 bytes).
Curious: Do you always store the data as a pointer or does it get stored inline for smaller types?
1. How many rows were actually present in the tables (in all)?
2. What did the VSZ and RSS for mysqld and memsqld(?) look like at peak and after benchmarking was completed?
3. What did the CPU usage of each mysqld and memsqld(?) look like at peak and what was the total amount of processor time used by each database?
4. As a percentage, how many statements were inserts and how many were queries - am guessing I might be able to get an idea my looking at the code, but it's easier to ask - esp. since you'll know more about the code.
5. AUTOCOMMIT ON or OFF? (dunno what the defaults are).
Do you mean you determine placeholders without actually parsing the SQL?
@jakejake The docs. seem to suggest that the benefits of prepared statements are won without using prepared statements in MemSQL.
SELECT * FROM table WHERE id > 5 LIMIT 10;
is actually of the same form as this query:
SELECT * FROM table WHERE id > 10 LIMIT 5;
without actually parsing the two. I mean you will incur a parsing overhead either way. What I think MemSQL is trying to optimize out is the actual execution of the query. I mean rather than interpreting and running it with checks performed in every iteration, they are compiling the code. I don't know how much of a difference this would make without actually seeing the numbers.
Additionally, skip-lists can be used for performing range queries as well. This means that "SELECT * FROM table ORDER BY id DESC LIMIT 5;" can be executed very efficiently as long as there is a skip-list on "id". Additionally, even COUNT(), MAX(), MIN(), etc... queries can be optimized in a similar fashion. Again, however, doing it in a lock-free manner might not be the most fun thing to do.
I agree with the fact that getting the optimization isn't intuitive, but the basic O(n^2) on which the idea is based is almost trivial.
"Have you ever used the STL? I did for about 6 months professionally, and I'm glad that I could go back to C and Python. More seriously, object orientation in C++ doesn't come for free. Aside from the pain of the syntax of using C++ templates (and difficult to parse error messages), all of that object orientation hides the complexity of method dispatch in what are known as vtables."
Seriously?
1. Python is faster than C++?
2. All OO features (method invocations) in C++ use the vtable?
IMHO, just using the right iterator abstraction and using the STL algorithms should be within performance limits.
I've posted 3 other links which should allow free access to the paper. I think these things should be open (and free) to start off with.
Another note: An LPE of a book that is available for INR ~500 would cost anywhere from INR ~4000 to ~6000 if the original publication is ordered. Generally the original is not stocked so there is a wait period of ~5 weeks from placing the order.
For example, suppose you start off with a B-Tree with size 1000. Once it reaches 2000, re-write it. This way, the query performance remains good. Next time it reaches 4000, re-write it again, then at 8000, and so on... This way, you get good query speeds even in the presence of random inserts.
You need not re-write the B-Tree on the 2000'th or 4000'th insert. Instead, you can start the re-writing process when the size of the B-Tree reaches O(n/log n). This way, for the last O(log n) values, every time one value is inserted, we copy O(log n) values from the old tree to the new one. When we are done, we have an almost sorted list of values that we inserted in the new B-Tree!