#database #right tool for the job #programming
epoch: 1300444500
After been forced to play with different NoSQL systems for a while in a high load environment, eventually I had an insight: You should never rely on a database system based on a technology with non-deterministic garbage collection. If you cannot say when exactly your memory is freed - don’t use it to implement a database. Don’t use Java or .NET, don’t even use Erlang regardless of the partially used reference counting GC. And don’t use a database written by third party for one of the former platforms.
What counts here is not plain performance or speed but knowledge. It should be predictable when exactly your program suffers from freeing memory, at least in a long running server it should be. A typical database usage scenario is to get a high number of requests per second from a number of different clients where each request is processed in very short time producing not that much data (with the exception of blobs). This results in a huge number of garbage being produced since all the stuff is no longer used or has to be recalculated after the request has been processed. What you really want is the garbage to be destroyed as soon as it is no longer of any use to have a smooth application behavior. It’s smoothness what counts if you want to scale horizontally. The opposite is a collection sometimes when the runtime decides. You need good luck not to hit a load peek that way. The longer the server is running the higher the chance to miss the luck.
At the end of the day I saw all those Java databases run into severe garbage collection problems with those stop-the-world-pauses and even the most sophisticated juggling with even the most secret -XX options could not sweep the fundamental problem out of the way. A database can only be build upon a completely predictable runtime. That’s why people in the know write databases with C/C++. Sure, the cost to produce such a system might be a little higher compared to Java or .NET or Erlang. Whatever: Databases are not build for the impatient but for the settled.
Powered by Tumblr; designed by Adam Lloyd and Ingo Schramm.
