#amqp #cluster #distributed computing #rabbitmq #erlang
epoch: 1322214060
Clustering RabbitMQ is very easy - if you know how. Unfortunately, the documentation on this topic is good but not good enough (cf. RabbitMQ Clustering). If you try to do it, you may get lost on the track until you find some insightful posts on the mailing list. This is why I summarize here how I got it to work.
Say, you want to create a cluster having two disc nodes and two ram nodes. If you do this on at least two machines, each having a disc and a ram node you achieve good fault tolerance and good scalability both with one setup. Your clients may connect to the ram nodes only or these are balanced by an additional load balancer.
But, how do I make a node a disc node and another node a ram node?
There’s no such command like “rabbitmqctl mkdisc” and there is no related configuration option. On one hand, this is a little counter intuitive, on the other hand this adds a lot of flexibility since you may alter the roles of nodes and restructure your cluster on the fly whenever necessary.
The rules are assigned by the way you call the “rabbitmqctl cluster” command. In our scenario, we have multiple nodes on the same host, so we need to wrap the calls to “rabbitmqctl” into shellscripts setting some environment variables (cf. RabbitMQ Configuration). If this has been done, you ensure all nodes of the cluster are running. Afterwards you execute a sequence of “stop_app”, “reset”, “cluster”, “start_app” commands for all nodes. If it comes to the “cluster” command, you add a space separated list of all disc nodes you want to create to the “cluster” command executed for each node. My mnemonic for this is that you copy the current node to all disc nodes. The whole sequence may look like this, with “rbctl.*” being your wrapper scripts:
host-of-disc1$ rbctl.disc1 stop_app
host-of-disc1$ rbctl.disc1 reset
host-of-dics1$ rbctl.disc1 cluster disc1@host-of-disc1 disc2@host-of-disc2
host-of-disc1$ rbctl.disc1 start_app
host-of-ram1$ rbctl.ram1 stop_app
host-of-ram1$ rbctl.ram1 reset
host-of-ram1$ rbctl.ram1 cluster disc1@host-of-disc1 disc2@host-of-disc2
host-of-ram1$ rbctl.ram1 start_app
host-of-ram2$ rbctl.ram1 stop_app
host-of-ram2$ rbctl.ram1 reset
host-of-ram2$ rbctl.ram1 cluster disc1@host-of-disc1 disc2@host-of-disc2
host-of-ram2$ rbctl.ram1 start_app
host-of-disc2$ rbctl.disc2 stop_app
host-of-disc2$ rbctl.disc2 reset
host-of-disc2$ rbctl.disc2 cluster disc1@host-of-disc1 disc2@host-of-disc2
host-of-disc2$ rbctl.disc2 start_app
If you have to add users, vhost and permissions, you better do it at the end of this procedure, otherwise the “reset” will delete all of this information. Also, if you want to change the cluster setup later, you should be careful with “reset”, omitting it for one disc node at least.
Another weak point with the whole clustering stuff is the location of the “.erlang.cookie” file. This file is essential for clustering and must have the same content for all nodes in the cluster. Documentation says RabbitMQ looks at “/var/lib/rabbitmq/.erlang.cookie” but I found this not always true. Supposed RABBIT_HOME points to the directory where the rabbit distribution is located, I copied the file to “$RABBIT_HOME/../.erlang.cookie” and RabbitMQ used this one. I’m not quite sure if this is a general rule.
#Functional #programming #erlang #scala
epoch: 1321520074
I regret. Before I learned to think and to act functional, I proclaimed X=X+1 for years. Have mercy!
#programming #Functional #scala #erlang
epoch: 1321520002
Why functional is better? So many reasons, one comes here: Programming languages are not made for computers, they are for humans to formulate generic solutions computers can solve faster in the details. The imperative style formulates the steps the computer has to follow, the functional style formulates the generic solution as such. Imperative style wants you to act like a machine, functional style allows you to think like a human.
#erlang #patterns #abstraction #mathematics
epoch: 1265974860
Thinking about how useful Erlang parameterized modules might be I found a special use case I may call a Functor. Consider the following code. This presents a generic module representing a monoidal structure.
-module(monoid,[Impl]).
-export([op/2,cmp/2,get_id/0,get_member/0,is_member/1,proof/0]).
%% ------ INTERFACE -----
% apply monoid operation
op(A,B) ->
case members([A,B]) of
true -> Impl:op(A,B);
false -> undef
end.
% compare two values if they are equal or not
cmp(A,B) ->
case members([A,B]) of
true -> Impl:cmp(A,B);
false -> undef
end.
% get the identity or neutral element
get_id() ->
Impl:get_id().
% get an arbitrary member element
get_member() ->
Impl:get_member().
% test whether A is a member or not
is_member(A) ->
case Impl:is_member(A) of
true ->
Impl:is_member(Impl:op(A,Impl:get_id()));
false ->
false
end.
% give proof of monoid laws
proof() ->
prove().
%% ------ PRIVATE ------
prove() ->
I = Impl:get_id(),
A = Impl:get_member(),
B = A,
Op = prove_closed(A,B),
C = op(A,B),
Assoc = prove_assoc(A,B,C),
Id = prove_identity(I,A),
neg(lists:member(false,[Op,Assoc,Id])).
prove_closed(A,B) ->
Impl:is_member(Impl:op(A,B)).
prove_assoc(A,B,C) ->
Impl:op(Impl:op(A,B), C) =:= Impl:op(A, Impl:op(B,C)).
prove_identity(I,A) ->
0 =:= Impl:cmp(A,Impl:op(I,A)).
neg(true) -> false;
neg(false) -> true.
% empty set is always member
members([]) ->
true;
members(L = [_|_]) ->
neg(lists:member(false, [ is_member(X) || X <- L])).
Having this you can write a module implementing a concrete monoid, this time the monoid of the natural numbers under addition. In a similar fashion you could implement vector or matrix operations.
-module(nadd).
-export([op/2,cmp/2,get_id/0,get_member/0,is_member/1]).
op(A,B) ->
A + B.
cmp(A,B) when A =/= B ->
-1;
cmp(A,B) when A =:= B ->
0.
get_id() ->
0.
get_member() ->
1.
is_member(A) when is_integer(A), A < 0 ->
false;
is_member(A) when is_integer(A), A >= 0 ->
true;
is_member(_A) ->
false.
When you pass this implementation as a parameter to the generic module you can use it as in the following code.
1> M = monoid:new(nadd).
{monoid,nadd}
2> M:proof().
true
3> M:op(1,2).
3
After reading this you may ask what these modules actually are. In terms of the Gang of Four, is the generic monoid an abstract factory? Is the implementation a strategy? Well, somehow both may be true. But exactly the “somehow” treats me to say, in fact all this is something different. It is a Functor. To put it more precisely it is the identity functor taking the monoid of the natural numbers to itself. You may check the functor laws in the shell again: 1) the functorial mapping of the identity arrow equals the identity arrow of the functorial mapping and 2) the functorial mapping of a composition of arrows equals the composition of the functorial mapping of these arrows. All this under the assumption that you treat the monoid as a category, and this is completely legal.
1> M = monoid:new(nadd).
{monoid,nadd}
2> N = monoid:new(nadd).
{monoid,nadd}
3> M:op(1,N:get_id()) == N:op(M:get_id(),1).
true
4> N:op( M:op(1,1), M:op(1,1) ) == M:op( N:op(1,1), M:op(1,1) ).
true
Sure, this is only a simple example of a Functor. In theory at least it is possible to add more structure to the generic module as well as to create more sophisticated functors. For example, you may add an inverse operation to turn the monoid into a group or you may create more complex functors with a little luck.
Monoids, Functors and similar structures appear to me the more functional variant of patterns and since Erlang is a (weak) functional language these should be patterns in Erlang as well.
#erlang #oop #patterns #undocumented #syntactic sugar
epoch: 1260352260
The story of module extension continues. Another still undocumented feature of recent Erlang versions is to have parametrized modules and module instances. Have a look at this code:
-module(mytest, [Var1, Var2]). -export([getInstvars/0]). getInstvars() -> [Var1,Var2].
With this you can have variables known in module scope. These variables are of course immutable. Moreover you can instantiate a module and bind a variable to that instance. Then you can use the variable as you would do with objects in an OOP language. This is a corresponding session, using the automatically exported new/2 function and a kind of anonymous instance:
1> H = mytest:new(1,2).
{mytest,1,2}
2> H:getInstvars().
[1,2]
3> {mytest,4,5}:getInstvars().
[4,5]
I’m still not sure if this is a good idea. The migration from your prefered imperative OOP language to the functional world of Erlang may become easier if you recover some of your well known structures but it may lead to misuse before you understood the benefits of functional programming well enough.
I’d rather appreciate the introduction of packages or namespaces to structure modules hierarchically in Erlang, or the nesting of modules like OCaml does. A blueprint of parametrized modules may be OCaml functors as well but I do not (yet) know enough about that language. We will see how the community will use or misuse this feature.
#best practice #erlang #patterns #undocumented #syntactic sugar
epoch: 1259593980
Surprisingly, there exist a number of Erlang features rather undocumented and obviously not part of the official picture of the language. One such feature is module inheritance. It is possible to extend a module by another and inherit or overwrite functions. This sounds like OOP and in fact it looks like class inheritance. The following code shows a simple example.
-module(alpha).
-export([f1/0, f2/0, f3/0]).
f1() ->
alpha_f1.
f2() ->
alpha_f2.
f3() ->
alpha_f3.
-module(beta).
-extends(alpha).
% f2 not exported
-export([f1/0, f3/0]).
f1() ->
beta_f1.
f2() ->
beta_f2.
f3() ->
?BASE_MODULE:f3().
And this is a corresponding eshell session:
Eshell V5.7.4 (abort with ^G)
1> alpha:f1().
alpha_f1
2> beta:f1().
beta_f1
3> alpha:f2().
alpha_f2
4> beta:f2().
alpha_f2
5> alpha:f3().
alpha_f3
6> beta:f3().
alpha_f3
Now one may ask why such an amazing and by a good amount of Java people eagerly awaited feature is hidden. Well, probably because it could become a source of bad habit.
In object oriented programing it is considered good style to prefer composition over inheritance. So why should Erlang support extension when it is already build into the language to enforce strong functional decomposition? Why mess around with bugs caused by nebulous call stacks? Function f() is called, but it produces not the expected output. Why! Why? Simply because you forgot to export it in the extending module and it is called at the base module. Without extension you would have a suggestion by the compiler.
There may be very rare situations when module extension makes sense, for example when you find no other way as to rewrite a heavy module like gen_server to meet your needs. Most of the time it is simply bad practice and should be avoided. Don’t push your bug statistics by using undocumented sugar.
#erlang #otp #system architecture #right tool for the job
epoch: 1243455720
Erlang/OTP is without any doubt a powerful tool. But as with any technology it’s not good for any use case imaginable. It has a lot of strengths but some weaknesses too. Erlang glances in everything related to distribution and reliability. If you want to write software to be arbitrarily distributed over multiple instances, CPU cores, mashines or even data centers without having to change the programming paradigm for any particular case you probably have the perfect tool with Erlang. To benefit from Erlang you should have a problem best solved with a distributed solution. This may be reliable message passing or rock solid in memory storage but it is probably not web page generation and delivery.
In the Internet world, Erlang is perfect for middle ware or backend systems like caches, message queues and exchanges, databases or storage abstractions. Also, map/reduce is build into the language. But it is probably not the right tool to write web applications. Sure, you may add a HTTP endpoint to your message middle ware or database but should you use Erlang in a way one usually uses JSP or PHP or Ruby? Probably not. Web applications have a very short life cycle. The business rules are in a constant flow and have to be changed over and over again. To define such rules in Erlang may turn out to be a very hard job. Despite of its expressiveness Erlang is not the language to be used as an embedded language as for Yaws dynamic content or in ehtml. It works - but it is comparably hard to maintain and not that fast in terms of execution time.
It’s not that Erlang is a functional language but it’s a functional language with a special syntax. There are all those commas, semicolons and dots and a pretty verbose notation for dictionaries, the records. The benefits of the syntax as for example how you deal with binary data on the basis of each particular bit is of little use when programming the business rules of a web application. It’s simply not designed for such a use case. It’s designed for writing reliable software dealing with network communication, binary protocols and related stuff. But it’s not made to express complex rules in a domain specific language.
On the other hand, the Erlang VM is highly optimized to spawn and execute a huge number of small processes but it’s probably not optimized to execute a thread in the shortest possible time ever. Modern Java implementations may beat current Erlang easily. Sure, the maintainers of the Erlang VM do a lot of work improving both performance and SMP scalability but all those optimizations are not yet at the end. High performance or number crunching is not the domain of the OTP.
When you ever think about using Erlang, the first question to ask should be whether your problem deals with either redundancy, scalability or distribution. Second is whether you not have to deal with high performance as well, and you should not have to deal with the low lifetime of business rules. If you answer these questions with yes you get a highly optimized and convenient tool for the job.
Powered by Tumblr; designed by Adam Lloyd and Ingo Schramm.
