ingo_schramm() Tech

Languages Are Overrated

Tue, 08 Jan 2013 11:12:39 +0100

By many people I met over the years of my professional career as a software engineer, programming languages seem to be vastly overrated. Every now and then I see a battle of this language against that language and a flame war and a shit storm. To what extend - I don’t know.

For me, languages are tools, no more no less. They are tools for humans to formulate repeatable problem solutions in a way a computer and other humans can deal with. Some languages fit better to that solution and some to another. Some ecosystems have that benefit and that drawback, some another. But, no single language supports you in finding the solution for the problem in question. And that’s the crucial point of engineering, to find solutions. Programming, in essence, is selection or creation of Algorithms, and not simply coding.

I never made a big deal about the language I use, and I used a lot. If I have to learn another, I do it, simple as that. Sure, I have a sense of beauty or smartness of a particular language, but in the end of the day I try to choose the right tool for my particular solution. And that’s what I care about the most, the solution, the algorithm. Moreover, this is the right starting point for optimizations by the way. I even helped optimizing code in languages I never used before just by analyzing the algorithmic complexity of the particular solution, an approach many programmers obviously have forgotten while discussing questions like: which language is the fastest.

It’s rather similar to what Nietzsche once said - who spoke a number of contemporary and ancient languages fluently -: it does not count how many languages you do know or do not know, what counts is if you have something to say in the first place. This is why I spent more time reading papers or books about algorithms instead of the most fancy “Programming in XYZ”. If one has something to say, he or she will find a way to say it. If one has not - no language can help.

Jun Fukuyama's P≠NP Page

Tue, 11 Dec 2012 10:04:30 +0100

Jun Fukuyama's P≠NP Page

Go Standard Library Web Server Rocks

Thu, 02 Aug 2012 20:35:00 +0200

These days I’m doing my first steps in the Go programming language. Today I wrote my web server hello world and I’m really excited. All I had to do was to write these lines of code.

package main

import (
	"runtime"
	"fmt"
	"flag"
	"net/http"
	"strconv"
)

func main() {
	// command line argments
	port     := flag.Int("port", 8000, "http port")
	maxprocs := flag.Int("maxprocs", 1, "GOMAXPROCS")
	flag.Parse()

	// setup processor usage
	runtime.GOMAXPROCS(*maxprocs)

	// http handler
	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintf(w, "Hello, %s", "world")
	})

	// http server
	s := &http.Server{
		Addr: ":" + strconv.Itoa(*port),
	}
	s.ListenAndServe()
}

Compilation took less than a second, then I could start the server:

$ bin/web -port=8000 -maxprocs=4

send some requests:

$ ab -c 100 -n 1000000 http://localhost:8000/

and this is what ab told me about performance:

Requests per second: 23454.77 [#/sec] (mean)

During that quick test, the web server allocated about 8MB of resident memory, while ab needed 34MB.

Sure, this thing does nothing useful. And also sure, I can reach similar numbers with an embedded Jetty or a POCO C++ embedded web server, but all I had to do to get this to work was to install Go, write that small code snippet and compile. No Maven XML, no painful dependency resolution, just some calls into the standard library and that’s it. Moreover, I have a single statically linked binary, 3.8MB in size, that I could deploy to whatever 64bit-Linux box I want.

This is practically amazing.

MVC, MOVE - Or Simply A State Machine?

Tue, 03 Jul 2012 10:32:00 +0200

In a blog I recently read, Conrad Irwin extends the well known MVC pattern to a MOVE pattern (http://cirw.in/blog/time-to-move-on). While his critics of MVC meets the right point (“… but the problem with MVC as given is that you end up stuffing too much code into your controllers …”) I cannot see what MOVE does better. It seems to be slightly more complex - and that’s what I would criticize.

What if we simplify things? All we need to get a separation of UI from data and processing in a stateful application are Views and a State Machine. And that’s all what MVC already is in its very core. We have a state machine with transitions initiated by events from the outside world and some kind of frontier or surface or membrane between that world and the state machine. That surface is what we know as View.

The view is a window into state with a given perspective. Not all may be visible through a given window, some state may be hidden, but moving over the surface will expose all state eventually. At some points at the surface we have sensory cells (those buttons) firing events into the state machine when touched, causing transition. And that’s what most MVC applications already do.

The perception of burdened controllers (and often views as well) is in essence caused by the fact that they simply doesn’t exist. Partly they are elements of the “M” - for Machine, partly elements of the “V” for View. And if you want a clean architecture based on a well understood computational structure, all you have to do is to accept the fact of having a State Machine and Views of state and nothing more.

BTW I willingly omitted the “F” of the (F)SM since this might be subject of discussion here, despite the fact that “infinite” is really huge in the world of computation.

Hardware Neural Networks: Future Of Computing?

Mon, 18 Jun 2012 10:46:05 +0200

Hardware Neural Networks: Future Of Computing?

Making a singular matrix non-singular

Thu, 14 Jun 2012 16:33:17 +0200

Making a singular matrix non-singular

Handle Huge Data With C++ and STXXL

Thu, 12 Jan 2012 10:24:00 +0100

If you ever have to deal with really huge data that does not fit in RAM anymore, but you still need a consistent interface and efficient handling - and you have not the time to write it yourself, as usual - have a look at STXXL. It has some downsides, but after all it works well and efficiently. For me it was the fastest way - both in runtime and implementation time - to deal with data in the TB range.

You may find it here: http://stxxl.sourceforge.net

Clustering RabbitMQ

Fri, 25 Nov 2011 10:41:00 +0100

Clustering RabbitMQ is very easy - if you know how. Unfortunately, the documentation on this topic is good but not good enough (cf. RabbitMQ Clustering). If you try to do it, you may get lost on the track until you find some insightful posts on the mailing list. This is why I summarize here how I got it to work.

Say, you want to create a cluster having two disc nodes and two ram nodes. If you do this on at least two machines, each having a disc and a ram node you achieve good fault tolerance and good scalability both with one setup. Your clients may connect to the ram nodes only or these are balanced by an additional load balancer.

But, how do I make a node a disc node and another node a ram node?

There’s no such command like “rabbitmqctl mkdisc” and there is no related configuration option. On one hand, this is a little counter intuitive, on the other hand this adds a lot of flexibility since you may alter the roles of nodes and restructure your cluster on the fly whenever necessary.

The rules are assigned by the way you call the “rabbitmqctl cluster” command. In our scenario, we have multiple nodes on the same host, so we need to wrap the calls to “rabbitmqctl” into shellscripts setting some environment variables (cf. RabbitMQ Configuration). If this has been done, you ensure all nodes of the cluster are running. Afterwards you execute a sequence of “stop_app”, “reset”, “cluster”, “start_app” commands for all nodes. If it comes to the “cluster” command, you add a space separated list of all disc nodes you want to create to the “cluster” command executed for each node. My mnemonic for this is that you copy the current node to all disc nodes. The whole sequence may look like this, with “rbctl.*” being your wrapper scripts:

host-of-disc1$ rbctl.disc1 stop_app
host-of-disc1$ rbctl.disc1 reset
host-of-dics1$ rbctl.disc1 cluster disc1@host-of-disc1 disc2@host-of-disc2
host-of-disc1$ rbctl.disc1 start_app

host-of-ram1$ rbctl.ram1 stop_app
host-of-ram1$ rbctl.ram1 reset
host-of-ram1$ rbctl.ram1 cluster disc1@host-of-disc1 disc2@host-of-disc2
host-of-ram1$ rbctl.ram1 start_app

host-of-ram2$ rbctl.ram1 stop_app
host-of-ram2$ rbctl.ram1 reset
host-of-ram2$ rbctl.ram1 cluster disc1@host-of-disc1 disc2@host-of-disc2
host-of-ram2$ rbctl.ram1 start_app

host-of-disc2$ rbctl.disc2 stop_app
host-of-disc2$ rbctl.disc2 reset
host-of-disc2$ rbctl.disc2 cluster disc1@host-of-disc1 disc2@host-of-disc2
host-of-disc2$ rbctl.disc2 start_app

If you have to add users, vhost and permissions, you better do it at the end of this procedure, otherwise the “reset” will delete all of this information. Also, if you want to change the cluster setup later, you should be careful with “reset”, omitting it for one disc node at least.

Another weak point with the whole clustering stuff is the location of the “.erlang.cookie” file. This file is essential for clustering and must have the same content for all nodes in the cluster. Documentation says RabbitMQ looks at “/var/lib/rabbitmq/.erlang.cookie” but I found this not always true. Supposed RABBIT_HOME points to the directory where the rabbit distribution is located, I copied the file to “$RABBIT_HOME/../.erlang.cookie” and RabbitMQ used this one. I’m not quite sure if this is a general rule.

Technical Quality is an Insurance Policy

Wed, 23 Nov 2011 09:39:29 +0100

Technical Quality is an Insurance Policy

Analysis of Facebook Graph

Tue, 22 Nov 2011 10:03:15 +0100

Very interesting paper: The Anatomy of the Facebook Social Graph

Reading this very interesting analysis of the structure of the Facebook graph I come to the conclusion that the main features qualitatively may be common to all graphs of social networks. For example, I’ve seen another social graph one order of magnitude smaller having very similar features. It would be really interesting to compare such graphs to other graphs, especially directed social graphs like Twitter or Google+ in terms of different measures. It would also be interesting to compare these graphs to offline social networks. Another interesting question could be to analyze the evolution of such graphs over time, for example using methods of the theory of random graphs.

Velocity is Killing Agility!

Thu, 17 Nov 2011 09:58:02 +0100

Velocity is Killing Agility!

If I had more time I would have written less code

Thu, 17 Nov 2011 09:56:48 +0100

If I had more time I would have written less code

I Regret

Thu, 17 Nov 2011 09:54:34 +0100

I regret. Before I learned to think and to act functional, I proclaimed X=X+1 for years. Have mercy!

Why Functional: One Reason

Thu, 17 Nov 2011 09:53:22 +0100

Why functional is better? So many reasons, one comes here: Programming languages are not made for computers, they are for humans to formulate generic solutions computers can solve faster in the details. The imperative style formulates the steps the computer has to follow, the functional style formulates the generic solution as such. Imperative style wants you to act like a machine, functional style allows you to think like a human.

Name Dropping Consultants

Thu, 17 Nov 2011 09:52:11 +0100

I don’t like name dropping consultants. Name dropping transports no knowledge at all, just obfuscates what is needed to find a decision. A little better is concept dropping, but only a little. What I expect of a consultant is, that she or he asks a lot of questions about my problem and afterwards shows two or three possible solutions for that particular problem, combined with an estimation of the costs.

The Measure Of Agile Is Quality

Thu, 17 Nov 2011 09:51:10 +0100

In agile, I guess, it’s all about quality, not quantity. Quality of code, quality of user experience, quality of customer satisfaction. It’s about maintaining quality in a world of change. It’s not at all about being as fast as possible, but as good as possible - while everything around us is changing constantly. It’s about being smart, not being “schneller”.

Remark: “schneller” is German for “faster, now”.

Tumblr Sucks When Posting Code

Fri, 11 Nov 2011 13:58:45 +0100

Tumblr really sucks when you try to post code snippets. You can edit your posts using HTML but when you save or when you edit again in fancy mode and save, a lot of extra HTML is added to your posts which often makes the code snippets unreadable. I reported this to be a bug a long time ago - a year or so - but it is still not fixed. Solution would be either to add a “pre” or “code” formatting to the HTML editor or to parse the input with some more intelligence.

Publish To A Local Maven Repository With SBT

Fri, 11 Nov 2011 13:47:00 +0100

The Simple Build Tool SBT to build Scala projects is simple most of the time, but if you need something special it can become hard to configure. Documentation is quite well, but not very detailed. More often than not I found me googling some receipts and follow some iterations of trial and error before I finally succeeded. Since I use Scala in real life commercial projects I’m oft confronted with scepticism about the ability to integrate with Java, Maven, Hudson, you name it. Then, I have no choice but to make it work for the environment. Changing the environment is not an option, not in the current phase when Scala is still not widely in use in the industry.

Here is a working receipt for XSBT (tested with 0.11.0) to “publish-local” to a local Maven repository at “~/.m2” instead to the usual local Ivy repository at “~/.ivy2”. You should add the following expressions to your “build.sbt”.

// publish to local ~/.m2
publishMavenStyle := true

otherResolvers := Seq(Resolver.file("dotM2", 
                                    file(Path.userHome + "/.m2/repository")))
publishLocalConfiguration <<= 
   (packagedArtifacts, deliverLocal, ivyLoggingLevel) map 
   {
     (arts, _, level) => new PublishConfiguration(None, 
                                                  "dotM2", 
                                                  arts, 
                                                  List[String](), 
                                                  level) 
   }

PHP 5.4 Traits and the Diamond Problem

Thu, 28 Jul 2011 13:35:00 +0200

PHP will introduce traits in 5.4. While testing this feature with 5.4.0alpha2 I was wondering how it deals with the well known diamond problem.

trait T1 {
    public function f1() { echo __METHOD__ . PHP_EOL; }
}

trait T2 {
    use T1;
}

trait T3 {
    use T1;
}

class Test {
    use T2, T3;
}

$t = new Test;
$t->f1();

This will output:

PHP Fatal error: Trait method f1 has not been applied, because there are collisions with other trait methods on Test in diamond.php on line 24

To make it work you shall use “insteadof”:

class Test {
    use T2, T3 {
        T2::f1 insteadof T3;
    }
}

Now you get this expected output:

T1::f1

That’s fine so far. Anyway, it burdens the programmer with the decision which path through the inheritance hierarchy to use, a hierarchy that’s no longer a tree but a graph with cycles. Hopefully, he’s not doomed with a poor documented library. In a non-trivial project this is not always the case. Here you may soon find yourself lost in a complex inheritance hierarchy which is even more complex if multiple inheritance is possible.

Though knowing it well from Scala I’m not a big fan of traits. They may make your code more expressive but they will make it harder to maintain. Interfaces seem to be less error-prone. In OO design, a deep inheritance hierarchy is usually considered bad design. A wide fan of traits should be as well. And the diamond is a gift from hell.

PHP Design Bug Still Active in 5.4 Alpha - And Worse

Thu, 28 Jul 2011 12:09:00 +0200

Today I tested my favorite PHP bug by design in current 5.4.0alpha2. It still works, and that’s the good news.

php > echo phpversion() . “\n”;
5.4.0alpha2
php > $foo = “bar”;
php > echo $foo[0] . “\n”;
b
php > echo $foo[“ouch”] . “\n”;
b
php > $bar = array(1,2,3);
php > echo $bar[0] . “\n”;
1
php > echo $bar[“ouch”];
PHP Notice: Undefined index: ouch in php shell code on line 1

Notice: Undefined index: ouch in php shell code on line 1
php >

What you see here is the attempt to access a single character of a string using array syntax with an array key that does not exists. Doing this bad thing with a real array at least gives you a notice. Doing it with a string is assumed fine by the interpreter. The magic behind all this is that the string is somehow treated like an array with integer keys and “ouch” is then dynamically casted to an integer and this is 0 in PHP’s strange logic.

But 5.4.0alpha2 knows how to deepen this mess. While in 5.3.3 you may see this:

php > echo $foo[1][0];

Fatal error: Cannot use string offset as an array in php shell code on line 1

Call Stack:
3768.9080 630080 1. {main}() php shell code:0

now you get this:

php > echo $foo[1][0] . “\n”;
a
php > echo $foo[“ouch”][0] . “\n”;
b
php > echo $foo[“ouch”][“ouch”] . “\n”;
b
php >

To name it: you get nonsense, and that’s the bad news.

Years ago my team has reported this to be a bug in PHP but the PHP guys say that’s a feature. Now they improved that feature. Great!

So, if you do not want your PHP software to show undesired behavior you have absolutely no idea where it originates from, always ensure to have an array and not a string when you expect an array: use type hinting excessively and always hint for arrays in function declarations. Never try to catch the “E_RECOVERABLE_ERROR”. Set “error_reporting = E_ALL | E_STRICT”. Always ensure you have a string and not an array if you expect a string (use is_string() since you cannot type hint for strings).

Do yourself a favor and write much more unit tests.

Alternatively: use strongly typed languages.