A question which I like to use when interviewing C++ programmers: what is the range of a 32-bit integer?

I don’t use this question very often anymore, or at least I don’t let it influence my decision very much (which is why I don’t mind spilling the beans here), because the correlation with other technical skills turns out to be not as strong as I thought it would be. Still, there are some interesting patterns in the kind of people who know the answer versus those who do not.

Among the people who do not know the answer, some of them react quite affronted that they would even be expected to. What is the point, they will ask, in having memorized some little piece of trivia which they could Google up in a few seconds? Isn’t “knowing where to find it” a much more useful skill? Ask me about architecture! Ask me about design patterns and data structures! All of these objections have some validity, and indeed we will certainly ask about those other things during the interview. Still, I believe that it is perfectly reasonable to expect a good developer to know the answer to the above question by heart.

There are two reasons why I believe that. The first is simple: if you don’t know the range of the data types you’re working with, how can you write reliable code? “I will look it up when I need it” is a common excuse, but that doesn’t fly with me. It would be a perfectly valid answer if I had asked you to recite from memory the arguments to some obscure library function, but we’re talking about the integers here! The most commonly used datatype in any programming language. Are you seriously telling me that, every single time you write

for (int i = 0; i < n; ++i)

you look up in the compiler documentation whether the range of i is large enough that you can safely assume n to fall within it? And then you promptly forget about it so that you need to look it up again the next time? So that, when you arrive in my interviewing room with a CV that says you have five years of C++ experience, you have looked up that particular bit of trivia many thousands of times by now, and you still don’t remember it? What’s more likely is that you have never looked it up, which means that you are basically, as Joel Spolsky would call it, programming by superstition.

The other reason why you need to know this stuff is that you will be a more effective programmer if you are able to think of what you are doing in terms of the underlying processor instructions — a skill which is useful even when working in very high-level languages, but which is absolutely necessary when working in C or C++.

For example, even a non-programmer may be aware that the reason why the world is rapidly moving to 64-bit processors and operating systems right now, is because the 32-bit ones are unable to use more than 4GB of memory. But as a professional programmer, I would expect you to know why that is. The answer is simple: it’s because 32-bit machines use 32-bit memory addresses, and with those you can only access 2³² = 4,294,967,296 different bytes. If you don’t know that, you are lacking a fairly basic piece of understanding of how computers work.

Now, of course I don’t expect people to know the exact number 4,294,967,296 by heart, but “a little over four billion” (or two billion for a signed integer) would not be too much to ask. If you cannot tell whether it is more or less than a million, as was the case with several ’senior’ candidates I had the pleasure to interview, you’d better hit the ball out of the ballpark on every other part of the interview. Which they did not. (Please note that I’m using the American usage of ‘billion’ to mean a thousand million.)

Another point is that C++ has largely fallen out of favour nowadays for general application programming. So when it is still used, it is mostly to support performance-critical algorithms or very high-volume data processing. In that kind of code, having a good mental picture of the data structures you’re using is really crucial. If you want to have millions of instances of a given object in memory simultaneously, shaving a few bytes off each instance can really matter — which means that you need to know when you can safely use a char or a short instead of just blithely using ints for everything.

So, here are the facts which I expect every C++ programmer, as well as the better C# and Java ones, to know more readily than their mother’s birthday:

• A byte is 8 bits, and can hold 256 different values. Examples: the number of different characters in a non-Unicode text file, the number of different shades of red, green and blue in a true-colour image. Also the maximum filename length on ext3 and NTFS, and an unfortunate limit on path length in many programs.
• A ’short’, on most C/C++ compilers, is 16 bits, which means it ranges from -32,768 to 32,767 when signed, or from 0 to 65,535 when unsigned. A decade or two ago, it was quite common to have the default size of int be 16 bits on PC-based C compilers, and you may still encounter this in the embedded world. On many systems today the wchar type is 16 bits, so that it supports all characters in the UCS-2 subset of Unicode, which contains all of the characters you actually care about. Beware of the difference between UCS-2 and UTF-16, though. Before true-colour displays became the norm, there were video cards which supported 16-bit colour; usually this was actually 15-bits, with 5 bits (32 different shades) each for red, green and blue, although sometimes the remaining bit was used to allow 64 different shades for green, which is the colour which the human eye can perceive most accurately. 16 bits is also the default integer size on some database systems, which can give you some nasty surprises when your e-commerce system has been in production for a few months and a customer submits the 65536th order.
• 24 bits (three bytes) gives you about 16.8 million (16,777,216) different combinations. Among other things, this is the number of different colours on a true-colour display. In November 2006, Slashdot had to do an emergency update of their database, since they used the MySQL ‘mediumint’ type for comment IDs, so the system broke when the 16777216th comment was posted.
• 32-bits is the default integer size for almost any modern programming language — yes, even on 64-bit platforms, which will break any code which assumes that a pointer can be cast into an integer and back. As stated above, a signed integer ranges from minus to plus 2.1 billion, while the unsigned range goes up to almost 4.3 billion. This is a fairly generous range for most problem domains, but sooner or later it will bite you if you’re not aware of it. Among many other things, this is why many programs have trouble with data files larger than 2GB. Also the absolute upper limit on the number of unique IP4 addresses (the actual limit is much lower because of the way the address space is organized).
• A float value uses 24 bits for the mantissa, which translates into 7 decimal digits of precision, which is less than most programmers intuitively expect. A double uses 53 bits, which is plenty for most purposes. Anyway, the most important thing you need to know about working with floating-point numbers is that it is fraught with gotchas and special cases which will trip you up sooner or later. And never, ever use them for monetary values.

I would also expect a serious programmer to know the powers-of-two table at least up to 2¹⁶ = 65,536. You simply run into those numbers so often that you can’t help memorizing them after a while, and it’s surprising how often this knowledge can help you in debugging various kinds of overflow issues.

So what about larger integer sizes? Well, unless you work in cryptography or some other specialized field (in which case I really hope that you didn’t learn anything new from this article), you can safely assume that 64 bits is enough for anything you will ever want to do. But here is a handy little rule-of-thumb I use to convert between bits and decimal digits: 10 bits is 1,024 and 3 digits is 1,000, so to go from number-of-bits to number-of-digits you just divide by 10 and then multiply by 3. (Or for an even rougher approximation, just divide by 3.)

For instance, an unsigned 64-bit address represents roughly 3 * 6.4 = 19.2 decimal digits, so it can safely be used to store numbers well over 1,000,000,000,000,000,000. Likewise, if you want to know how many bits you need to store a 12-digit number, you divide by 3 and multiply by 10, which tells you that you need approximately 40 bits.

(Update: it now turns out that after the escape, Mr. Davidson killed his wife and one of his children, and then committed suicide. This blog post, intended as a light-hearted tongue-in-cheek piece, was written before I learned of this. Kind of throws the whole thing in a different light.. Kids, please don’t become spammers — it really isn’t worth it.)

One obvious observation that can be made here is that, unless he is very certain that he will be able to evade justice for the remainder of his life, this was a pretty stupid thing to do: the punishment for escaping from prison is likely to be a lot harsher than the 21 months in minimum-security he received for his original crime.

Another question that came up in the /. comments is whether 21 months in minimum-security prison is an appropriate punishment for a spammer. Many people made the point that, although obnoxious, spamming is a non-violent crime and should therefore be treated less seriously than, say, murder.

I disagree.

How much human life does a typical murderer take? Of course, some people kill more than one victim, but many murderers probably get caught after the first offense, and anyway most modern societies seem to have decided that taking a single life is already worth the highest punishment which that society is willing to hand out. So if we assume a single victim, and if we furthermore assume that your average murder victim is 25 years old and would have otherwise lived to the age of 75, then our hypothetical murderer has taken away 50 years of another person’s life. Clearly, this deserves harsh punishment.

How does “Eddie” Davidson compare to that? Well, to the best of my knowledge he never killed anybody with his own hands, and it is difficult to imagine a scenario where any single one of his e-mails directly led to another person’s death. Maybe some of the pharmaceutical products he peddled were unsafe, but in that case the appropriate thing to do would be to prosecute him separately for that.

Nonetheless, it is a proven fact that Mr. Davidson sent out hundreds of thousands of unwanted e-mails, over a period of more than four years according to the Wikipedia article. I wouldn’t be surprised if “hundreds of thousands” is actually a low-end estimate; based on the amount and the indiscriminate nature of the spam I find in my own mailbox, I suspect that a prolific spammer sends out millions of e-mails in a single spam run. Of course, many of those addresses will be invalid and a large percentage of the spam will be filtered before reaching the recipient. So let’s assume that for each spam run he did, 200,000 mails ended up in the inboxes of actual humans, where they cost on average about a second of each person’s life to recognize the mail as spam and to delete it. Furthermore, let’s assume that during his career, he was responsible for 10,000 spam runs.

That would mean that, in total, he took away 2,000,000,000 seconds out of those people’s lives without their consent. That is more than 63 years! So although he may not have caused as much disruption in any single person’s life as a murderer would, if we look at the sum total of how much life he has taken away from other people, he caused more damage than if he had gone out and killed one person.

Furthermore, the odds of actually being caught and convicted for spamming are probably a lot lower than for murder. As David Friedman explains in his excellent book Law’s Order, with a low chance of being caught should go a correspondingly higher punishment, because otherwise rational criminals will do the math and decide that the rewards of the crime are worth the risk. Hence, the obvious conclusion is that the punishment of a prolific spammer should be roughly equal to that of a serial killer who took several dozen lives.

(Yes, I am being slightly facetious here.)

As you can see, the ADSL router and anything connected to it through the WiFi is considered untrusted: the real access point to my internal network is the Linux machine.

Among other things, the Speedtouch has the ability to support Voice-Over-IP by attaching an analog phone. Unfortunately, this functionality does not work in combination with the “assign public IP to a machine on the local network” setting. Which is a pity, because behind the router is my Linux server, running a web- and mailserver among other things, and I really want that server to have my public IP address. Partly because having the server NATted could cause problems with mail, in particular, in the sense that when I send out mail to another server, some suspicious spamblocker software may take offense if the address reported in the headers of my outgoing mail does not match my actual IP. But mostly because having a web/mail/FTP/whatever server hidden behind a NAT, just feels wrong.

So, I have two conflicting desires here: I want to use VOIP over a phone connected to the Speedtouch, and I want my server to believe that it is listening directly on my public IP. A little googling confirms that a lot of other people have this same problem, but if anybody has found a solution already, I didn’t see it.

There does exist a way to expose the server to the outside world without breaking VOIP: use the “game and application sharing” feature to forward all TCP and UDP ports except port 5060 (which is the SIP port used by the VOIP service) to the server. But then we are using NAT again, which is not what we want. What I want is: port 5060 is handled by the Speedtouch, all other packets are sent straight to my Linux server, which should receive them on the public IP address of my Internet account. Unfortunately, it seems that there isn’t any way to configure the Speedtouch like that.

How I eventually solved this problem is by adding an iptables rule on my server, which uses DNAT to translate the source address of each packet back into my public IP before the server sees it.

Like this:

• A packet comes in from the outside world on my public IP address (82.95.250.5).
• The router sees it, and if the packet is not aimed at the VOIP port (5060), sends it on to my server, which has a local IP of 10.0.0.1 (assigned to it by the router through DHCP).
• An iptables rule on my server intercepts the packet and changes the source address back to 82.95.250.5. This way, all services running on my server can pretend that they are connected directly to the Net, without any special configuration needed.
• As my server sends a response to the packet, the destination address is changed back to the router’s address (10.0.0.138).
• The router performs a second layer of NAT, translating the destination address of the response packet back into 82.95.250.5 again, before sending it along to its final destination.

So every packet exchanged between my server and the rest of the world, is NATted twice: once by the router, once by the server. Not a particularly elegant solution, but it will have to do until somebody comes along with a better way to bend the Speedtouch to his will (or until I buy a better ADSL router).

Here’s the magic iptables statement:

iptables -A PREROUTING -t nat -i $EXTERNAL -d $FAKEPUBLICIP -j DNAT –to-destination $PUBLICIP

In my case, $EXTERNAL would be eth1 and $FAKEPUBLICIP is 10.0.0.1.

As I said, it’s not a particularly elegant solution. One annoying consequence is the fact that if you try to browse to mwolf.net from the WiFi network, you’ll get an error because the router will get confused about where to send packets for 82.95.250.5. This could be solved with a bit of DNS magic.

But maybe somebody has already found a nice, clean way to configure the Thomson router to do what I want, and I just missed it. Any ideas would be appreciated.

The jump was a birthday present for my father, who has always been very interested in everything related to planes. But of course, when we gave him the gift certificate, it went without saying that me and my brother would jump as well.

And it was COOL! All three of us loved it.

Of course, it was a tandem jump — an experienced parachuteer does all the work, and you’re just along for the trip. Nonetheless, since the experienced guy needs to be on top for obvious reasons, that means that once the door opens, you get to do the honours of swinging your legs over the side and pushing off. After that, there is 30 seconds of free fall, which is really wild — imagine sticking your head out of the window of a car doing 200kmh, and you have a fair idea of what it feels like. The jump starts at a height of 3000m, and in those 30 seconds you cover roughly half that distance. Then there’s a brief shock as the parachute opens — and the next second you’re just hanging there, seemingly not moving at all, and it’s so quiet that the two of you can hold a conversation without raising your voice.

The view is incredible — you’re still high enough to see for miles around, and there’s nothing between you and the scenery. After a while however, you notice that you’re not completely motionless, and in fact the objects on the ground are getting larger quite rapidly. Moments later, the landing point comes into view. The landing was surprisingly soft; if it wasn’t for the inconvenience of being two people strapped together, it would be more like stepping off an escalator than jumping down from a height.

It was definitely exciting, but not quite as scary as I expected it to be. I felt some nerves as the airplane door opened, but then as I looked out, there was never really a feeling of “I’m three kilometers above the ground and if anything goes wrong I will die.” Rather, looking from the plane it seemed as if everything was weightless — as if, even without a parachute, I could just step out of the plane and float around a little. After that, there was too much to do to be scared: step out, grab harness with both hands, push off, hold head backwards, arc body, wait for a tap on the shoulder, move arms into the stabilizing position — and then the ‘chute opens and you’re seemingly weightless again. There’s just no time to be scared.

On the other hand, some of the other people doing tandem jumps on the same day, weren’t quite so stoic about it — some of our fellow jumpers seemed about to pass out beforehand, and the couple who were scheduled after us were still debating whether or not to go through with it, when we left them.

So, would I do it again? Definitely! Although, at 200 euros for four minutes of fun, it’s not something you do every week. But I have already been looking into the options for taking solo jumping lessons..

(Bias alert: my friend Dirk-Jan works for Nokia in Finland as a project manager on the N810, so that made me a little more interested in this gadget than I would otherwise have been.)

With a 4.1″ screen, the N810 is a little bigger than your average PDA or smartphone. From the negative side, that means it’s just too large to comfortably fit in my pants pockets (especially since it still requires a separate phone to make UMTS calls — the device does not have built-in UMTS capability) while not being large or powerful enough to be a laptop replacement. From the positive side, it’s a lot more pleasant to do webbrowsing or e-mail on than a regular PDA (after all, not so long ago many people would have considered an 800-pixels-wide screen perfectly adequate for a laptop or even a desktop) while still fitting easily in a pocket of my jacket. My Sony Vaio SZ is pretty lightweight for a laptop with a full-sized keyboard, but I expect to leave it home more and more often now that I have the N810.

Apart from the size, it’s a really cool-looking toy: sleek, with good “fit and finish” and no more bumps and frillies on the outside than necessary. And of course there’s the slide-out keyboard, the lack of which was what made me decide to pass on the N800. In several reviews, I had read some complaints that it takes a while to get used to the keyboard, because there’s no space between the keys so it’s easy to unintentionally hit multiple keys at once. However, I didn’t have much trouble with that. The trick, in true Zen style, is not to worry about hitting multiple keys: just hit the one you’re aiming for right-on, without consciously avoiding the eight keys around it, and the keyboard will usually register only the one in the middle. On the other hand, if you try to awkwardly avoid hitting multiple keys by touching them with the edge of a finger or with a fingernail, typing will be slow and frustrating.

But of course, the real reason why I went for this device instead of the hundreds of other PDAs on the market, is that it runs Linux — Maemo to be precise. By default, you get a shell, a bunch of standard Unix tools (from Busybox), Perl, and a minimalist version of vi, in addition to the stuff you would normally expect on a device like this, such as a webbrowser, e-mail client and media player. Needless to say, there is already a sizeable community around Maemo, which has ported all kinds of Linux software such as the SSH client and server, Vim, Python, Ruby, MPlayer, lots of games including LXDoom, rdesktop and many others. Except for a couple of toy scripts in Perl and Python, I haven’t tried building any software myself yet; from what I understand, Maemo has its own GUI framework so porting an X11 app may require some code changes, but porting a simple command-line tool should be a matter of just doing a cross-compile to the ARM platform, in many cases.

What’s a bit disappointing is the lack of tooling to synchronize e-mail and calendar entries with Outlook/Exchange. I guess when you specifically go out of your way to get a Linux-based device rather than the much more common Windows Mobile based ones, you don’t really have much standing to complain about that. Nevertheless, since this is one of the most obvious uses of such a device, and most businesses use Exchange, it would have been nice if something were included by default. But of course, there are various open-source options being worked on by third parties. Haven’t tried them yet, though.

Another thing I’m still looking for is the perfect media player for this device. The built-in player would be perfectly adequate for my needs, except for one snag. By default, it does not support Ogg Vorbis audio files. Fortunately, that’s easy enough to add. However, when you do that, suddenly hundreds if not thousands of .ogg files from the Navicore directory (which contains a demo version of the Wayfinder route planning software) are added to the library. There does not seem to be any way to tell the media player to ignore that directory. The alternative UKMP skips the Navicore directory by default, but it doesn’t seem very stable — it has already crashed on me several times. Then there’s Kagu, which is mostly written in Python so it’s very easy to modify the list of directories to be searched. However, while it looks nice, the user interface does not work well for me at all — you apparently need to add all your songs to the playlist individually before you can play anything. The ‘add all’ button only appears when you’re already at the list of songs for a particular album. Hmm, it’s written in Python — how hard could it be to modify that? In the meantime, I’m open to suggestions on either a different media player to use, or on how to solve the problems I’m having with the three ones mentioned.

However, last Thursday the same thing happened again, and this time after a reboot all I got was the message “primary hard disk not found”. So that’s why my server was unreachable for the past couple of days. It’s quite possible that I lost some mail, too — I have a fallback mailserver through my DNS provider, but I’m not sure that they will have retained everything for three days. If you sent me an e-mail recently and have not received a response yet, please send it again.

I decided to drop the idea of using an old laptop as a server, so I went out and told the local computer shop to give me the cheapest hardware they could find — which turned out to be an Asus 2GHz dual-core board, 1GB of memory, two 80GB SATA drives (that’s the main reason why I didn’t just get another laptop drive: software RAID) and an extra network card. Turns out that Ubuntu 6.06LTS does not support the on-board network card, which is rather annoying, so I installed the desktop edition of Feisty and turned it into a server system.

The important stuff seems to work again, but there may be some more minor outages over the coming week, as I’m further tweaking the new system. No more mail should be lost, however.

Powered by ScribeFire.

So yesterday, as happens every couple of months, a spammer somewhere in Pakistan decided to randomly pick the mwolf.net domain as the fake ‘from’ address for his various unsavoury commercial offerings. Which means, of course, that I get a few hundred bounces from well-meaning but naive mailservers, configured by well-meaning but naive admins.

Hello everybody: spammers lie! Their pills don’t work, their stock tips are scams, they won’t deposit several million dollars into your bank account if you just let them use your account number for a couple of days, and their return addresses aren’t valid. If you have determined that a given e-mail is probably spam, then sending anything to the ‘from’ or ‘reply-to’ address is just about the least useful thing you could do. It makes you a part of the problem, not the solution. By sending an automated response to that address, you are allowing the spammer to use your server to effectively spam me. I get plenty of spam myself, but SpamAssassin deals with that pretty well; the bounce messages are a bigger problem.

In a world where e-mail to a non-existent username is a lot more likely to be spam or a virus than an innocent typo, sending reply messages to such mail should be condemned as bad netiquette. Either accept all mail for your domain and swallow the spam silently, or set up your system so that it generates an error message as part of the SMTP exchange. And never, ever send any kind of response to a message which your filter software has already identified as being probable spam.

And then, of course, there are the ‘callback’ systems, which send an automated response to every mail they receive, asking the server to click on a link or in some other way prove that they are a real person. Of course, this also means that they send out one harassing message to an innocent third party for every spam message they receive, thus effectively becoming a spammer themselves.

In other spam news, and related to my previous post: it turns out that some of my own legitimate mail is not being received because it is being identified as spam by over-eager filters. Why? Because my ADSL account, with a server running 24*7 on a fixed IP address, is listed as a dial-up in some blacklists. Now, I can kind of see the logic behind that — after all, blocking dial-up users is probably fairly successful in getting rid of a lot of spam from botnets. But there’s a baby in the bathwater: a lot of technically savvy people like to run their own mailserver, ironically often with spam filtering as an important motive. As I once tried to explain to my previous employer: are these really the people whose job applications you want to block? There are much better ways of spam filtering, which don’t yield so many false positives. Please don’t do it.

And yes, I am aware that I have not posted anything new on this blog for the past half year or so..