I must confess I’m pretty much adept to Debian, but I was forced to be unbiased and start researching for better options for achieving my goal. Also because the environment I was going to port was already running Debian, but the application’s traffic growth was increasing at high speed and it seemed something was going on with that server.

Why not Debian?

Well, I had to choose some distribution where I could have my packages being compiled locally to comply with a few standards, like packages must be compiled specifically for the processor they’re running under, improve at compilation how programs make their calculations and it can be done only knowing exactly where you’re running your program, and more. Unfortunately, Debian maintains their packages pre-compiled, which means that they don’t know where they’re installing into, so they’re compiled with generic options and it can be a performance issue when talking about millions of users connected to your Apache. Let’s say that if the package takes 50KB of memory per request more than a package specifically compiled to the computer set it’s running under, for each 5000 users connected to my application I’ll need more 240MB of RAM. Well, things get bit harder if we multiply this by 10 and so on.

This thing was upsetting me, I didn’t want to simply abandon Debian without doing some research before. I found great distros to fulfill my requirements, such as Gentoo or Slackware… but I was leaving Debian…

Then I was about to really adopt Gentoo to achieve my goal, when I was talking to a friend of mine (which also is a Debian addict) and he said that it was possible to recompile packages locally with not much efforts and close (not that much, actually) to the Gentoo’s portage.

And then I found apt-build. It was pretty good! Despite the fact it took the whole week to recompile the whole distribution, it works pretty fine and I could get pretty close to my objective. Well, it was not that easy to get things exactly the way I wanted, but finally I could see it was possible.

Why Debian?

Indeed, I could really drop Debian and start using Gentoo or Slackware easily. But I had to be persistent to figure out a way to keep using Debian. Why? Well, package management is quite stable, I like the way they manage versions and Debian is a distribution I trust for years. Despite the fact I can get support very easy on the net and if I really need something installed quickly or if it’s a non-regular service I can still using aptitude (or apt-get).

The Result

After much work and efforts, I got an AMI with a high-tuned Debian inside. A clean and base install of Debian GNU/Linux 5.0.3 Lenny. Right now, I just have the 32 bit for m1.small and c1.medium, but soon I will get a 64 bit done.

Features

Kernel, RamDisk, Kernel modules and Filesystem Userspace:

• Uses the Canonical 2.6.28-12-xen-i386 Kernel (aki-21f01148) patched to be used on Debian Lenny.
• Uses the Canonical 2.6.28-12-xen-i386 RAM Disk (ari-3bf01152) patched to be used on Debian Lenny.
• All 2.6.28 kernel modules installed and running.
• Updated with Fuse 2.8.1.

Improved Performance:

• All installed Debian Packages _recompiled_ with GCC 4.3.2 and libc6 (xen and i686) for improved performance using Optimization level 3 (read more about GCC optmization options at http://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html and http://gcc.gnu.org/onlinedocs/gcc/i386-and-x86_002d64-Options.html) and:
• march=i686 (set PentiumPro instructions. Amazon EC2 – currently, Intel Xen 2.6GHz single core emulation – processor and Kernel in use are 686 compliant, but Debian packages are compiled as i386 for generic usage).
• mfpmath=sse and msse for scalar floating point instructions to increase applications’ speed and avoid the default 387 instability problems.
• ffast-math to increase processor speed with math optimizations (Note that I used this flag only on IEEE and ISO math operation compliant applications. Don’t use it if you’re not sure if your application is compliant to IEEE and ISO specifications for math operations).
• Still, I didn’t skip the debugging output (-g), so if you realize that any package is performing weirdly, please, debug it and let me know commenting in this post. I will fix it and raise a new AMI with the fix.
• Kernel modules recompiled from the source (2.6.28.10) retrieved from www.kernel.org, optimized to the Amazon’s Xen environment. All security patches applied.

Packages:

• Packages up-to-date.
• OpenBSD SSH and Linux Screen for remote administration.
• OpenSSL and CA Certificates for SSL and TLS support.
• Amazon EC2 first boot script.
• Amazon EC2 Tools 1.3.
• Unused packages fully purged from system.

Security:

• Shadow passwords enabled.
• root password locked. Remote access is made only using the Keypair used to launch the instance.
• ssh PasswordAuthentication disabled.

Conclusion

I have proven to myself I could really have Debian running for an HPC environment smoothly, what  makes me even prouder of this distro.

If you want to check the result, the AMI ID is ami-2d759644. If you do so, I’d be glad to hear from you.