Jeremy's Weblog

It should be no secret that this site and my other co-location needs are hosted at Linode.com, for which I am a happy customer running Gentoo Linux.

The reason for this post is that after an announcement today. All I had to do was reboot and then I received a 42% RAM increase. Yay.

Thanks Linode, you are exceeding your competition!
(Shameless plug for my referral code )

For some reason, Linode.com (my review) sets up their hosts to use dhcpd to grab the static IPv4 address on boot. This is in contrast to Host Virtual which uses the “Gentoo-way” to set static addresses. Now, there isn’t anything exactly wrong with using dhcpd on hosts with static addresses, actually, it may be simpler (and this is probably why they did it). However, I don’t like it for a few reasons, booting takes longer as it probes for IPs and it uses extra space for dhcpd binary on a low resource host – this includes extra time for updating. I know these are minor issues, but they bother me, at least. So, let’s take the easy way and assign eth0 the IP is should have:

Snippet from: /etc/conf.d/net
config_eth0=(
    "69.164.197.24 netmask 255.255.255.0 broadcast 69.164.197.255"
    )
routes_eth0=(
    "default via 69.164.197.1"
    )

So, that makes sense for all the right reasons and there is not much more to say. Let’s shift the attention to IPv6. Linode doesn’t offer IPv6 by default unlike their competition. To be honest, I don’t need IPv6, but it is something fun to play with and I have been learning something. It turns out that my tunnel from HE.net (free) is actually lower latency to some parts of the world than my IPv4 route and almost always less hops. Using the great examples from Robin (robbat2), I was able to put my IPv6 tunnel in /etc/conf.d/net too, so that is it created on reboot as tun0. Makes sense to do, right?

Snippet from: /etc/conf.d/net
HE="2001:0470" # 2001:470::/32 is the HE.net allocation
v6net64="${HE}:1f0f:2a0" # your initial /64 allocation from HE.net

# HE.net tunnel configuration
link_tun0="eth0" # tunnel IFACE (internet-facing iface, eg ppp0/eth0)
# tunnel IPv4 endpoint, remote, HE.net tells you this
iptunnel_tun0_remote="216.218.224.42"
# tunnel IPv4 endpoint, local
# this is the address of IFACE ${link_tun0}
iptunnel_tun0_local="69.164.197.24"

iptunnel_tun0="mode sit remote ${iptunnel_tun0_remote} local
${iptunnel_tun0_local} ttl 255 dev ${link_tun0}"
mtu_tun0=1280
config_tun0="${v6net64}::2/64" # /126
routes_tun0="default via ${v6net64}::1"

This is not exactly perfect, because I am using my tunnel’s /64 as my IPv6 address. The purists might say something about this practice, I respect that, but don’t really mind for my personal use. Of course, if you didn’t have a tunnel and instead had native IPv6, it would look a lot simpler because you just add the IP and route to the interface it is on, probably eth0.

My linode is now my personal DNS resolver. I have officially ditched the ISP nameservers from this point forward now that I found unbound. Unbound is a lightweight, recursive resolver that is perfect for your LAN, co-located host, or even a single host.

For your single host, emerge unbound, start the service, add 127.0.0.1 to the first nameserver in /etc/resolv.conf. Unbound is setup (by default) to accept connections from localhost and refuse anything else. If you are using dhcp at home (likely) then also emerge openresolv and uncomment name_servers=127.0.0.1 in /etc/resolvconf.conf, openresolv then “intercepts” dhcpcd when it tries to write to /etc/resolv.conf and adds 127.0.0.1 as your first nameserver For your LAN, just configure your router to look to the host that you setup unbound on, with additional configuration.

Finally, you can also have unbound run on your co-located host. Just edit /etc/unbound/unbound.conf to a) listen on an outside interface and b) allow your other host to query it. This will be left as an exercise for the reader, it is easy to figure out.

Lastly, a shout-out to Linux Gazette for an excellent write-up on GoogleDNS (and why you should use something like unbound) and DNS/LAN metaphors. Suggested reading if you feel out of your league with DNS internals, like me.

A quote from the above linked article: “Why outsource to anyone, when you can do a better job locally, at basically no cost in effort?“ and really, that is the truth. Have fun.

If you are running PHP on a limited-resource box, like a VPS then you may have seen your PHP pages randomly hang. I was able to trace this issue down because the PHP pages were hung up and the normal html pages were still being served. The problem was ‘solved’ when I restarted the web server. Some research later, and talking to Thilo (bangert), I found out about PHP_FCGI_MAX_REQUESTS. This is an environment variable that PHP respects, it basically tells how many requests to serve before respawning fcgi. In my case, 500 seemed like a good number after testing. Your mileage may vary, but it is worth a try if you have those symptoms.

%% cat /etc/lighttpd/mod_fastcgi.conf
server.modules += ("mod_fastcgi")
fastcgi.server = ( ".php" =>
    ( "localhost" =>
        (
            "socket"   => "/var/run/lighttpd/lighttpd-fastcgi-php-" + PID + ".socket",
            "bin-path" => "/usr/bin/php-cgi",
            "max-procs" => 2, # default 4
            "bin-environment" => (
                "PHP_FCGI_CHILDREN" => "2", # default 1
                "PHP_FCGI_MAX_REQUESTS" => "500" #default 1000
            )
        )
    )
)

About a month ago, I installed Gentoo on the new-to-me Acer Aspire1. Installation went like anything else, it is just a normal x86 host after all. I don’t have everything on it working, because I don’t care. If you are looking for additional resources on getting the extras working, you may want to look here or here.

The exciting part, that I got working and am ready to announce publicly, is my new atom-x86 binpkg repo. What makes this repo different than the binpkgs located on tinderbox.dev.gentoo.org/default-linux is that this repo has CFLAGS specific to the Intel Atom processor. I identified the compiler flags by using the following gcc command: gcc -Q --help=target -march=native and set the following -march=prescott -mtune=generic -msahf. On my linode (review) host, I have a chroot that builds all new packages in my world file once a day which comes from the aspire1. In this manor, I am able to always have binary packages available to me whenever I update my aspire1. Now, I have all the benefits of a source distro and the speed of a binary distro.

If you would like to use this repo, set PORTAGE_BINHOST in /etc/make.conf and add ‘getbinpkg’ to FEATURES (or use the emerge options directly). Be advised, that thought this works for me, I make no guarantees for you.

PORTAGE_BINHOST="http://tinderbox.jolexa.net/atom-x86/"
FEATURES="${FEATURES} getbinpkg"

I also have an html view of the packages available.

I had this genius idea about using sshfs with rtorrent. I thought that this use case would fit best in situations where you have good bandwidth but not much diskspace, such as my linode VPS (review). So, I’ll attempt to share my findings in this regard.

If you are not familiar with rtorrent. You just need to know that it is a powerful, lightweight bittorrent client. It has a “watch” feature that watches a directory for new torrents, and obviously it can put downloaded files in a specified location. I tried both of these with sshfs.

First, I was having trouble with rtorrent just ‘freezing’ up when I put a torrent file in the sshfs accessible watch dir. I didn’t quite know what was wrong here. Research led me to rtorrent bug 322 and that sshfs did not support filesystems without mmap properly. Darn. More research led me to a recent kernel commit that looked promising. Low and behold, reboot my host with 2.6.31.x kernel and rtorrent works with sshfs watch and destination directory. Yay.

Well, not so fast…

The performance is quite poor with the destination directory on sshfs. This is to be expected because now your download speed for torrents is limited to the download speed of your final destination. But, rtorrent was only giving me a sustained speed of 1/4 of that demonstrated with a simple file copy to the destination. I speculate that this is from the rtorrent overhead or maybe fragmenting? Not sure exactly and I don’t care. My solution to this was to use the rtorrent “move on finished” feature that downloads the file to local disk and then moves it to sshfs destination after it is finished. Amazingly, this works quite well.

My testing scenario was the following:
-79MB Gentoo 2008.0 install cd torrent. With the complete sshfs solution, it took ~6 minutes to download (to the sshfs destination) and then 5 minutes to check the hash. So, roundtrip of 11 minutes from start download to seeding. With the on_finished solution, it took 1 minute to download (to local disk) and 1 minute to check the hash and move to the sshfs destination. For a roundtrip of ~2 minutes from start of download to seeding.

In conclusion, this isn’t the perfect solution because you impose a large bottleneck into the mix and unintended I/O activity on the local disk. However, it works for me and what I am doing. Maybe it will give someone else some ideas in the future.

I recently had a bit of downtime on my linode. If you are wondering what a ‘linode‘ is, check out my review or the website. And a big thank you to the folks that used my referral code when they got setup with linode themselves, you guys rock!

So, about my recent 1/2 day downtime. It was self-inflicted because I wanted to move to a different datacenter. I moved my linode from Newark, NJ to Dallas, TX. It is quite a long story, but it boils down to a problem with my ISP (Comcast). I was only able to pull 100K/s from the Newark datacenter and 2-3M/s from the others. This was unacceptable. I tried to get it escalated past Comcast’s frontline support but they kept asking me questions like “Do you use a router? If so, each computer only gets 1/2 the speed” & “Every computer is different. I’m glad that you can get 3M/s from another host, that is really good” Sigh.

At least Linode’s customer server was helpful and allowed me to work around the ISP. The steps to move a linode are as follows:

1. File a support request. (My initial request was answered in 11 minutes)
2. Shutdown your linode
3. Hit the ‘migrate’ button, after support sets up your migration
4. Wait for the transfer. My total transfer time was ~43 minutes (~6G to transfer). This was pretty fast throughput, in my opinion
5. Meanwhile, update your DNS for your new IP.
6. Since you can queue up a boot job, I just let it go and checked in on it a couple hours later. Magic, it was online.

So, to finish the story off. Linode++, Comcast–. I wish I didn’t need to do something like this, I wish my ISP was…I don’t know…smart?

This is a follow up to my initial linode post.

Linode is a VPS provider. Linode stands for “Linux Node.” They offer relatively up to date initial Gentoo installations, among other distros.

Overall Grade: 9.5/10 (because no one is ever perfect)

Performance:

This is probably the one area that everyone is concerned about from VPS providers. Linode provides linodes on pretty beefy hardware, 4 CPUs/host:

%% cat /proc/cpuinfo |grep "model name"
model name : Intel(R) Xeon(R) CPU L5420 @ 2.50GHz

With the Linode 360 plan, there are 40 guests on each host. That means I am splitting the cpu time with 39 others evenly and I have the full potential of CPU time if no one else is using the CPU cycles. I am a nice citizen of the host and only have my MAKEOPTS set to -j2. With that I get times like this:

%% genlop -t gcc
Mon Apr 6 04:54:53 2009 >>> sys-devel/gcc-4.3.2-r3
merge time: 24 minutes and 9 seconds.
%% genlop -t www-servers/lighttpd
Fri Jan 2 19:48:06 2009 >>> www-servers/lighttpd-1.4.20
merge time: 1 minute and 38 seconds.

The above times compare very nicely to my personal hardware that I have, so I cannot complain about CPU contention.

I do notice some I/O contention during “peak” hours. This will result in some slower compile times for the short packages. Since the 360 only offers 360MB of RAM, I cannot leverage that either. This is not a large concern on my part and I am sure it is alot better than other companies that may oversell their hardware.

%% sudo hdparm -Tt /dev/xvda
/dev/xvda:
Timing cached reads: 7350 MB in 1.99 seconds = 3685.77 MB/sec
Timing buffered disk reads: 36 MB in 3.12 seconds = 11.54 MB/sec

These timings are not great, I agree.

Customer Service / Service Requests:

I have not had a reason to submit a hardware service request. So, my experience in this area is limited to two administrative issues that I needed to take care of with regards to billing changes. Linode gets a 10/10 in this area. The customer service staff member (Tom Asaro) resolved my requests within hours at very odd times. I normally do most of my hobby work during the evening, I submitted these requests around midnight (GMT -5) and they were resolved by the time I woke up. Keep in mind, that my requests were not even close to top priority – I would have been happy if they were resolved within a week.

Regarding hardware issues, the Linode staff members are very responsive to any DDoS attacks, host crashes, etc. The forums activity by staff members is top notch, always keeping us informed. You can find staff members in the irc channel very frequently, even answering my stupid questions.

Hosted Services:

My Linode host runs only 3 domains (incl this blog) and email. I share it with 2 other friends, we all host pet projects on it. (As an aside, splitting it is a very cheap way to get a VPS)

%% rc-status
Runlevel: default
dovecot                                                            [ started  ]
lighttpd                                                           [ started  ]
local                                                              [ started  ]
mysql                                                              [ started  ]
net.eth0                                                           [ started  ]
netmount                                                           [ started  ]
ntpd                                                               [ started  ]
postfix                                                            [ started  ]
rsyncd                                                             [ started  ]
saslauthd                                                          [ started  ]
sshd                                                               [ started  ]
syslog-ng                                                          [ started  ]
uptimed                                                            [ started  ]
vixie-cron                                                         [ started  ]
xinetd                                                             [ started  ]

UI / Admin Interface:

The administrative interface is a pleasure to use. I can see my bandwidth usage, cpu usage, and I/O usage very easily. They look like the standard munin graphs – very handy.

DNS Manager: This is a VERY handy tool, especially when you don’t have the patience/time to set up your own DNS solution. The DNS manager is a very nice interface to Linode’s DNS servers. It is esentially one click DNS management. Nothing more to say, it rocks.

The admin interface get a 8.5/10.

Other Thoughts:

• Well, everyone I have talked to has been thrilled with Linode. In my opinion, they have great products and at great prices. I must say I have not been disappointed one bit in the 3 months of experience I have with them.
• My current uptime is 80 days. This is so low because I choose to reboot the host when testing to make sure that all my services were set-up correctly.
• My current uptime is 6 days, because I choose to reboot to take advantage of the free space upgrade.
• 10% off if you prepay for a year, no contracts so you can cancel at any time.
• I did some stress testing, I can invoke the OOM killer with MAKEOPTS="-j5" & --jobs=2 while emerging. I think it happened during glibc emerge. I guess this is to be expected for what I have access to.

Shameless Plug:

If you decide to get a Linode plan after reading this, please use my referral code: http://www.linode.com/?r=b4fa70eb87c890e08baf7b0c7852fb7cecd8963b Thanks!