Jeremy's Weblog

My linode is now my personal DNS resolver. I have officially ditched the ISP nameservers from this point forward now that I found unbound. Unbound is a lightweight, recursive resolver that is perfect for your LAN, co-located host, or even a single host.

For your single host, emerge unbound, start the service, add 127.0.0.1 to the first nameserver in /etc/resolv.conf. Unbound is setup (by default) to accept connections from localhost and refuse anything else. If you are using dhcp at home (likely) then also emerge openresolv and uncomment name_servers=127.0.0.1 in /etc/resolvconf.conf, openresolv then "intercepts" dhcpcd when it tries to write to /etc/resolv.conf and adds 127.0.0.1 as your first nameserver For your LAN, just configure your router to look to the host that you setup unbound on, with additional configuration.

Finally, you can also have unbound run on your co-located host. Just edit /etc/unbound/unbound.conf to a) listen on an outside interface and b) allow your other host to query it. This will be left as an exercise for the reader, it is easy to figure out.

Lastly, a shout-out to Linux Gazette for an excellent write-up on GoogleDNS (and why you should use something like unbound) and DNS/LAN metaphors. Suggested reading if you feel out of your league with DNS internals, like me.

A quote from the above linked article: "Why outsource to anyone, when you can do a better job locally, at basically no cost in effort?" and really, that is the truth. Have fun.

If you are running PHP on a limited-resource box, like a VPS then you may have seen your PHP pages randomly hang. I was able to trace this issue down because the PHP pages were hung up and the normal html pages were still being served. The problem was 'solved' when I restarted the web server. Some research later, and talking to Thilo (bangert), I found out about PHP_FCGI_MAX_REQUESTS. This is an environment variable that PHP respects, it basically tells how many requests to serve before respawning fcgi. In my case, 500 seemed like a good number after testing. Your mileage may vary, but it is worth a try if you have those symptoms.

%% cat /etc/lighttpd/mod_fastcgi.conf
server.modules += ("mod_fastcgi")
fastcgi.server = ( ".php" =>
    ( "localhost" =>
        (
            "socket"   => "/var/run/lighttpd/lighttpd-fastcgi-php-" + PID + ".socket",
            "bin-path" => "/usr/bin/php-cgi",
            "max-procs" => 2, # default 4
            "bin-environment" => (
                "PHP_FCGI_CHILDREN" => "2", # default 1
                "PHP_FCGI_MAX_REQUESTS" => "500" #default 1000
            )
        )
    )
)

About a month ago, I installed Gentoo on the new-to-me Acer Aspire1. Installation went like anything else, it is just a normal x86 host after all. I don't have everything on it working, because I don't care. If you are looking for additional resources on getting the extras working, you may want to look here or here.

The exciting part, that I got working and am ready to announce publicly, is my new atom-x86 binpkg repo. What makes this repo different than the binpkgs located on tinderbox.dev.gentoo.org/default-linux is that this repo has CFLAGS specific to the Intel Atom processor. I identified the compiler flags by using the following gcc command: gcc -Q --help=target -march=native and set the following -march=prescott -mtune=generic -msahf. On my linode (review) host, I have a chroot that builds all new packages in my world file once a day which comes from the aspire1. In this manor, I am able to always have binary packages available to me whenever I update my aspire1. Now, I have all the benefits of a source distro and the speed of a binary distro.

If you would like to use this repo, set PORTAGE_BINHOST in /etc/make.conf and add 'getbinpkg' to FEATURES (or use the emerge options directly). Be advised, that thought this works for me, I make no guarantees for you.

PORTAGE_BINHOST="http://tinderbox.jolexa.net/atom-x86/"
FEATURES="${FEATURES} getbinpkg"

I also have an html view of the packages available.

I had this genius idea about using sshfs with rtorrent. I thought that this use case would fit best in situations where you have good bandwidth but not much diskspace, such as my linode VPS (review). So, I'll attempt to share my findings in this regard.

If you are not familiar with rtorrent. You just need to know that it is a powerful, lightweight bittorrent client. It has a "watch" feature that watches a directory for new torrents, and obviously it can put downloaded files in a specified location. I tried both of these with sshfs.

First, I was having trouble with rtorrent just 'freezing' up when I put a torrent file in the sshfs accessible watch dir. I didn't quite know what was wrong here. Research led me to rtorrent bug 322 and that sshfs did not support filesystems without mmap properly. Darn. More research led me to a recent kernel commit that looked promising. Low and behold, reboot my host with 2.6.31.x kernel and rtorrent works with sshfs watch and destination directory. Yay.

Well, not so fast...

The performance is quite poor with the destination directory on sshfs. This is to be expected because now your download speed for torrents is limited to the download speed of your final destination. But, rtorrent was only giving me a sustained speed of 1/4 of that demonstrated with a simple file copy to the destination. I speculate that this is from the rtorrent overhead or maybe fragmenting? Not sure exactly and I don't care. My solution to this was to use the rtorrent "move on finished" feature that downloads the file to local disk and then moves it to sshfs destination after it is finished. Amazingly, this works quite well.

My testing scenario was the following:
-79MB Gentoo 2008.0 install cd torrent. With the complete sshfs solution, it took ~6 minutes to download (to the sshfs destination) and then 5 minutes to check the hash. So, roundtrip of 11 minutes from start download to seeding. With the on_finished solution, it took 1 minute to download (to local disk) and 1 minute to check the hash and move to the sshfs destination. For a roundtrip of ~2 minutes from start of download to seeding.

In conclusion, this isn't the perfect solution because you impose a large bottleneck into the mix and unintended I/O activity on the local disk. However, it works for me and what I am doing. Maybe it will give someone else some ideas in the future.

I recently had a bit of downtime on my linode. If you are wondering what a 'linode' is, check out my review or the website. And a big thank you to the folks that used my referral code when they got setup with linode themselves, you guys rock!

So, about my recent 1/2 day downtime. It was self-inflicted because I wanted to move to a different datacenter. I moved my linode from Newark, NJ to Dallas, TX. It is quite a long story, but it boils down to a problem with my ISP (Comcast). I was only able to pull 100K/s from the Newark datacenter and 2-3M/s from the others. This was unacceptable. I tried to get it escalated past Comcast's frontline support but they kept asking me questions like "Do you use a router? If so, each computer only gets 1/2 the speed" & "Every computer is different. I'm glad that you can get 3M/s from another host, that is really good" Sigh.

At least Linode's customer server was helpful and allowed me to work around the ISP. The steps to move a linode are as follows:

1. File a support request. (My initial request was answered in 11 minutes)
2. Shutdown your linode
3. Hit the 'migrate' button, after support sets up your migration
4. Wait for the transfer. My total transfer time was ~43 minutes (~6G to transfer). This was pretty fast throughput, in my opinion
5. Meanwhile, update your DNS for your new IP.
6. Since you can queue up a boot job, I just let it go and checked in on it a couple hours later. Magic, it was online.

So, to finish the story off. Linode++, Comcast--. I wish I didn't need to do something like this, I wish my ISP was...I don't know...smart?

This is a follow up to my initial linode post.

Linode is a VPS provider. Linode stands for "Linux Node." They offer relatively up to date initial Gentoo installations, among other distros.

Overall Grade: 9.5/10 (because no one is ever perfect)

Performance:

This is probably the one area that everyone is concerned about from VPS providers. Linode provides linodes on pretty beefy hardware, 4 CPUs/host:

%% cat /proc/cpuinfo |grep "model name"
model name : Intel(R) Xeon(R) CPU L5420 @ 2.50GHz

With the Linode 360 plan, there are 40 guests on each host. That means I am splitting the cpu time with 39 others evenly and I have the full potential of CPU time if no one else is using the CPU cycles. I am a nice citizen of the host and only have my MAKEOPTS set to -j2. With that I get times like this:

%% genlop -t gcc
Mon Apr 6 04:54:53 2009 >>> sys-devel/gcc-4.3.2-r3
merge time: 24 minutes and 9 seconds.
%% genlop -t www-servers/lighttpd
Fri Jan 2 19:48:06 2009 >>> www-servers/lighttpd-1.4.20
merge time: 1 minute and 38 seconds.

The above times compare very nicely to my personal hardware that I have, so I cannot complain about CPU contention.

I do notice some I/O contention during "peak" hours. This will result in some slower compile times for the short packages. Since the 360 only offers 360MB of RAM, I cannot leverage that either. This is not a large concern on my part and I am sure it is alot better than other companies that may oversell their hardware.

%% sudo hdparm -Tt /dev/xvda
/dev/xvda:
Timing cached reads: 7350 MB in 1.99 seconds = 3685.77 MB/sec
Timing buffered disk reads: 36 MB in 3.12 seconds = 11.54 MB/sec

These timings are not great, I agree.

Customer Service / Service Requests:

I have not had a reason to submit a hardware service request. So, my experience in this area is limited to two administrative issues that I needed to take care of with regards to billing changes. Linode gets a 10/10 in this area. The customer service staff member (Tom Asaro) resolved my requests within hours at very odd times. I normally do most of my hobby work during the evening, I submitted these requests around midnight (GMT -5) and they were resolved by the time I woke up. Keep in mind, that my requests were not even close to top priority - I would have been happy if they were resolved within a week.

Regarding hardware issues, the Linode staff members are very responsive to any DDoS attacks, host crashes, etc. The forums activity by staff members is top notch, always keeping us informed. You can find staff members in the irc channel very frequently, even answering my stupid questions.

Hosted Services:

My Linode host runs only 3 domains (incl this blog) and email. I share it with 2 other friends, we all host pet projects on it. (As an aside, splitting it is a very cheap way to get a VPS)

%% rc-status
Runlevel: default
dovecot                                                            [ started  ]
lighttpd                                                           [ started  ]
local                                                              [ started  ]
mysql                                                              [ started  ]
net.eth0                                                           [ started  ]
netmount                                                           [ started  ]
ntpd                                                               [ started  ]
postfix                                                            [ started  ]
rsyncd                                                             [ started  ]
saslauthd                                                          [ started  ]
sshd                                                               [ started  ]
syslog-ng                                                          [ started  ]
uptimed                                                            [ started  ]
vixie-cron                                                         [ started  ]
xinetd                                                             [ started  ]

UI / Admin Interface:

The administrative interface is a pleasure to use. I can see my bandwidth usage, cpu usage, and I/O usage very easily. They look like the standard munin graphs - very handy.

DNS Manager: This is a VERY handy tool, especially when you don't have the patience/time to set up your own DNS solution. The DNS manager is a very nice interface to Linode's DNS servers. It is esentially one click DNS management. Nothing more to say, it rocks.

The admin interface get a 8.5/10.

Other Thoughts:

• Well, everyone I have talked to has been thrilled with Linode. In my opinion, they have great products and at great prices. I must say I have not been disappointed one bit in the 3 months of experience I have with them.
• My current uptime is 80 days. This is so low because I choose to reboot the host when testing to make sure that all my services were set-up correctly.
• My current uptime is 6 days, because I choose to reboot to take advantage of the free space upgrade.
• 10% off if you prepay for a year, no contracts so you can cancel at any time.
• I did some stress testing, I can invoke the OOM killer with MAKEOPTS="-j5" & --jobs=2 while emerging. I think it happened during glibc emerge. I guess this is to be expected for what I have access to.

Shameless Plug:

If you decide to get a Linode plan after reading this, please use my referral code: http://www.linode.com/?r=b4fa70eb87c890e08baf7b0c7852fb7cecd8963b Thanks!

If you have rss feeds to my old WP hosted blog, you may want to check them now and update if needed. That blog is closed for comments and I'll rid google of it after it fully picks up this site. (Over 20,000 hits over there, thanks!)

My new home is located on a Gentoo VPS from a provider called Linode. I have nothing but good things to say about Linode right now. Service, response time, user community, admins in irc, price, etc. I have been working on setting up a webserver (this blog), email server - which caused me much pain overall, and a few other random things. For the most part, my new host is fully setup now and relatively hands free for major services.

Expect more [written] news from me soon here now that I got this project done. Oh, by the way, Linode has a referral program. If you appreciate some of the work I do for Gentoo, you should use my referral code to make a purchase there. Here is the link: http://www.linode.com/?r=b4fa70eb87c890e08baf7b0c7852fb7cecd8963b