Using sshfs with rtorrent
I had this genius idea about using sshfs with rtorrent. I thought that this use case would fit best in situations where you have good bandwidth but not much diskspace, such as my linode VPS (review). So, I'll attempt to share my findings in this regard.
If you are not familiar with rtorrent. You just need to know that it is a powerful, lightweight bittorrent client. It has a "watch" feature that watches a directory for new torrents, and obviously it can put downloaded files in a specified location. I tried both of these with sshfs.
First, I was having trouble with rtorrent just 'freezing' up when I put a torrent file in the sshfs accessible watch dir. I didn't quite know what was wrong here. Research led me to rtorrent bug 322 and that sshfs did not support filesystems without mmap properly. Darn. More research led me to a recent kernel commit that looked promising. Low and behold, reboot my host with 2.6.31.x kernel and rtorrent works with sshfs watch and destination directory. Yay.
Well, not so fast...
The performance is quite poor with the destination directory on sshfs. This is to be expected because now your download speed for torrents is limited to the download speed of your final destination. But, rtorrent was only giving me a sustained speed of 1/4 of that demonstrated with a simple file copy to the destination. I speculate that this is from the rtorrent overhead or maybe fragmenting? Not sure exactly and I don't care. My solution to this was to use the rtorrent "move on finished" feature that downloads the file to local disk and then moves it to sshfs destination after it is finished. Amazingly, this works quite well.
My testing scenario was the following:
-79MB Gentoo 2008.0 install cd torrent. With the complete sshfs solution, it took ~6 minutes to download (to the sshfs destination) and then 5 minutes to check the hash. So, roundtrip of 11 minutes from start download to seeding. With the on_finished solution, it took 1 minute to download (to local disk) and 1 minute to check the hash and move to the sshfs destination. For a roundtrip of ~2 minutes from start of download to seeding.
In conclusion, this isn't the perfect solution because you impose a large bottleneck into the mix and unintended I/O activity on the local disk. However, it works for me and what I am doing. Maybe it will give someone else some ideas in the future.
I was thinking about a similar setup for my linode.
I don't need it for torrents anyway, but one of the sites I'm running has a page where users publish and share theyr works, mostly in pdf format, and space is becoming an issue.
Anyway I'd prefer to create a vpn with my home server and mount a folder via NFS.
What would be the advantages of using sshfs?
A VPN would be a valid solution too. I was going for ease of maintenance using an existing service (sshd) that I already has set up. I have wanted to setup a VPN for myself for awhile now..
Buy a new HD, it's cheap
The 1/4 speed comes from the fact that when you download a torrent, you already consume bandwidth[1], and also seed[2] at the same time.
[1] takes away available downstream bandwidth needed by [2] to fetch the parts that need seeding. In other words, what would otherwise be consuming upstream bandwidth only, now needs downstream too since it needs to get the data first before seeding them.
heh, "buy a new HD" only works if the server is not co-located. In other words, I can't just "buy a new HD" =P
I'm aware that bittorrent uploads while it is downloaded. However, for purposes of the test there was no uploading going on because no one else was downloading the torrent as you can see here: http://torrents.gentoo.org/index.php?order=seeders&sort=descending