Jeremy's Weblog

Gentoo Prefix allows you to place a “prefixed environment” wherever you would like. So, if you want to be able to access your prefix on a NFS network it would make sense to put the prefix in /home for example. I don’t have any solid numbers but I can imagine that the IO for the nfs server is pretty high when emerging. I would rather not suffer the penalties of compile on NFS but also I WOULD like to access PORTAGE_TMPDIR from any host. For the longest time, I was trying to think of a solution for this, that is..to not compile on a NFS share but also be able to see/access PORTAGE_TMPDIR no matter what host I am on in the network. This is a tricky situation that can be solved by setting PORTAGE_TMPDIR to another NFS mount which just happens to reside on the local machine!

I like it!!

My solution is to symlink $EPREFIX/var/tmp to the other NFS mount on the localhost. In my case, /net/$(hostname)/public/tmp.

UPDATE: The above ‘solution’ isn’t that great. It resulted in a total time increase of about 12%. However, making the symlink point to the local hard drive resulted in a total time decrease of about 31%. (on emerge -e system) Therefore, my latest recommendation is to create the symlink as described originally and override it with PORTAGE_TMPDIR set to the local path.

Yesterday, zmedico wrote about building multiple packages in parallel with Portage-2.2_rc2. In Gentoo Prefix, we had a sneak peak to this feature, so I have had some time to play with it on my dual-quad core box. Some timing results that you may like:

emerge -e system (excluding sys-devel/gcc)

As a baseline:

With --jobs=1 and MAKEOPTS=16, load-average=9:
real 77m54.290s
user 41m46.086s
sys 29m14.598s

Because I was skeptical of what --jobs could really do, I decided to start with small number of parallel jobs:

With --jobs=3, MAKEOPTS=16, load-average=9:
real 61m30.181s
user 42m23.398s
sys 32m32.009s

While that was running, I noticed a very significant amount of time where my cores were idle, thanks to the handy little xfce-extra/xfce4-cpugraph widget. So, I turned --jobs up again:

With --jobs=5, MAKEOPTS=16, load-average=9:
real 58m5.388s
user 42m35.721s
sys 34m46.950s

Meh, not much improvement there. Surprising, but I suspect that I may be reaching the limits of the parallelization (dependencies, etc).

With --jobs=10, MAKEOPTS=16, load-average=9:
real 58m9.824s
user 42m43.525s
sys 37m57.234s

(And actually, a quick visual scan showed load averages staying below 4. Only a few times did I see the average above 8 )

Relying solely on load-average to keep my system usable:

With --jobs=40, MAKEOPTS=40 load-average=15:
real 58m45.106s
user 43m15.129s
sys 40m47.949s

The highest load average I visually saw was 23. But my load average was mostly always greater than 4, so this means that my procs are obviously getting used more but I must have hit another bottleneck.

NOTES:
-emerge -pe system was preformed before each time trial to assume the depgraph was in cache.
-84 packages total
-no ccache/distcc running

Conclusion: 20 minutes? ~30% speedup. Wow. Good! Quite significant even. Assuming you have cores/procs to spare, go ahead and crank up those --jobs. It is nice to be able to make the ./configure step not be the bottleneck anymore. I will keep testing and see if I can get the time down even farther (although, unlikely based on the last time trial).

Test requests? Please leave a comment.