After almost a year, I finally published an update to my Writing udev rules document. I’m still amazed at the amount of email this brings me – please remind me not to leave it so long next time, going through so much built-up email was quite an effort! Now I just need to reply to it all.
If you are using udev out-of-the-box and don’t know about its customization abilities, you should give this document a quick read.
Since I initially raised the topic of using an ebuild to build from a custom source location (rather than unpacking from the tarball as it normally would), a few people have questioned me — is there really a way it can be done without modifying the ebuild at all?
Well, the answer is YES! I’ve just done it.
This is largely thanks to portage-cvs’ ebuild shell capabilities. This really is a fantastic piece of work.
Here’s a quick proof-of-concept example. Note that portage-cvs is required for this.
For many packages, you might say that I can easily point it to use my own set of sources instead of the tarball by simply hacking a value of ${S} to point at my own sources, and not running the ebuild’s src_unpack at all.
Well, what about the ebuilds that do useful things in src_unpack after unpacking the tarball? It would be nice if we didn’t have to also skip them.
Again, I’m using gamin as an example. gamin-0.0.26-r6.ebuild has a src_unpack function which looks like this:
src_unpack() {
unpack ${A}
cd ${S}
# patch to work with inotify 0.21
epatch ${FILESDIR}/${P}-inotify_use_fd.patch
}
So, this is a pretty standard scenario, a tarball unpack and a small patch fix being applied afterwards.
Now onto the real task. You need to install two files:
Next up, we’ll obtain some gamin sources. The target scenario is where you have already got gamin sources lying around in your home directory and you’ve hacked on them somewhat, but for the sake of this demonstration, we’ll put them in place ourselves. Here’s a link to gamin 0.0.26 from the Gentoo mothermirror: gamin-0.0.26.tar.gz.
From your home-directory of your non-root user, extract the tarball as normal.
# tar xzf gamin-0.0.26.tar.gz # cd gamin-0.0.26
Enter the pseudo-ebuild-environment shell:
# ebuild-env
At this point, you can run commands that you normally would inside ebuilds (inherit, unpack, epatch, use_enable, etc) – but don’t do this right now, some of them require some hackery to get going and we should keep the environment clean.
Now run use-ebuild to prepare the environment for working with gamin-0.0.26-r6.ebuild. use-ebuild is a function included in my ebuild.bashrc – it finds the ebuild you specify, sets up some environment variables such as ${P} and ${FILESDIR}, sources the ebuild, and sets ${S} to be the location where you called use-ebuild from (in this case, our locally unpacked gamin sources in our homedir).
# use-ebuild gamin 0.0.26-r6 QA Notice: ECLASS 'eutils' illegal conditional inherit in /gamin-0.0.26-r6 QA Notice: ECLASS 'multilib' illegal conditional inherit in /gamin-0.0.26-r6
Ignore the QA notices for now. You may notice a few other odd messages appearing in this demonstration. I’m ignoring the non-critical ones for now. The important part is that the ebuild has been sourced and the environment has been modified accordingly.
So now lets unpack the ebuild, like this:
# src_unpack * Would normally unpack: * Applying gamin-0.0.26-inotify_use_fd.patch ... [ ok ]
Notice that the normal stage of unpacking the tarball from /usr/portage/distfiles was skipped, it just worked on the sources it found in the current directory. Notice also that the rest of src_unpack was executed as normal – i.e. the patch was applied.
Compilation through the normal portage processes is just as easy
# src_compile
Portage will now run econf and emake with all the other useful voodoo which comes with it.
And thats all there is to it. I haven’t tried src_install or merge onto live filesystem yet, but you can see I’m a good deal of the way there.
I’ve had to include some hackery in the ebuild.bashrc file to replicate the portage environment closely. At the moment, I’ve included the bare minimum to get the gamin example to work. Don’t be suprised if it fails with other packages right now, but you’ll probably find it easy to fix – it only took me 10 minutes to write the whole script.
Many thanks to ferringb for writing the ebuild-shell functionality. I’m very impressed with how simple it was for me to chain onto portage in this fashion. I’ll add this example to the development platform page soon.
Continuing on with the Gentoo development platform venture, I’ve updated the Portage-HEAD installation guide to take account for the fact that sys-apps/sandbox is now in the Portage tree, and also added a Portage patch which allows non-root users to run ebuild for normal operations.
I attempted to tackle an initial scenario: Patching a package (gamin) with a big patch which I got out of their CVS. Unfortunately too much had changed elsewhere in CVS for this to be successful against the latest gamin release, but I reached a milestone in that I hacked ebuild enough to work as a normal user, extracting source into the users home directory, which is slightly more convenient than having to be root and getting it extracted into /var/tmp with restrictive permissions.
I also added a utility to portage, dev-util/rej, which aims to help you apply diff/patch rejects. In this situation, it managed to automatically resolve some of the rejects, which was nice. A handy utility to keep in mind.
dpbuild is a wrapper script for ebuild, which currently just sets environment variables allowing ebuild to be used as a normal user (it will be extended to incorporate other hacks in the future). I’m only thinking in terms of unpacking/compiling here, installation to / obviously won’t work. See the Gentoo development platform page for more info and where to find it.
I recieved some feedback after my last weblog post on this topic. It’s good to know that there are people besides myself who would find some of the ideas I have mentioned to be beneficial to their working environment.
I recently started a discussion on gentoo-dev about some ideas I’ve been considering about making Gentoo more suited for a software development environment.
In the mail, I raised just one issue that I’ve been thinking about – using upstream CVS-sources for a package as opposed to the sources included in the latest release tarball, while still installing the software through the portage build system. It seems that some of the people who replied either missed the point I was trying to make, or thought I’m making a fuss out of nothing – they pointed out that I can just write a CVS ebuild and be done with it. Yeah, that’s not too much trouble, but I’m talking about streamlining processes here guys. If I was to write a CVS ebuild, it would be almost identical in content to the “release” ebuild from the portage tree, so why should I have to write it in the first place? I’ve got the sources, someone elses has written an ebuild, let me combine the two with minimal effort :)
I did get some useful information from the discussion though. The development version of Portage makes some of the things I’m thinking about much more realistic. I’ll talk more about specifics later, for now I just want to document the procedure of installing the development version. Thanks to Stefan (genstef) for sharing this knowledge with me!
Installing Portage-CVS
Since the release of Portage 2.0.51, the portage developers have been doing some pretty rapid development, which appears to be mainly focused on new features, rewriting bottleneck code, and improving general performance. There’s some other good news in here: there’s a few features which really help make progress to improving gentoo as a development platform.
Portage-CVS (aka HEAD) is progressing quickly but has no guarantees for stability. If this software burns your house down, you are to blame for installing it in the first place. Don’t expect it to be flawless at this point in time.
That aside, here’s how to install it on your running system:
cvs -d YOURUSERNAME@cvs.gentoo.org:/var/cvsroot co gentoo-src/portage cd portage ./autogen.sh ./configure
make dist cp portage-cvs.tar.bz2 /usr/portage/distfiles
emerge -av --digest portage
While I’m going to be posting most of the progress I make on this topic here on my weblog, I’m going to archive and maintain everything here as well.
After seeing it requested, I recently ported some of the Beagle code for manipulating extended attributes on files into the core Mono library base. It’s been added in the Mono.Unix.Syscall class, which is set to replace Mono.Posix.Syscall once Mono 1.2 is released and widespread.
This is my first contribution to Mono, with thanks to Jon Pryor (Mono.Unix maintainer) who pointed out that my first attempt at this was incorrect, and kindly went out of his way to describe the possible solutions in depth. Now to write some usage documentation…
Despite the fact it happened a few months ago, I still see people claiming that it is impossible to write CD’s with recent 2.6 kernels due to the internal changes. This is not true!
The first thing to check is that you aren’t using SCSI emulation. You don’t need this in Linux 2.6 – its ugly, and apparently doesn’t work. Disable SCSI emulation and use the new ATAPI features to burn “directly” (no extra configuration options needed other than general ATAPI cdrom support).
Next up, check you are running the latest version of cdrtools/cdrecord, even if this means going into the testing/unstable tree. The latest at the moment is cdrtools-2.01.01_alpha01-r1. The same applies for any other cd recording software you might be using.
It’s also important that you are running the latest stable kernel. This is currently 2.6.11. If you aren’t running this, then you are in no position to complain that things don’t work or file bugs.
A few months ago, the kernel code changed to restrict the commands which can be sent to CD writers. The restriction only allows commands that are related to CD reading and writing. The commands are listed in a table. At first, the table was too small, but thanks to user feedback I helped expand it to cover all commands needed for CD writing.
It seems that many users keep advocating a patch which completely bypasses the command table – and this patch is still included in some patchsets! This is no longer needed! If it does make a difference, then its a bug in either your CD writing software or the kernel command table is missing a valid command entry. If this is the case, send me an email, and I’ll fix things up with minimum effort involved. I’ve done this a few times before and I’m happy to do it again.
Another kernel change made at the same point in time resulted in the ability to write CD’s being restricted to users with write access to the CDROM device. That makes sense, right? (Previously you just needed read access.)
So the first thing to do is check that you have write access to your CDROM node. Assuming your CDROM drive is /dev/hdc, run the following:
# [ -w /dev/hdc ] && echo "OK" || echo "Fix up your permissions"
If you see a message telling you to fix up your permissions, the usual procedure is to use ls to see which group owns the node (usually cdrw or cdrom), add yourself to that group, and then try again. This requires that the group has access to write to the node too.
Moving on. Some software used to advise that you set the suid bit on your cdrecord binary. This means that cdrecord will always run with root priveleges, even if you execute it from a standard user account. Some kernel memory management changes broke cdrecord’s ability to run as root for a while. Even though it’s fixed now, it’s worth turning this off for the security implications alone:
# chmod -s /usr/bin/cdrecord
Note that xcdroast does weird things with suid bits too. For the purpose of testing, I suggest you keep away from xcdroast.
In terms of software usage, people seem to be historically used to running commands like cdrecord dev=ATAPI:0,0,0. An alternative method is cdrecord dev=/dev/hdc. There’s been an eternal debate as to which way should be encouraged – the cdrecord author doesn’t like the dev=/dev/hdc method, as you can probably gather from the warnings that appear. Right now, all you want to do is write CD’s, so no matter what you think, you need to use the dev=/dev/hdc method. Also, do avoid the dev=ATA:x,y,z notation too.
Another possible problem is that your CD writing software is opening up the CD node as read-only. The kernel will reject any write commands sent to it. The software needs to be modified to open the CD node as read-write. I fixed up some of the software such as dvd+rw-tools and cdrwtool a while back – but if there is any more problematic software, please let me know. (This is why its essential that you are running the latest versions of the software in question)
Finally, burn as your standard user account, not as root.
Hopefully this clears some things up. If you do experience problems after following the above precisely, please file a bug.
I’ve noticed that I’m now aggregated onto Planet Beagle. I have no idea who maintains this, but thanks! As a result of this, I’ve started writing more about my beagle work, see yesterdays and todays posts on the configuration system I’m working on. Comments welcome. I’ll hopefully post a basic and working implementation before the end of the week.
I finally recieved my exam timetable today. It’s mostly good news because it means that attending GUADEC is possible, even if I do have to take risks with a Java exam before I fly out, and fly back a day early in order to catch my maths exam. Also looks like I’ll get to meet Stefan while out there. If any other Gentoo’ers are thinking of attending, please let me know!
It seems that Beagle is kicking up some attention at Novell’s BrainShare conference, articles on theinquirer, yahoo, yahoo, and computerworld, while largely the result of the media speculation, give some good PR to the project :)
This timed well with the release of Beagle 0.0.8 – the result of a few weeks of rapid bugfixing. This is by far the most usable release that I’ve been involved in, and is being shipped as a “preview” in Novell’s next desktop release.
If the stability remains, I’ll work on getting Beagle into Portage soon. There are some prerequisites to be completed first but progress is being made.
Yesterday, Stefan Schweizer (a developer who I work with) brought a very cool thing to my attention: getdelta.
getdelta is based on deltup which is a way of storing and applying differences between files. Kind of like diff and patch, but designed for binary files such as compressed tarballs.
deltup is very useful in the context of upgrading distfiles, because typically very little changes between foobar-0.1 and foobar-0.2, so a deltup diff file which could upgrade foobar-0.1.tar.bz2 to foobar-0.2.tar.bz2 would be a much smaller download than downloading the entire foobar-0.2.tar.bz2 file.
The magic of getdelta is that it integrates into portage for downloading your distfiles. As an example, I have subversion-1.1.1.tar.bz2 present in /usr/portage/distfiles, but I now want to upgrade to version 1.1.3.
dsd ~ # emerge -f subversion
Calculating dependencies …done!
>>> emerge (1 of 1) dev-util/subversion-1.1.3-r1 to /
>>> Downloading http://gentoo.blueyonder.co.uk/distfiles/subversion-1.1.3.tar.bz2
Searching for a previously downloaded file in /usr/portage/distfilesWe have following candidates to choose from
subversion-1.1.1.tar.bz2The best of all is … subversion-1.1.1.tar.bz2
Checking if this file is OK.
Trying to download subversion-1.1.1.tar.bz2-subversion-1.1.3.tar.bz2.dtu
[…snip the wget download verbosity…]
17:41:57 (490.22 KB/s) – `subversion-1.1.1.tar.bz2-subversion-1.1.3.tar.bz2.dtu’ saved [828684]
GOT subversion-1.1.1.tar.bz2-subversion-1.1.3.tar.bz2.dtu
Successfully fetched the dtu-file – let’s build subversion-1.1.3.tar.bz2…
subversion-1.1.1.tar.bz2 -> subversion-1.1.3.tar.bz2: OK
cleaning up
This dtu-file saved 5 MB (87%) download size.>>> subversion-1.1.3.tar.bz2 size ;-)
>>> subversion-1.1.3.tar.bz2 MD5 ;-)
>>> md5 src_uri ;-) subversion-1.1.3.tar.bz2
The above process basically found that I had got a previous subversion tarball already downloaded, so it just downloaded the upgrade deltup patch from the getdelta server, which saved 87% of my download for subversion-1.1.3. You’ll notice that portage does its usual md5 checking independant of the deltup process so I don’t see any possible problems relating to binaries being hijacked.
I did a lot of package upgrades yesterday and for every one where I had an older distfile present and I was watching, getdelta typically saved me 70-95% of the downloading, and only once refused to download the deltup patch because it was bigger than the tarball it was going to construct.
The only disadvantage of this is that constructing the new file can sometimes be time consuming, especially for the big files. Here’s an example where I construct linux-2.6.11.2.tar.bz2 (~35mb) from the 2.6.10 distribution:
linux-2.6.10.tar.bz2 -> linux-2.6.11.2.tar.bz2: OK
cleaning upreal 3m42.395s
user 2m2.787s
sys 0m4.462s
Is this any quicker than downloading the whole thing? Perhaps not while I’m staying at uni, where I can get 500kb/sec from UK mirrors, but at home where I’m on standard/unreliable broadband, it definately would be. If you are a dialup user, this is a godsend.
It seems that there are plans to at least trial this as an official Gentoo service, but this is a very nice preview of what might be coming up. To get started, just emerge getdelta and follow the basic instruction to set a new FETCHCOMMAND in make.conf
Went to the Gentoo 2005 UK meet yesterday, here in Manchester (well, Salford, apparently they are different cities even though one encloses the other)
I had put the slides for my talk on the internet and also brought it on floppy just in case. Well, the internet wasn’t working at first and nobody had a floppy drive. But once Rob (tigger) got internet going on his laptop, we sorted it out and I started my talk about 20 minutes late. Felt a bit pushed for time, but managed to get through everything and got some positive comments from those who listened.
One thing I found particularly interesting is that my talk was done from Rob’s laptop running Flash Linux. For whatever reason, he didn’t have OpenOffice installed on his laptop, but there were several USB keys floating around with Flash Linux on, which (optionally) include OO. It was very impressive to see a live system boot from a USB key like this. It’s much better than using a LiveCD as you can directly write back to the flash disk after its booted – it’s literally a live and portable Linux system running a GNOME desktop with common applications such as Firefox/OO on it. It uses genkernel’s autodetection routines on top of Gentoo’s baselayout init scripts to start up the system, so you can literally take your working system on your USB key to any system and it will autoconfigure itself. Rob also seemed impressed by it, he had been introduced to it the day before.
Unfortunately the joys of owning an nvidia-based motherboard mean that I can’t boot from USB. But I think you can use the FlashLinux livecd to boot the USB stick – a question I forgot to ask. Maybe I’ll check this out and buy a USB stick.
After my talk, the Flash Linux developer (Gareth) did a talk on Flash Linux. I’m glad this was rescheduled as I would have missed it if it had been done first thing in the morning. A lot of thought has been put into this and it is a very impressive piece of technology. What I would like to see now is some of this being fed back into Gentoo, and expanding Gentoo’s own build tools to be more suited for building flash-based systems.
Rob did a talk on his source code auditing project for Gentoo, which is a project that I didn’t even know existed. I found that interesting, and Rob will be happy to know that he’s won another gcc+SSP convert, and also that he’ll have the pleasure of looking over some of our older kernel security bugs which we have doubts are actually real issues.
Tom (slarti) did a quick presentation on simple ways to work on the command line better. The presentation was good, but what impressed me most is the live demonstration of the things he’d just talked about which he just improvised on the spot. Great job!
I also enjoyed meeting some other people that I hadn’t met before. I fired off about 20 things about genkernel at Tim (plasmaroo) and he fixed and commited them within a few minutes. That was very useful and shows that developers meeting up in the future to do development work would be very productive.
We ended the afternoon with a discussion on how Gentoo UK should progress in the future (more meetings, etc), before going out for a few final drinks.
As promised, the slides for my talk are here, in SXI (openoffice) format. Apparently this will be available for download as video sometime soon (ugh!). Thanks to everyone that turned up and listened to me :)