Monthly Archives: March 2005

Beagle in the news

It seems that Beagle is kicking up some attention at Novell’s BrainShare conference, articles on theinquirer, yahoo, yahoo, and computerworld, while largely the result of the media speculation, give some good PR to the project :)

This timed well with the release of Beagle 0.0.8 – the result of a few weeks of rapid bugfixing. This is by far the most usable release that I’ve been involved in, and is being shipped as a “preview” in Novell’s next desktop release.

If the stability remains, I’ll work on getting Beagle into Portage soon. There are some prerequisites to be completed first but progress is being made.

Using getdelta to reduce size of distfiles download

Yesterday, Stefan Schweizer (a developer who I work with) brought a very cool thing to my attention: getdelta.

getdelta is based on deltup which is a way of storing and applying differences between files. Kind of like diff and patch, but designed for binary files such as compressed tarballs.

deltup is very useful in the context of upgrading distfiles, because typically very little changes between foobar-0.1 and foobar-0.2, so a deltup diff file which could upgrade foobar-0.1.tar.bz2 to foobar-0.2.tar.bz2 would be a much smaller download than downloading the entire foobar-0.2.tar.bz2 file.

The magic of getdelta is that it integrates into portage for downloading your distfiles. As an example, I have subversion-1.1.1.tar.bz2 present in /usr/portage/distfiles, but I now want to upgrade to version 1.1.3.

dsd ~ # emerge -f subversion
Calculating dependencies …done!
>>> emerge (1 of 1) dev-util/subversion-1.1.3-r1 to /
>>> Downloading http://gentoo.blueyonder.co.uk/distfiles/subversion-1.1.3.tar.bz2
Searching for a previously downloaded file in /usr/portage/distfiles

We have following candidates to choose from
subversion-1.1.1.tar.bz2

The best of all is … subversion-1.1.1.tar.bz2

Checking if this file is OK.

Trying to download subversion-1.1.1.tar.bz2-subversion-1.1.3.tar.bz2.dtu

[…snip the wget download verbosity…]

17:41:57 (490.22 KB/s) – `subversion-1.1.1.tar.bz2-subversion-1.1.3.tar.bz2.dtu’ saved [828684]

GOT subversion-1.1.1.tar.bz2-subversion-1.1.3.tar.bz2.dtu

Successfully fetched the dtu-file – let’s build subversion-1.1.3.tar.bz2…

subversion-1.1.1.tar.bz2 -> subversion-1.1.3.tar.bz2: OK
cleaning up
This dtu-file saved 5 MB (87%) download size.

>>> subversion-1.1.3.tar.bz2 size ;-)
>>> subversion-1.1.3.tar.bz2 MD5 ;-)
>>> md5 src_uri ;-) subversion-1.1.3.tar.bz2

The above process basically found that I had got a previous subversion tarball already downloaded, so it just downloaded the upgrade deltup patch from the getdelta server, which saved 87% of my download for subversion-1.1.3. You’ll notice that portage does its usual md5 checking independant of the deltup process so I don’t see any possible problems relating to binaries being hijacked.

I did a lot of package upgrades yesterday and for every one where I had an older distfile present and I was watching, getdelta typically saved me 70-95% of the downloading, and only once refused to download the deltup patch because it was bigger than the tarball it was going to construct.

The only disadvantage of this is that constructing the new file can sometimes be time consuming, especially for the big files. Here’s an example where I construct linux-2.6.11.2.tar.bz2 (~35mb) from the 2.6.10 distribution:

linux-2.6.10.tar.bz2 -> linux-2.6.11.2.tar.bz2: OK
cleaning up

real 3m42.395s
user 2m2.787s
sys 0m4.462s

Is this any quicker than downloading the whole thing? Perhaps not while I’m staying at uni, where I can get 500kb/sec from UK mirrors, but at home where I’m on standard/unreliable broadband, it definately would be. If you are a dialup user, this is a godsend.

It seems that there are plans to at least trial this as an official Gentoo service, but this is a very nice preview of what might be coming up. To get started, just emerge getdelta and follow the basic instruction to set a new FETCHCOMMAND in make.conf

Gentoo UK 2005 meet

Went to the Gentoo 2005 UK meet yesterday, here in Manchester (well, Salford, apparently they are different cities even though one encloses the other)

I had put the slides for my talk on the internet and also brought it on floppy just in case. Well, the internet wasn’t working at first and nobody had a floppy drive. But once Rob (tigger) got internet going on his laptop, we sorted it out and I started my talk about 20 minutes late. Felt a bit pushed for time, but managed to get through everything and got some positive comments from those who listened.

One thing I found particularly interesting is that my talk was done from Rob’s laptop running Flash Linux. For whatever reason, he didn’t have OpenOffice installed on his laptop, but there were several USB keys floating around with Flash Linux on, which (optionally) include OO. It was very impressive to see a live system boot from a USB key like this. It’s much better than using a LiveCD as you can directly write back to the flash disk after its booted – it’s literally a live and portable Linux system running a GNOME desktop with common applications such as Firefox/OO on it. It uses genkernel’s autodetection routines on top of Gentoo’s baselayout init scripts to start up the system, so you can literally take your working system on your USB key to any system and it will autoconfigure itself. Rob also seemed impressed by it, he had been introduced to it the day before.

Unfortunately the joys of owning an nvidia-based motherboard mean that I can’t boot from USB. But I think you can use the FlashLinux livecd to boot the USB stick – a question I forgot to ask. Maybe I’ll check this out and buy a USB stick.

After my talk, the Flash Linux developer (Gareth) did a talk on Flash Linux. I’m glad this was rescheduled as I would have missed it if it had been done first thing in the morning. A lot of thought has been put into this and it is a very impressive piece of technology. What I would like to see now is some of this being fed back into Gentoo, and expanding Gentoo’s own build tools to be more suited for building flash-based systems.

Rob did a talk on his source code auditing project for Gentoo, which is a project that I didn’t even know existed. I found that interesting, and Rob will be happy to know that he’s won another gcc+SSP convert, and also that he’ll have the pleasure of looking over some of our older kernel security bugs which we have doubts are actually real issues.

Tom (slarti) did a quick presentation on simple ways to work on the command line better. The presentation was good, but what impressed me most is the live demonstration of the things he’d just talked about which he just improvised on the spot. Great job!

I also enjoyed meeting some other people that I hadn’t met before. I fired off about 20 things about genkernel at Tim (plasmaroo) and he fixed and commited them within a few minutes. That was very useful and shows that developers meeting up in the future to do development work would be very productive.

We ended the afternoon with a discussion on how Gentoo UK should progress in the future (more meetings, etc), before going out for a few final drinks.

As promised, the slides for my talk are here, in SXI (openoffice) format. Apparently this will be available for download as video sometime soon (ugh!). Thanks to everyone that turned up and listened to me :)

Announcing Planet Gentoo

I’m pleased to announce the launch of Planet Gentoo, a resource intended to improve communication between the user and development communities, plus enhance communication internally between developers.

Contributing developers write articles to their weblogs, which will then automatically be published on Planet Gentoo, providing a good overview of what is being developed and where things are going. At time of launch, we have over 50 contributing developers, a number which will grow further as more and more developers see the light :)

Planet Gentoo started life as an Enhancement Proposal: GLEP 30. After some inactivity, we’ve finally got it launched, and its looking great.

I’d like to say a quick word of thanks to the following people, who helped out in some way or other: Stuart Herbert, Alexander Plank, Kurt Lieber, Lance Albertson, Karl TK, and anyone I forgot. Also, thanks – the developer feedback from the last day has been great, I’m very pleased with the response so far and that we’re able to launch this with such a big proportion of our active developer base involved. For those developers who are missing out, just email me and I’ll get you involved. If you don’t have a weblog, thats not a problem, we can provide you with one.

I’ll be talking some more about this tomorrow at the Gentoo UK 2005 Conference here in Manchester. It’s now free admission, so if you live nearby, just turn up at the door!

GLEP discussion

In light of the recent GLEP discussion, I see my Planet Gentoo GLEP being used as an example.

My opinion is that Planet Gentoo should never have been a GLEP. We’d discussed the idea in detail on the developers mailing list. There were no arguments against it, and even those that admitted they weren’t the blogging type were able to see that at least aggregating existing developer weblogs would be a good thing.

I became involved, when at the Gentoo UK 2004 meetup, I was chatting to Stuart and a couple of others about it – I recognised the interest. It was obvious that we had demand from users. We had demand from developers. There were no concerns raised in the discussion. The concept was obviously working well for other open-source projects. And setting up a Planet wasn’t exactly going to change much in the way Gentoo works. Its not like having to use repoman instead of CVS commit or anything like that, its totally independant. But we were told we had to “GLEP it” for it to have any chance of happening.

The only major criticism that came up through the GLEP discussion is that the Planet might be seen to be harming our image if there wasn’t enough activity there and it was just left to rot. With 50+ developers on the roll and 10+ emails in my Inbox from developers who want blog’s setting up, its fairly obvious that this is nothing to be concerned about.

As for my views on the GLEP process, after having to go through it, I’d say it was a fairly easy process. Maybe thats just because nobody properly opposed it. I see the timeframe of 4 months being quoted. It would have been much quicker, but with me starting university, getting more involved in other things, I was low on time. Also, the activity of Kurt Lieber, the infrastructure contact who was in charge, was decreasing and this made it harder. Once Lance Albertson took over, and we went over some of the finer details, we had it set up within a couple of weeks. Infact, it only took me part of one evening to get Planet and B2evo (developer log hosting) set up.

On the other hand, the extra time it took might have been a good thing. During that time, the Gentoo Experimental guys demonstrated that the developer demand and interest in this had grown much more. We can now launch the Planet with more than twice the amount of contributors I was expecting when I wrote the GLEP, and thats great.

Announcement coming later today :)

Planet Gentoo update

With Ramereth’s help we finished getting Planet Gentoo set up tonight. It’s currently running at a secret URL and is looking really good :)

I’ve sent a mail out to gentoo-core. Please read it, and if you aren’t listed or are on my I-need-to-talk-to list, please send me an email.

Planning to release and announce this on Thursday or Friday, in time for the Gentoo UK 2005 meet on Saturday. So please get yourselves added quickly! There are currently 29 developers on there.

gentoo-dev-sources-2.6.11-r1 is ready to rock

With help from Kirk Reiser of the Speakup project, we’ve got speakup working with 2.6.11 which means that our 2005.0 LiveCD’s will now be based on the just-released 2.6.11 kernel. This means we’ll have bang-up-to-date support for the latest hardware. I also patched in the new SysKonnect gigabit ethernet driver (I was wrong in my last post, this was never sent to Linus) as I know a few people have asked for it.

No bug reports for 2.6.11 yet, this is shaping up very well :)

Linux 2.6.11

Linux 2.6.11 was released today. Seems like another high quality release incremental from the last one. Many people seem interested in “whats new in this release” and I rarely see any short summaries, so here’s mine:
(I’ve probably missed some important things, this is just what I have observed)

  • Support for SysKonnect Gigabit Ethernet
  • Lots more SATA device support
  • Support for more Promise ATA controllers
  • libata (the SATA driver set) now supports ATAPI (CD/DVD) devices out-of-the-box
  • Support for various components of Intel’s new ICH7 chipset
  • Big direct rendering rework (splitting core from personality)
  • Updated Usermode Linux support – hopefully no more external patches necessary
  • 4 level page tables – memory management improvements
  • Optimization for pipes
  • Kernel userspace events (kevent) – something I need to investigate more, but it allows the kernel to communicate with userspace better
  • Many other fixes, new features, and enhancements (full changelog). If you are having any trouble with your current kernel or have hardware which is unsupported, you should definately upgrade before filing bugs

gentoo-dev-sources-2.6.11 is in portage and any early testing would be appreciated. We are considering using this kernel to power our 2005.0 LiveCD’s, and assuming there are no major bugs early on, our only showstopper is Speakup – it seems that the large amount of changes since 2.6.10 have stopped this patch from applying, compiling, or functionally working. Fortunately one of the main developers, Kirk Reiser, is very responsive and is already on the case :)

Speaking of software releases, Beagle 0.0.7 came out yesterday. As you can see in the release announcement, I’ve been fairly successful at learning C#/Mono and contributing to the software. The core developers have been very helpful and have even granted me access to commit to the GNOME CVS repository, where the beagle sources are managed!
I have more Beagle contributions planned but uni and other work is eating all of my time right now :(

And a quick update on the Planet Gentoo situation. Ramereth says he is almost ready to set it up on the Gentoo infrastructure, hopefully we will get most of the way there tomorrow. Thanks Lance!