Category Archives: Gentoo

Recent hacking on VIA 82Cxxx IDE driver

I’ve recently been doing some hacking on the Linux VIA IDE driver. The driver now supports multiple controllers on the same system (if such a situation could ever exist) and support has been added for the VT6410 and VT8251 chipsets.

All of this is included in Linux 2.6.15.

A side effect of this work is that we had to drop the /proc/ide/via file, a dynamic text file which provided some advanced information and stats about the VIA IDE hardware present on the system, and the attached disks. This kind of querying should be done in userspace anyway, so I have produced viaideinfo to do the same thing. viaideinfo has been tested by quite a few people and is now at version 0.3. It is in portage as sys-block/viaideinfo.

Beagle 0.1.2

Beagle 0.1.2 was released on Friday and has just been added to Portage.

Highlights this time around include a fair reduction in memory consumption, various fixes and optimizations, and the ability to search for matches in a certain date range.

We now require Mono 1.1.10, so the issue where the mono-beagled process would hang at random times is gone.

Users of the firefox plugin will should reinstall the plugin, as it has been improved to use vastly fewer resources (we don’t start a new process every time you visit a webpage any more!).

Unfortunately indexing of word-files has been temporarily dropped in the Gentoo package, until gsf-sharp gets added to portage. This will hopefully happen soon.

For more info, see the release notes.

Booting Linux userland from an External USB Flash Disk

For a long time, it’s been possible to boot Linux from an external drive such as USB flash, without requiring an initrd (initial ramdisk). You can do this with the “root=/dev/sda1” kernel parameter, where sda1 is the partition on the USB flash device containing your userspace (may vary depending on your partitioning setup and if you have SCSI or SATA devices).

(Whether or not your BIOS can boot from USB or if you have to store the bootloader and kernel on a more common boot medium is another story)

As of Linux 2.6.10, the USB flash storage initialization changed, and a side effect of this made it no longer possible to boot Linux from USB flash like this. The change meant that, in situations like this, the kernel would try to mount the root filesystem (on the USB flash disk) before the usb-storage driver was fully initialized.

Users saw errors on bootup along the lines of:

VFS: Cannot open root device 'sda1' or unknown-block(8,1)
Please append a correct 'root=' boot option
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(8,1)

Reverting this change was not an option (it was justified) so we had to find an alternative way to solve this problem.

After a few discussions, we decided that at this point in time there was no safe and elegant solution to the problem, so we had to go for a more simplistic approach. I wrote a patch which was accepted into Linux 2.6.11.

My patch added a rootdelay kernel parameter, which allows the user to specify a delay (in seconds) that the kernel should wait before trying to mount the root device. Since the kernel performs tasks in parallel, the usb-storage driver will continue initializing while the block layer is waiting for the specified delay, thus correcting the sequence of events.

So, to overcome this problem, you might use a grub configuration section such as:

title=My kernel which boots userland from a USB flash disk
root (hd0,0)
kernel /bzImage-2.6.14 root=/dev/sda1 rootdelay=10

Most users will be able to get away with a shorter delay (try 5 seconds), but some flash devices will require a longer delay.

This is a relatively minor feature, the change occurred a fair while ago, and the parameter is documented in Documentation/kernel-parameters.txt, but I’m still seeing a fair few people running into this problem. Hopefully this post will make the solution easier to find, until a more transparent solution is developed.

Digital Persona UareU Fingerprint Sensor driver for Linux

I recently mentioned my new project, writing a Linux driver for the “Microsoft Optical Desktop with Fingerprint Reader” keyboard. The keyboard works fine already (its just HID), but the fingerprint reader sensor requires its own driver, which I’m aiming to create.

Thanks to Tony Vroon from Gentoo for donating this device to me, and to the people who sent emails/comments after the last mention of this. Through this information and some extra googling, I now know:

  • The fingerprint sensor is almost certainly designed/manufactured by Digital Persona, a biometrics company recently assimilated by Microsoft.
  • The fingerprint sensor is probably a Digital Persona UareU 4000
  • If this is true, the fingerprint image is encrypted…
  • …but a developer familiar with the UareU Windows API suggests that the encryption key is defined by the host computer at installation time.
  • …and this forum post suggests that the Microsoft keyboard lacks some features of the UareU 4000.
  • The encryption algorithm might be XTEA (source)
  • VALinux Japan have done some biometrics work, including fingerprint-based login using UareU devices. A presentation (html, pdf) suggests that they have produced/obtained a uareu kernel module which has a /dev/usb/uareu char interface. I wonder if they can help me out.
  • The device is firmware-based, or at least, the driver sends a 1904 byte blob of data found inside one of the windows DLL files on every plugin.
  • This guy also seems to have run into the uareu module. Can anyone help me find his email address?

I’m naming my driver dpfp and the initial work can be found here. So far I have created a script to extract the firmware from the windows driver distribution, and a skeleton driver which pulses the light around the sensor 3 times on plugin (looks very cool!). I’m a bit stuck on the next step, the sniffed data suggests the Windows driver is doing things which are impossible (sending data in the wrong direction). I’ve been experimenting trying to guess what it might be doing, but I haven’t been able to provoke the expected response yet (which is the device sending a section of the firmware back to the host computer — again strange…).

If any other Linux users are in possession of one of these keyboards or a standalone UareU reader, then please email me. I will send in an article to linuxbiometrics.com soon.

Update November 2007: I see I’m still getting a lot of hits from “digitalpersona linux” on Google (I’m still #1). I’ve recently launched a new project called fprint which supports DigitalPersona devices under Linux. Click to go to the project homepage.

Alauda driver is complete

The driver

I completed development of the Alauda driver in September and submitted the driver for inclusion to the Linux kernel.

The driver duplicates some code (checksum, media ID table) which is also present in other drivers, and it looks like we want to figure out a good way to share this code before including my work, which is a fair point – we don’t want to duplicate this yet again.

For now I’m publishing my driver as a standalone patch which people can use until we figure out the real integration details. These devices seem to be more common than I originally thought. Patch available here (against Linux 2.6.14).

Juice Box

In true open-source style, a group of hackers have taken my work and used it for something I didn’t design for: hacking the Juice Box – a portable media player based on ucLinux.

From what I gather, these devices boot from a small amount of NAND flash. To customise the device to a decent level, you need to replace this flash with your own.

Fortunately, xD media is basically NAND flash with a slightly different pin configuration, so they have done crazy things such as solder a pre-programmed XD card to the PCB:

However, you can’t just pre-program these XD cards on any old reader/writer. You need to use a device which gives you access to the physical block layout of the media, so that you can write to block 0 (amongst other things). Almost all XD reader/writer devices on the market handle physical block translation in hardware, and only provide a logical block interface to the host operating system, which does not satisfy the needs of these hackers.

The Alauda is probably the most common device that provides physical access, making writing a driver considerably harder, but allowing you to hack the media in ways such as this. I may even donate my spare Alauda device to their project.

Beagle data storage: Lucene 1.9 and SQLite 3

Beagle sure has come a long way in terms of maturity over the last few months.

I’ve been getting involved with Beagle’s interaction with dotLucene which is the C# port of Apache Lucene – a very powerful text search architecture. Beagle stores text content of indexed files within Lucene ‘databases’ and uses Lucene’s impressive search features to query on behalf of the user.

We previously used dotLucene 1.4.3 within Beagle, but I recently upgraded us to 1.9 RC1. Beagle is mostly unaffected by the changes, but there are some bug fixes and optimizations included. Perhaps the biggest win was the result of my extensive testing to make sure the upgrade didn’t break anything – I did identify and fix two bugs, and they were both also present in the 1.4 code.

The first bug was a file descriptor leak in a common code path (inside Beagle code), and the other, a fairly significant locking bug which was causing the locking often to not be having any effect at all. This explains some of the strange behaviour that has cropped up time to time in the past which we’ve never been able to pinpoint.

I also looked at some traces through the codepaths. I noticed that dotLucene was dealing with throwing and catching exceptions a hell of a lot – hundreds of exceptions being dealt with while indexing a small range of files. dotLucene was using exception catching where simple if/else combinations would work just fine. Exception handling is expensive as the runtime must jump through hoops keeping track of where to jump to if a certain type of exception occurs, so by greatly reducing the amount of exception handling that takes place, we have a nice small optimization in place.

After landing dotLucene 1.9, I’ve now turned some attention to another aspect of Beagle’s data storage mechanism. Beagle uses SQLite to store file attributes when extended attributes are not available, and for its file text cache.

Currently, Beagle only uses SQLite 2.x. Attempting to ‘port’ it to SQLite 3 revealed a problem in our SQLite interaction. You must always query a SQLite database from the same thread that the connection was originally established. Beagle is multi-threaded and we are using the same connection over multiple threads, which is (apparently) unsafe, and SQLite 3 explicitly checks this and returns error if you go beyond the original thread.

This creates a non-trivial problem to solve, and is a poor design decision from the SQLite developers. We’re going to stick with SQLite 2.x-only, as it seems to work just fine even despite sharing the connection over our thread pool. SQLite 3 wouldn’t bring any major benefits to us, and we are unable to use it due to its new explicit thread checking restriction. Sigh.

various stuffs

2.6.13 is almost ready to go stable in Gentoo, especially now that the evil AMD64 SMP bug has been solved (this also affected the last few kernel releases).

Beagle 0.1.0 is out, the result of much hacking from all directions over the summer. The release announcement pretty much says it all. On a sidenote this will be available in Gentoo’s package tree sometime soon.

Alauda driver is pretty much finished – reading, writing, hotswapping – to both XD and SmartMedia cards, even simultaneously, on 2 devices at the same time. The only problem right now is that a tester has reported reading of 8mb smartmedia does not work – this is difficult to track down as I do not own any cards this small, and the address space is different on this media (but the driver is written so that this should work…)

I’ve been donated a MS keyboard with fingerprint reader with the task of getting the fingerprint reader working on Linux. There is a major complication here though, the device appears to simply send an image of the fingerprint to the host computer, but the I think the image is encrypted. Can’t be an impossible problem to solve, right?

Regarding the spam-attacks on the Gentoo hosted weblogs, I can globally remove and blacklist spam (based on keyword or URL) very easily so please just report it to me. If anyone knows of good ways to automatically combat spam in b2evolution or feels like hacking something up then please let me know. I’m not too fond of the “type the letters from this image” schemes, but something like an additional confirmation screen (where the user just has to click a button) if the user included 3 or more URL’s in the same comment would probably confuse the spambots enough to quieten things down.

Update: Missed this earlier, but it looks like the new b2evo release has improved antispam capabilities. Will see how this turns out…

I’ll be offline for a while as of Monday, moving back up to Manchester into a new house to start my 2nd year of university.

Alauda driver now reads all XD

Quick update on the Alauda driver status:

Figured out the rest of the block addressing, so it can now support more card sizes. It also should detect the media size automatically and work “out of the box”, at least it does with the two XD cards I have here.

XD media reading is now pretty much complete, except for a few performance improvements which will be made at a later date. Next up I’ll be getting my hands on some SmartMedia and implementing read support for that.

Code is available from SourceForge CVS.

entagged-sharp in Beagle

I’ve recently been hacking on entagged-sharp, a C# library for extracting tags from audio files. This is used in Banshee, Muine, and probably some other projects too.

It provides a nice simplistic interface for extracting tags, e.g.:

AudioFileWrapper afw = new AudioFileWrapper("/path/to/audio.file");
Console.WriteLine("{0} - {1}", afw.Artist, afw.Title);

…and this will “just work” for all of the audio file formats that are supported.

I worked on extending the AudioFileWrapper interface, reducing internal abstraction, MIME support, unit tests, bug fixing, and adding Amiga module audio file support. ASF/WMA tag filtering is coming soon.

entagged-sharp has now been imported into Beagle CVS, replacing the filtering code we had previously. This pretty much closed all the audio-filtering bugs that we had, and added support for more formats (m4a, m4p, xm, sm, it, mod). Hopefully nothing broke at the same time :)