Category Archives: Misc

Open-source AI on low powered devices

With the recent industry advances around Artificial Intelligence, I was curious: Is there progress towards having this kind of technology usable offline, on low-end computing devices, free of charge, with an element of openness (i.e. ability to manipulate the software or underlying data)?

Perhaps the most well known product within the current whirlwind of AI attention is ChatGPT, which provides a textual interface for conversing with a “chatbot” virtual agent. You can try this out for free, accessing it online using your web browser. However, it’s a centralised service that requires an internet connection. You can imagine it being a beast of great complexity, run by some massively powerful computing cluster. Beyond the web chat version, you have to pay for any other way of using it, and the software and data is private.

Are there other variants of this technology that do not face such restrictions? Anything that could run on a smartphone in my pocket?

My knowledge into AI topics does not run deep – and while I answered a few questions in the few hours I spent here, I’m probably still under-informed or wrong about certain aspects. I’m also not yet personally convinced that recent developments in AI justify all the hype around it. But anyway, here is what I have learned:

Open language models

AI solutions are heavily dependent upon a “language model” of sufficient quality and depth. Production of such models is tricky, requiring complex tooling, plentiful computing resources, and significant human tuning to ensure quality.

The availability of high quality open models has been limited, but that changed to some degree when Meta recently published LLaMA, an enormous and powerful language model, for free. Actually though, it’s not available for download, you have to fill in a request form and they then email you the download link (except I didn’t get any email). Unsurprisingly it has since been “leaked” online, but that doesn’t facilitate any kind of general availability if we’re looking to have some kind of open chatbot solution within reach of a general audience.

But, LLaMA does seem to have somehow spurred a proliferation of several other “open” models, some of which have public download links. For example GPT4All and OpenLLaMA look like they may be providing openly downloadable, decent quality models.

Open source AI solutions

Now we need some software to work with the model. Meta (Facebook) is perhaps the leader in developing AI technology “in the open”, through development of PyTorch, an advanced toolkit for AI-related things.

PyTorch, being written in Python, makes it a poor fit for running on my Android phone. But fortunately llama.cpp has been developed as a compact and portable C++ reimplementation of much of this. llama.cpp also requires the model to be converted to a new “GGML” format, which results in the LLaMA model shrinking from 13GB to 4GB. That is an important innovation in taking these large models towards low end devices.

Putting these together brings perhaps the most fascinating moment of this experiment. llama.cpp is a very small codebase, easy to set up, and once combined with a single model file, you have a human-like chat experience running directly on your PC. Of course, there are no guarantees about the accuracy and fairness of such chatbot, but it is seriously spooky to witness such a small codebase provide such a dynamic experience. It becomes evident that this impressive result builds upon many years of research.

llama.cpp chat demo

Sherpa

I then shifted attention towards running this on my 2019 Samsung Galaxy S10 smartphone. I wanted to use this opportunity to create something with Flutter, Google’s app development platform that lets you build cross-platform interactive applications, but I swiftly discovered that Bip-Rep already combined Flutter with llama.cpp to produce a mobile app: “Sherpa”.

While a promising demo, it didn’t quite work right on my PC, and was crashing on my smartphone. I set about fixing that, before encountering encountering a more serious issue. My smartphone has 8GB RAM, but the operating system uses about 4.5GB, and the remaining RAM was not enough for the LLaMA model. The app could not function.

I then discovered the recent emergence of smaller “3B” models, such as orca_mini_3b, for which the freely downloadable GGML file comes in at under 2GB. To have this run in Sherpa, I had to carefully update the llama.cpp version it was using, and I took the opportunity to rework some internals for improved performance and lower memory usage. Here is the imperfect but promising result running on my smartphone:

Sherpa smartphone demo

While there are undoubtedly some downsides of using such a small model, you can see that the technology is clearly within reach of relatively low end devices, and you can expect the situation to only improve in future, with likely further innovations in the model format as well as the ever increasing memory and processing power in smartphones and similar devices. Additionally, this is open source software, using a freely available model, and works completely offline.

If you would like to try it, I have uploaded a prebuilt Android APK file, and you can combine this with the readily available orca-mini-3b.ggmlv3.q4_0.bin GGML model.

Recent university projects

I’m currently studying innovation & entrepreneurship at Manchester Business School, and I’ve recently finished some exciting industry-linked projects.

In one group, we performed a market study of the offshore wind energy generation sector, working in collaboration with Cella Energy. Cella is commercialising a new approach to hydrogen storage which could have great implications on the energy sector in future. Through primary research with various offshore wind companies and industry experts, we identified the key challenges and benefits that would be offered by replacing costy undersea transmission cable’s with Cella’s hydrogen storage technology – just one potential application where this breakthrough could be applied.

In another group, we performed some research for Corporate Finance North West (Funding Enterprise). Through speaking to local small businesses, venture capitalists and business angels, we identified the key challenges associated with Investment Readiness of small firms in the area, and the difficulties in matching the right investor with the right opportunity. We found that there is a high availability of capital in this part of the country, but identified problems within the procedures under which investments are made. A section of our final report has been published by the client.

Thanks to everyone who collaborated on these highly rewarding and engaging projects.

Learning Spanish quickly

I studied Spanish for about 5 weeks before arriving in Paraguay, a few hours each day, starting with absolutely no knowledge of the language. While I’m still unable to understand most conversations (mostly due to the speed of speech), I’m making a lot of progress each day. Here are the resources I have been using to learn, most of them for free:

  • ProSpanish has a couple of free lessons, which act as a great introduction to the very basics.
  • SpanishDict has about 25 video lessons available. These are good, if you can stand the horrible American accent of la profesora!! And there are some interactive activities after each lesson.
  • BBC Spanish Steps complements the above nicely, with some good practice for listening.
  • LiveMocha is a fascinating website combining language learning with social networking. After completing some interactive learning exercises, you are tested on writing (answering a short essay question) and speaking (recording into the microphone). The magic part is that your work is then handed off to a native Spanish speaker (another community member, who might, for example, be using the site to learn English), who provides feedback and corrections.
  • NotesInSpanish is a useful podcast to complement the above, which we listen to in the car.
  • I bought the book “Spanish Verb Tenses” by D. Richmond. Very useful, it’s a textbook which includes many exercises that drill the verb conjugations into your head.
  • I also had a few private lessons at my house before leaving.

Birthday

I’m 22 today. Celebrating tonight with a few friends.

My internship at OLPC continues to be very enjoyable; we’re making inroads all over the world, and we’re working on communicating our successes more effectively. Big things are happening. My internship ends in a month’s time, and I’ll be leaving the US soon after.

I’m applying for a business/management masters to continue my studies in September 2009. This leaves a gap of 11 months to fill. I’m working on finding opportunities to spend that time assisting OLPC deployments around the world.

All that plus a small business venture is keeping me very busy.

OLPC’s Learning Lab

The learning lab is one of the aspects of One Laptop per Child which you don’t hear much about. It took me a couple of weeks at the office before I stumbled across their lab space in the corner of our office.

The learning lab is staffed by some key people, including David Cavallo from the MIT Media Lab and Cynthia Solomon, one of the original founders of Logo. These people have years of experience working alongside top scientists such as Marvin Minsky and Seymour Papert.

Last week, David Cavallo gave an enjoyable presentation about the education model behind the efforts of learning group. He shared some experiences from around the world and started a discussion on how to better integrate such ideals into the XO laptop and its software. The session has been published on dailymotion – if you’ve got an hour to kill, I’d definitely recommend watching it.

Summer at OLPC

Last week, I took my final exam for my BSc Computer Science degree at The University of Manchester, results pending. I was originally planning to graduate with a masters degree a year from now, but because of some political brain-damage at university, I ended up switching to the shorter degree so that I can take up a great opportunity this summer:

Today was my first day of a 14 week internship at One Laptop per Child, an exciting company producing cool laptops to distribute to children in developing nations with the aim of improving education. I’m going to be working on aspects of the XO software, starting by helping the efforts to upgrade to Fedora 9 for the v8.2.0 August release.

OLPC is based in Cambridge, MA, USA, and this is my 2nd time in this area. I’m living in Brookline until September, and then will need to find somewhere else to live for a month.

My plans for after the summer are uncertain. I’m thinking about hunting for some work in Europe somewhere, I might write more about that nearer the time.

Dreaming

The Pav√£o Quartet have released their fourth album today, “Dreaming.” The lead violinist, Kerenza, is a relative of mine.

Dreaming is similar to their 2006 debut album Someone to watch over me – a mix of well-known songs with some classical tracks too, all uniquely arranged for the quartet.

Buy it from HMV or Amazon.


Track list

  1. Grieg’s Nocturne
  2. Moon River
  3. Dream a Little Dream
  4. Somewhere Over the Rainbow
  5. When You Wish Upon a Star
  6. Schumann’s Traumerei
  7. Gershwin’s Lullaby
  8. Elgar’s Lullaby
  9. Moonlight Sonata
  10. Forgotten Dreams
  11. Fly Me To the Moon
  12. Clair de Lune
  13. New World Symphony

Hunting for a DOS game: Europe map quiz

I remember playing a DOS game on my 386 computer, probably when I was about 10 years old. The game presented an unlabeled map of Europe on-screen, then proceeded to ask me to click on Portugal, Germany, Switzerland and all the other countries in a different order each time you played the game.

The game kept track of my high scores in terms of both the number of countries I got right/wrong, and also the time it took me to identify them all. I played this repeatedly, trying to beat my own records. I learned a lot about European geography this way, and it was fun. It’s similar to this online game except the game I’m thinking of had metrics to keep track of your performance and highlighted the right answer whenever you clicked on the wrong country.

This is a great example of hard fun, one of the principles of educational constructionism behind the OLPC project. As such, I’m considering writing a clone of the game for the XO laptop.

I’d love to dig out the old DOS game again, but I can’t remember what it was called! Does anyone else know of its name, or have any recollection of this game?

OLPC this week

I’ve been back in Boston this week, and I spent some time visiting the OLPC offices in Cambridge. People aren’t joking when they talk about laptops hanging from the ceiling.

Formal testing is a big thing at the moment, so I spent some time helping out testing the upcoming release branches. I also sneaked in a few bug fixes here and there, and managed to solve some irritating interface quirks.

This was my first interaction with the laptops, and I must say, those little XO machines are incredible. I’m overly impressed on all accounts, especially with the vast level of improvement of the currently-being-finished Update.1 software release over the current stable Ship2 release. There are a number of great people there, from an interesting variety of origins – mirroring the multicultural aims of the organisation, I suppose. Many thanks to those who helped me fit in and get started.