Solving PC Battery Drain Issues: A Case Study of Endless Laptop’s Technical Journey

Originally posted at Endless OS Foundation

In June 2022, a community of low-income women entrepreneurs in the United States were approached with the possibility of obtaining their first personal computer under Endless Laptop innovative financing plans. They attended a local event, made a small down payment, and walked away with a powerful Personal Computer full of tools and content ready to support the growth of their businesses and the education of their families.

The Endless Laptop

Our device access program was finally underway, reaching families that had never had a PC in their home before. This initiative was swiftly extended to underserved communities in Guatemala and Mexico, but as the user base grew, a significant flaw emerged: the laptop battery would run down very quickly when the system was in sleep mode.

In this article I wish to share the story of how Endless OS Foundation’s highly skilled Tech team relentlessly chased this issue through several twists into the guts of the PC architecture, eventually pinpointing the problem to the most unlikely of things: a misbehaving device driver corresponding to a type of disk not even present in the product.

Due to the complexity and specificity of this problem, this recap unavoidably needs to draw on fairly technical jargon. I hope it is accessible to those who have at least a loose interest in PC and operating systems architecture!

Solving excessive battery drain through a stroke of good luck?

We first became aware of issues caused by rapid battery depletion in sleep mode at the same time that our Taipei lab staff were coincidentally debugging a different power management issue on a newer variant of the same laptop. It turned out that the new product, which was being considered as a successor, could not sleep at all: upon waking up, it would lose access to the disk.

Our software platform, Endless OS, is based on open source software (Linux), which allows us to tap into an extensive public developer community. In wider discussion about this issue, we tested and approved a workaround for this issue, which would have both laptop models revert to an older implementation of sleep mode.

In addition to allowing the newer product to sleep and wake without issue, this change had the additional effect of greatly reducing the amount of power used during sleep on the original Endless Laptop model that we had already delivered to our userbase. This exercise had coincidentally solved the power usage issue being seen by our users. “That’s handy!”, we thought, as we swiftly rolled out this change to our user base via a software update.

Failures to wake up from sleep mode

Our attention was called back to this issue when we later identified a slow but steady stream of support requests from our users reporting that the device would occasionally get stuck in sleep mode. When this issue was encountered, the system would be unresponsive to any attempt to wake it up from the low power state. It was very hard to reproduce this failure, but we were eventually able to hit the failure and characterize it in detail.

Our workaround to the battery drain issue above was causing these systems to use S3 legacy suspend, a historical implementation of PC sleep mode. In this mode, control of the device is fully handed over to the system firmware when going sleep mode, and the CPU and RAM are powered down. Because the wakeup failure was happening in this mode, it was apparent that the issue was emerging at system firmware level, beyond the reach of the operating system. It is perhaps not surprising that such a firmware issue may exist: this product was not designed for S3 legacy suspend, S3 is likely untested and unsupported on this device, and we should probably not be using it.

Despite the initial indication that we had got lucky with the workaround to use S3 legacy suspend, it turned out to be unreliable and we knew we had to drop this and go back to understand the original problem in more depth. We had two questions to answer:

  1. Why was the newer variant of the product failing to access the disk after waking up, before we put the (problematic) workaround in place?
  2. Why was the device draining so much power during sleep mode, before we put the (problematic) workaround in place?

Intel Volume Management – failed disk access after wakeup

We compared the two product variants closely and spotted the reason why the newer variant had disk access issues after wakeup: it had the disks configured differently.

The original product had been rolled out with Intel VMD, a system function enabling powerful data storage setups, not entirely relevant for our home PC use case. The newer sample had been configured to access the disks in the traditional way, without VMD. And the non-VMD configuration was experiencing the lack of disk access after waking up.

We looked closely and found that our Linux-based operating system was completely powering down the disk in non-VMD sleep mode. This makes sense, because you want to save as much power as possible while the system is sleeping. But we observed that the device was unable to restore power to the disk from that state, and using advanced debugging tools, we observed that Windows, a different operating system, was not cutting the disk power during sleep mode on this product.

Advanced disk power state debugging

We still don’t know why the power is retained in that configuration, nevertheless we updated the Linux behaviour to match. The problem was now avoided, but this time in a way where we had a far more precise understanding of the issue.

Modern Standby: understanding power usage

Now we had both laptop models able to sleep and wake up, regardless of disk configuration, without using the problematic legacy suspend method. It was time to return our attention to the original problem: why is so much power consumed when the system is asleep?

This product uses a Modern Standby design where the core system processor and operating system actually remain active during sleep mode. However, the operating system attempts to turn off as many hardware components as it can (screen, Wi-Fi, disk, etc), pause all apps, and get the processor into an ultra-low power mode where it has almost no work to do. The goal is that power consumption will reduce so drastically that the system can be in sleep mode for days, even though technically you could regard the core system as being awake and running.

In our case, clearly this power consumption goal was not being met. The battery was being drained in a matter of hours in sleep mode.

We called upon some low-level debugging features of the Intel processor that identify which specific parts of the system are reaching their lowest power states during suspend, and which are not. This revealed that the SATA disk controller was preventing the CPU from going into low power mode.

This was a very surprising finding. SATA refers to a type of disk, but this product uses a more modern type of storage (NVMe) – not SATA! Why on earth would the unused SATA controller be getting in our way? What could cause it to prevent the CPU from deep sleep?

The mysterious Tiger Lake SATA power savings issue

Harnessing the power of the open source community, we were able to ask those questions directly to Intel engineers highly familiar with the workings of the hardware. That quickly gave us the exact direction we needed: SATA power savings had been intentionally disabled for this specific Intel “Tiger Lake” processor family on Linux. When power savings had been enabled at an earlier point, it had caused multiple users to mysteriously find themselves unable to boot their computers; nobody knew why.

This suggested that there was probably a whole range of products suffering from this power drain issue. It also meant that we would have to solve this SATA disk issue in order to make progress, despite our product not even making use of SATA.

Refocusing around this challenge, Endless’s Jian-Hong Pan impressed us all by quickly spotting a peculiar detail that had evaded everyone else for years: the code being used to turn on power savings for Intel SATA controllers was quietly and unexpectedly activating an additional behavior change for these devices. Much older Intel SATA controllers needed a “quirk” in order to support multiple disks, and this behavior change had been intended to be restricted to Intel hardware up to around 2017, but 6 years later, Linux was inadvertently applying the quirk to most present day Intel SATA controllers. And for whatever reason, applying this obsolete quirk to the Intel Tiger Lake processors would cause the SATA disks to become completely inaccessible.

Mission accomplished, time to sleep

Thanks to our findings, the Linux SATA maintainers were able to restrict the application of the SATA quirk and activate power savings for Tiger Lake SATA, which should improve power usage on a whole range of devices in addition to ours. We then prevented our disk being problematically turned off during sleep mode and re-enabled Modern Standby for this product, which is now able to achieve around a week of battery life in sleep mode. These fixes were all incorporated into official versions of Linux, and rapidly rolled out to our userbase in Endless OS 5.1.2. With the problem incidence rate subsequently dropped to zero, we can comfortably conclude that Endless Laptop’s first-time PC users in underserved communities are now enjoying long battery life of their devices.

That was a long, hard, fascinating ride. What started with power usage issues took us through suspend mechanisms, firmware issues, disk power management, and quirks for unrelated hardware predating our product by several years. This example demonstrates the skill and resilience of the Endless team, the power of open source communities, and the importance of solving technical issues through truly understanding their root cause, no matter how deep you have to go.

Credit to Jian-Hong Pan and Cassidy Blaede at Endless for their detailed investigation of this issue, David Box and Mika Westerberg from Intel for their speedy and invaluable direction, and Linux SATA maintainers Niklas Cassel and Mario Limonciello for pushing the crucial fixes over the finish line.

Open-source AI on low powered devices

With the recent industry advances around Artificial Intelligence, I was curious: Is there progress towards having this kind of technology usable offline, on low-end computing devices, free of charge, with an element of openness (i.e. ability to manipulate the software or underlying data)?

Perhaps the most well known product within the current whirlwind of AI attention is ChatGPT, which provides a textual interface for conversing with a “chatbot” virtual agent. You can try this out for free, accessing it online using your web browser. However, it’s a centralised service that requires an internet connection. You can imagine it being a beast of great complexity, run by some massively powerful computing cluster. Beyond the web chat version, you have to pay for any other way of using it, and the software and data is private.

Are there other variants of this technology that do not face such restrictions? Anything that could run on a smartphone in my pocket?

My knowledge into AI topics does not run deep – and while I answered a few questions in the few hours I spent here, I’m probably still under-informed or wrong about certain aspects. I’m also not yet personally convinced that recent developments in AI justify all the hype around it. But anyway, here is what I have learned:

Open language models

AI solutions are heavily dependent upon a “language model” of sufficient quality and depth. Production of such models is tricky, requiring complex tooling, plentiful computing resources, and significant human tuning to ensure quality.

The availability of high quality open models has been limited, but that changed to some degree when Meta recently published LLaMA, an enormous and powerful language model, for free. Actually though, it’s not available for download, you have to fill in a request form and they then email you the download link (except I didn’t get any email). Unsurprisingly it has since been “leaked” online, but that doesn’t facilitate any kind of general availability if we’re looking to have some kind of open chatbot solution within reach of a general audience.

But, LLaMA does seem to have somehow spurred a proliferation of several other “open” models, some of which have public download links. For example GPT4All and OpenLLaMA look like they may be providing openly downloadable, decent quality models.

Open source AI solutions

Now we need some software to work with the model. Meta (Facebook) is perhaps the leader in developing AI technology “in the open”, through development of PyTorch, an advanced toolkit for AI-related things.

PyTorch, being written in Python, makes it a poor fit for running on my Android phone. But fortunately llama.cpp has been developed as a compact and portable C++ reimplementation of much of this. llama.cpp also requires the model to be converted to a new “GGML” format, which results in the LLaMA model shrinking from 13GB to 4GB. That is an important innovation in taking these large models towards low end devices.

Putting these together brings perhaps the most fascinating moment of this experiment. llama.cpp is a very small codebase, easy to set up, and once combined with a single model file, you have a human-like chat experience running directly on your PC. Of course, there are no guarantees about the accuracy and fairness of such chatbot, but it is seriously spooky to witness such a small codebase provide such a dynamic experience. It becomes evident that this impressive result builds upon many years of research.

llama.cpp chat demo

Sherpa

I then shifted attention towards running this on my 2019 Samsung Galaxy S10 smartphone. I wanted to use this opportunity to create something with Flutter, Google’s app development platform that lets you build cross-platform interactive applications, but I swiftly discovered that Bip-Rep already combined Flutter with llama.cpp to produce a mobile app: “Sherpa”.

While a promising demo, it didn’t quite work right on my PC, and was crashing on my smartphone. I set about fixing that, before encountering encountering a more serious issue. My smartphone has 8GB RAM, but the operating system uses about 4.5GB, and the remaining RAM was not enough for the LLaMA model. The app could not function.

I then discovered the recent emergence of smaller “3B” models, such as orca_mini_3b, for which the freely downloadable GGML file comes in at under 2GB. To have this run in Sherpa, I had to carefully update the llama.cpp version it was using, and I took the opportunity to rework some internals for improved performance and lower memory usage. Here is the imperfect but promising result running on my smartphone:

Sherpa smartphone demo

While there are undoubtedly some downsides of using such a small model, you can see that the technology is clearly within reach of relatively low end devices, and you can expect the situation to only improve in future, with likely further innovations in the model format as well as the ever increasing memory and processing power in smartphones and similar devices. Additionally, this is open source software, using a freely available model, and works completely offline.

If you would like to try it, I have uploaded a prebuilt Android APK file, and you can combine this with the readily available orca-mini-3b.ggmlv3.q4_0.bin GGML model.

Endless Orange Week 2021

Originally posted at Endless OS Foundation

Last year, one of our valued engineers came to the management team with the question of how we could increase time for pursuit of personal interest topics, relevant to the work being done at Endless but falling outside the day-to-day tasks assigned to our engineers. For example, how could our product engineers find some time to field test Endless Key within their local community?

We discussed this within our management and leadership teams and came up with the Endless Orange Week concept, a week-long period dedicated to learning, experimentation and growth, where each participant puts their ordinary responsibilities aside and instead work on a project according to their personal interests and motivations.

We communicated the plan to the organisation a couple of months before the scheduled Endless Orange Week dates, requesting everyone’s participation. We then asked everyone to brainstorm project ideas within the following categories:

  1. Public contributions to community initiatives that relate to your domain of work
  2. Teaching initiatives where you share your professional skills and knowledge around a particular topic to a group of participants
  3. Directed study towards skills that are at least tangentially related to the professional skills you employ at Endless.
  4. Advocacy initiatives related to Endless’s work or the technologies used at Endless
  5. Taking Endless’s work out further into the world
  6. Advancing one of your team’s interesting, long-neglected projects or ideas

We encouraged the sharing of a broad selection of project ideas – in addition to sharing personal projects selected by each participant, we requested further ideas to be shared that could act as a source of inspiration for their peers. We encouraged projects to be worked on in teams, as well claimed by individuals. We encouraged projects to be experimental, seeking personal learning & development over successful execution & results.

As the week arrived, we communicated the final plan, and asked everyone to begin their week-long project focus, additionally requesting the following communications & deliverables:

  1. “A-ha!” moments & milestones reached shared on Slack, in addition to a progress summary at the end of each day
  2. Creative assets (code, graphics, etc) produced as part of the initiative shared openly online
  3. At least one blog post explaining the project and the outcomes, to maximize the community learning aspect.
  4. A 5 minute presentation or demo, to share the project & results with the whole team.

Projects & results

Selected projects included directed study towards professional qualifications, public writing about Endless-adjacent topics, accompanied by a host of software engineering efforts. Here are a selection of articles explaining our results:

Avatar Editor for Hack character creation.
Experimenting with graphics acceleration on Raspberry Pi 4
Improving the screen share experience in GNOME.

Feedback & reflections

“This was great fun and it seemed great for team morale! I hope we can have a repeat or another similar activity in the not-too-distant future.”

“Thanks for the really fun experiment. I hope we’ll get to do this again! I already have a bunch of project ideas in anticipation.”

The initiative was a clear success! We surveyed the participants at the end of the week, and recorded 100% agreement that participation was beneficial for their professional development and/or advancing Endless’s mission. The following week, the project presentations were strung together in two organisation-wide gatherings, which made for an exciting and engaging finale.

“I especially enjoyed seeing the work everyone was doing and having everyone working in the same space on Slack. I think several projects were a good opportunity for people to interact with other organisations doing cool things, as well, which felt useful and rewarding.”

“I was able to complete a significant portion of the classes during Orange Week. I absolutely feel there were positive and successful aspects of this initiative as it allowed me to get that much closer to obtaining my Not-for-Profit II certificate.”

“My impression is that the majority of the team engaged wholeheartedly with the initiative, and it was generally well-received, not only by engineers but by the broader organisation. It was refreshing for me personally to have an opportunity to focus almost exclusively on one thing for a week, in contrast to my regular work in past years.”

We enjoyed wholehearted cooperation with our suggestion to issue quick and vibrant updates about progress during the week, and that created an air of excitement amongst the group. And we had a real sense of cross-organisation “togetherness” through having the entire staff participate at the same time.

“One very positive aspect I found was reading the updates on the foundation channel. Sometimes updates can be very hard to follow, when they are very technical, and it’s hard to connect when you are not involved on the project, but during the week I think many of us tried to give updates to a wider audience which was great.”

Being the first Endless Orange Week, we also came away with some valuable observations and areas for improvement:

  1. Many participants felt that the week flew past surprisingly quickly. Many projects had been planned with optimism, but when we got down to the details, things went slower than planned.
  2. Even though everyone felt like they advanced their knowledge, several participants reported feelings of demotivation during the event. This happened when the project was found to be more complex than anticipated, or even infeasible.
  3. There were one or two impromptu drop-in “coffee chats” where participants discussed their progress and challenges; these were remarkably successful in reinvigorating any lost optimism.

“I sure did learn a lot about what I needed to do to make this project happen, but I believe I just set the bar too high in regards of what was the expected outcome of the project.”

With the fantastic engagement and results from this 2021 initiative, we are now exploring the possibility of repeating the event, incorporating some changes based on what we learned. We are also considering some different parameters, like further encouragement of team projects, and further pushing for projects to take steps away from our usual day-to-day work.

Turning Smartphones into Computers

Originally posted at Endless OS Foundation

Endless was born from the idea that a smartphone could be connected to a TV in order to provide the full experience & educational capability of a Personal Computer, without requiring the separate purchase of comparatively expensive PC hardware. Because the number of TVs and smartphones distributed around the world far exceeds that of the PC, we still believe this is a game-changing opportunity waiting to be realized in terms of reducing the digital divide.

Endless was born from the idea that a smartphone could be connected to a TV in order to provide the full experience & educational capability of a Personal Computer.

An accidental discovery

At home, I make use of a small gadget on my desk, to which I have connected a keyboard, a mouse, a HDMI display, and my laptop charger. When I get home, I quickly plug this box into my laptop. At this point, my laptop transforms itself into a desktop computer – seamlessly making use of the keyboard, mouse and display. It charges the laptop battery at the same time. This is known as a USB-C/Thunderbolt hub, it is a modern day version of the docking station and it is very convenient.

USB-C Hubb 5 portar

One evening, with my smartphone battery running low, I decided to plug the hub into my phone instead of my laptop, hoping it would charge like my laptop does. Indeed, the smartphone battery started charging. But I was stunned to see that my smartphone immediately took advantage of this connectivity to present a PC-like desktop environment on my monitor, as you can see in the photo at the top of the article.

A smartphone. Connected to, essentially, a TV, plus keyboard and mouse. The result: a PC-like experience. This is Endless’s original idea, turned reality a decade later.

The Samsung DeX experience

My smartphone is a Samsung Galaxy S10, and the functionality I’d stumbled across here is known as Samsung DeX. And it’s rather well done.

Getting started with DeX is quick and convenient. No configuration is needed – you literally just plug in the hub and then you have the desktop on screen within seconds.

And this is not just your phone showing up on a bigger screen. It’s a re-imagination of your smartphone’s user experience, carefully adapted for more of a desktop setup. At Endless we comprehend the effort required to produce a compelling and delightful desktop experience, and it’s clear that plenty of thought went into DeX’s presentation and interaction models.

It’s a re-imagination of your smartphone’s user experience, carefully adapted for more of a desktop setup.

It was really exciting to see this and I immediately knew what I had to do next: I shut down my laptop, put it on the shelf, and moved to using DeX for my professional and personal PC needs for a couple of days as an experiment. Here are my main observations and reflections.

App-based desktop convenience

Like many other digital citizens, I have taken steps to synchronize my smartphone with my laptop. I have accounts and apps set up so that I can read my email or check my calendar on both devices. I am logged into WhatsApp on my laptop so that I can access messages sent to my phone. I use online banking on my laptop but also have installed my bank’s app on my phone for occasional use there. This has taken some effort as I’ve essentially had to set up multiple services on two distinct devices.

Now, under the lens of my smartphone fully replacing my PC, one of my first big realizations was that all my stuff is already there, in full, in the DeX setup. My existing smartphone apps can be run in this environment and they access the accounts and data already on my phone. With a single device serving as both the portable smartphone and also as the home PC, there’s no longer any need to set things up twice. Convenient and refreshing.

An even more exciting realization here is that DeX promotes the app-based desktop concept.

One of our innovative steps in Endless OS is to build the user experience around a central concept of apps, much like a smartphone. This is in contrast to a traditional PC interface, where users typically spend almost all of their time within the web browser. An even more exciting realization here is that DeX promotes the app-based desktop concept. As a PC user I’ve developed the habit of gravitating towards doing everything in the web browser, but after a short period of adaptation, under DeX I find myself preferring to use the individual dedicated apps for reading my email, booking things on the calendar, checking the map, logging into my bank account, etc. Similarly to the way we use smartphones, this feels logical and organized.

Productive results

With no setup required, and my apps, accounts and data already present within the environment, I was able to get my DeX-driven working day underway smoothly. That included my usual tasks of writing documents, reading/writing emails, web browsing, communication on Slack, video conferences via Google Meet, etc. In the evenings I did some online shopping, instant messaging, and also enjoyed being able to use some smartphone-only apps as part of my home PC setup.

Overall the experiment was both uneventful and remarkable. Uneventful in that I was able to get a bunch of work done. And remarkable, because my usual designated workhorse (my laptop) sat to one side gathering dust. My smartphone replaced my laptop. Performance was good, I had all the tools I needed, and my productivity was at its normal level.

That’s not to say there weren’t imperfections. Getting down to the details:

  • Some UI elements want you to swipe (e.g. dismissing notifications) and this does not feel natural with a mouse.
  • Big emails and documents hang for a few seconds, before being notified that grammar checking has been disabled because of too much text.
  • At one point, the Alt-Tab window switcher got stuck on screen and I couldn’t find any way to recover other than rebooting the smartphone.
  • Regular phone notifications are shown on-screen but it took me some time to spot where that was happening.

There were some other niggles related to the apps themselves which have been designed for a smartphone, where they are used one-at-a-time with touchscreen interactions only. Under DeX, they are displayed as large, resizable windows and are driven by the keyboard and mouse. When running apps in this ‘foreign’ environment, I hit some pain points such as Google’s Calendar app being more limited than the web version, and I couldn’t find a way to include images in mails sent in the Gmail app. Some apps require you to do a pinching motion, which you can’t do on a mouse. And some apps, like Slack, don’t make good use of the huge amount of space now available, continuing to behave as if running on a small screen (unlike the behavior of their PC & web versions).

These technicalities do not seem insurmountable though. And Google’s Chrome app shows just how well this can work. On my phone, the Chrome app runs in its regular minimalistic mode, but when running that same app on the big screen, it impressively transforms itself to look and behave like the PC desktop version of Chrome.

Software freedom

Endless is a strong proponent of Free and Open Source software being the right base for sustainable, scalable access to PCs. And as exciting as it feels to put my laptop to one side and switch to a PC experience produced by my smartphone, I faced a sinking feeling in this area: this is the first time in around 20 years that I’m not using an open source platform for the tasks I do on a PC. That feels like an unfortunate step backwards. While built on some open technologies, you cannot really consider the Samsung Galaxy Android OS to be a FOSS project, and the DeX functionality seems to be entirely proprietary.

That said, this does highlight an inconsistency in ideals. Why do FOSS proponents such as myself feel strongly about the need for open technologies on PC desktops, while simultaneously holding comparatively closed mainstream smartphones in our pockets?

Closing remarks

At the start of my 2-day experiment of using DeX instead of my PC, I had the feeling that it would fail. I had the perception that my laptop was an essential part of my work, I figured that a handheld smartphone would not be able to offer enough performance to replace a PC, and I expected the DeX experience to be sufficiently rough to have me craving the return to my laptop.

I was wrong on all these counts. I remain astounded that this experiment went as smoothly as it did. The DeX experience even surprised me with its smooth realization of the app-based PC desktop experience and the convenience of already holding all my apps and data. I did face minor inconveniences which left me considering if I should grab my laptop, but these could just be ordinary switching costs. The technical issues could likely be solved by developers without much effort. The overall experience seems so close to being a drop-in PC replacement.

From an Endless perspective, it’s hugely exciting progress in two of our focus areas. Firstly, the implication that a smartphone can replace a PC is hugely powerful, because smartphones are more affordable and much more widespread than PCs within the underserved communities we seek to benefit. Secondly, it delivers the app-based desktop experience in the PC-like environment, which is the same direction of travel that we have within Endless OS. The downside is that it’s a step backwards in terms of software freedom, but I am hopeful that open source implementations could emerge in future.

My biggest question: Why is this game-changing feature so hidden and unknown? Despite having so much relevance for our work at Endless, and with two DeX-capable smartphones at home, it still took 3 years for me to come across this – purely by accident.

Now that I’m aware of it, I see user discussions around DeX going back to 2018, and I see similar solutions such as Motorola Ready For and Huawei Easy Projection. There’s even the NexDock which uses DeX or similar to convert your smartphone into a laptop.

This feature delivers a compelling user experience, to the point where I will continue actively using it while reconsidering the importance of a traditional PC going forward.

Imperfections aside, this feature delivers a compelling user experience, to the point where I will continue actively using it while reconsidering the importance of a traditional PC going forward. As Matt Dalio writes in his accompanying article on this topic, we now need this feature to mature to the point where it is common in low-cost smartphones, and at that point, it could bring great positive impact to emerging markets where PC adoption is still in its infancy.

Chromebook app building discoveries

Originally posted at Endless OS Foundation

50 million students & educators use Chromebooks worldwide, but these devices are limited without broadband at home – a problem for up to 16 million students in the USA. Here are 6 things we’ve learned so far as we test making the Endless Key available as a Chromebook app.

  1. Most education-focused Chromebooks have limited storage, 16GB or 32GB internal disk – a major challenge for making our normal 128GB of curated Endless Key learning and discovery content fully available.
  2. Chromebooks distributed through schools ordinarily block installation of additional apps. Typically only the school (or district) can explicitly permit further app installation. This already limits what after-school clubs can access through the school’s devices.
  3. Some education-focused Chromebooks lack SD card slots, or have limited USB ports – a major challenge for storing additional content to make it easily available to offline students.
  4. Chromebooks distributed through schools are ‘managed’ and tend to include a high degree of monitoring and filtering. Both human and automatic reviews of what is being accessed on the device happen.
  5. Chromebooks support apps in two main formats: pure webapps (PWA) or Android apps (running inside a container).
  6. There are many deployments where students can take devices home, but they are generally required to return them to school at the end of the year, and won’t have them over their summer break.

We’re working through these challenges and more with Learning Equality, starting with Kolibri process management and access to external storage. Porting an app from one platform to another is rarely straightforward, and usually requires some architecture changes.

An Engineering Team in transition

Originally published at Endless OS Foundation

Endless previously operated as the for-profit entity Endless Mobile for around 7 years, before completing the transition to the nonprofit Endless OS Foundation in April 2020.

This transition did not involve a change of mission – even as a for-profit we were built around the social impact goals of empowering the world through technology – but the strategy and the approach to making our mark changed significantly.

It took us a good amount of head-scratching and going back to the drawing board to really grasp what it means to run a streamlined nonprofit organisation, especially while carrying around years of history from a different structure. I’d like to share some of the challenges faced by our engineering team during this transition, and show how we have adapted for our new organisational shape.

Breaking away from the for-profit model.

Previously, as a for-profit, we sought to build a thriving social enterprise where our success as a business would provide plentiful resources to further grow the impact and reach of our work. This did have an impact on our execution. For example, it entered us into the competitive platform game, where our business was reliant on people adopting Endless’ own software in order to bring possibilities for commercial growth (for example, by offering premium features or extra educational content at very low cost).

This led to our earlier focus on building Endless OS, our own Linux-based PC operating system, which provided exclusive access to our own education solutions and content. In some circumstances though, this all-inclusive product approach presented potential adopters with an awkward “us or them” question, where they could adopt either Endless’ platform or an alternative from another provider. Having to choose one solution would exclude them from receiving the benefits offered by the others. Ultimately, this aspect of our business model was holding back our ability to deliver social impact in some cases.

Our commercial business model was holding back our ability to deliver social impact in some cases. Now engineering impact is maximized when value is created that extends beyond our own platform.

Focusing our work on our mission & strategy.

Now as a nonprofit, our focus is not on achieving commercial success to sustain future growth. Instead, our prospects for success depend on delivering on our mission around the world, creating the maximum amount of impact for social good.

It took us a couple of rounds of discussions to grasp what this means in a comprehensive sense, with such conversations leading us to an entirely new strategy. In essence, we realised that it is no longer necessary for us to “win” at the platform game; in fact Endless’ impact can be maximized if the value created can extend beyond our own platform to a much wider audience.

With that in mind, we derived new initiatives around the 4 digital equity barriers that we wish to overcome:

  1. Devices, where our innovations around making laptops more affordable increases access to PC ownership.
  2. Connectivity, where we join forces with Kolibri and Flathub to deliver offline learning solutions.
  3. Literacy, or the ability to use your PC to learn, where we collaborate with the GNOME ecosystem around improving free and open access to desktop computing.
  4. Engagement, where our work increases the quantity and quality of free educational material available for offline use to everyone.

Our revised, nonprofit strategy opens up significant new opportunities for impact that we would have been unlikely to consider in our previous commercial structure. While we continue to believe in Free and Open Source personal computing as the most scalable way to extend the benefits of education & learning on PCs into all corners of the world, we can now also seek to benefit existing audiences of alternative platforms. One such example is our recent Endless Key initiative which specifically targets students with access to a Windows PC. Under our old for-profit mindset, our solutions would have been focused exclusively on our own platform instead of Windows.

The “discovery” channel of our Endless Key for offline Windows learners.

Now as a nonprofit, our team’s focus is on delivering our mission around the world via our work, creating the maximum amount of output for social good.

Community & transparency as an ingredient for growth of social impact.

Endless has always been a strong proponent of building software in open source community settings, but there was formerly a need to make some elements private in order to retain control over business opportunities. Now, as an impact-focused nonprofit, we have little reason to keep anything private. On the contrary, we want to enable and encourage others to take, adapt and build upon our work in any way that brings benefit to the world.

To go about this, we are working to provide full, open access to Endless OS’s code and surrounding infrastructure. As one step here, we recently released the Endless OS Image Builder which is the tool that we have used for years to offer tailor-made, custom versions of Endless OS for specific deployment contexts.

We’re also seeking to increase our transparency. On a technical level, we’ve taken steps here such as opening our Debian package repository and providing access to the latest, bleeding-edge Endless OS development version. We are currently looking at improving communication about our development direction and providing public access to our bug & task tracker.

We hope that our efforts towards increased transparency and access to our technologies will continue to support a growing community where we enjoy receiving independent impact deployment reports, development contributions, highly-skilled technical assistance, user guides, and more.

As an impact-focused nonprofit, we have little reason to keep anything private. On the contrary, we want to enable and encourage others to take, adapt and build upon our work in any way that brings benefit to the world.

Increasing engineering efficiency & streamlining product focus.

Being an effective nonprofit includes sustaining our team & efforts in the most efficient way possible, to maximize the resources that can be directly focused on executing our core mission.

Through our previous activities in a commercial context, where we had adapted and refined our software for use in specific environments and locations, we had accumulated a lot of miscellaneous features. In the engineering world, the existence of features carry a continued maintenance overhead, even if the aim is just to retain them in their existing state without developing further improvements.

At a fundamental level, we build upon a Free and Open Source software base, developed and supported by large worldwide communities. In some cases, our activities had led to us making modifications to some of these components. Just like product features, these deviations were generating a maintenance burden. We knew we had to reconsider the amount of effort put in these areas, and prioritizing our team’s time toward our new strategy became the lens for our re-evaluation.

We have maximized the engineering resources that directly link to executing our core mission.

With our efforts refocused around the specific impact areas detailed above, we took a step back to look at everything we had built, and decided to reduce and remove a significant amount of features and deviations that were not critical to achieving our mission.

  1. We reduced our software to a simpler and leaner form, leaving only the key features that have a direct connection to our strategy and initiatives. This included removing integration with Android smartphones, our prototype offline news-feed for discovery, and our limited solution for managing websites and web links.
  2. We dropped most of our modifications to Open Source software components, such as support for obscure hardware platforms no longer of relevance and our own printer driver manager.
  3. We revised our processes for increased efficiency, including greatly simplifying the Endless OS build process to use Debian’s software repository directly, rather than building thousands of components.
  4. We consolidated and reduced our server infrastructure and services, and for new initiatives we have been calling upon serverless solutions instead, which require less maintenance.

This far-reaching streamlining of our efforts turned out to be more work than anticipated, but the team recognised the sustainability gains at play, embraced the task, and everyone is feeling pleased with the results.

Collaboration and transparency with the larger open source community is a critical strategy at Endless.

Direct collaboration with upstream partners.

Combining the threads of maximizing impact, increasing sustainability and streamlining the software, we made some bold strategic shifts to work more closely with related technologies instead of creating our own solutions to some problems.

The most visible example is our handling of educational content. During the platform game years, Endless built an elaborate cloud-based system for ingesting content and generating offline content apps, exclusive to Endless OS. However, with our new strategy in mind, we made the difficult decision to turn that system off entirely and dedicate our educational content efforts towards Kolibri, a learning system produced by our partner Learning Equality.

Our users frequently praise the usability of our software, which we built from years of research amongst first-time PC users in emerging markets. To reduce the engineering cost of sustaining usability features, we have re-engaged the GNOME desktop computing community and contributed to usability research and redesign efforts that became part of GNOME 40. In the long term we hope to improve the GNOME desktop usability to the point where it is no longer necessary for Endless to maintain its own design; this would be a win for both sustainability and impact.

The Endless OS desktop, praised for its ease-of-use.

Above:

Combining the threads of increasing sustainability and streamlining software, we made some strategic shifts to work closely with related technologies and organizations — instead of creating our own solutions.

Getting closer to our users.

While transitioning to launch the Endless OS Foundation, we also identified the need to retain and improve our strong connections to our end users and the partners who take our work out into the world.

We experimented by creating a Deployment team within our Engineering division, responsible for the communication and technologies needed to support our partners. However, we later realised that this structure was creating distance between engineers and users, and moved to a structure where the responsibility for ensuring our software’s relevance for our users is something that explicitly belongs to each and every engineering team member.

Frequent user testing was one of the original pillars of Endless’ design and engineering efforts, and this must be kept amongst our regular engineering activities. We recently performed PC desktop usability testing on multiple continents, and tested our new Endless Key design and content with students across the US. Our engineers join these testing sessions, which aids them to relate better to our end users and make optimal engineering decisions.

Our software’s relevance for our users is a goal that explicitly belongs to each and every engineering team member.

There’s still work to be done after our team’s successful transition.

After years of operation under a different business structure, we faced many challenges in the transition to a nonprofit. However, after several demanding strategic discussions, we’ve been able to make significant changes to the structure and activities of our Engineering group directed towards greater social impact. These changes brought exciting progress through in 2021 and I’m excited about our team’s continued growth along these lines as we look towards a highly impactful future in the coming years.

Kolibri entirely in the web browser

Earlier this month, I participated in Endless Orange Week, a program where the entire Endless team engages in projects designed to grow our collective learning related to our skills, work and mission, while our ordinary work responsibilities and projects were put on pause.

I used the opportunity to do some experimentation around Kolibri, Endless’s chosen solution to provide offline access to educational content.

Especially with the proliferation of Chromebooks distributed by schools during the pandemic, there’s a clear opportunity for Kolibri to increase its impact by becoming available as a ChromeOS app. Kolibri can already be used on ChromeOS when there is a Kolibri server available over the network, however we are now focusing more on the independent learner case, providing access to educational content in limited-connectivity settings. We would like to make the entire Kolibri solution available to run on ChromeOS directly.

However, Kolibri has an advanced architecture due to its position as a highly featured learning environment, and that makes it challenging to bundle in a way that is compatible with constraints under which ChromeOS apps must operate. On a technical basis, Kolibri uses web technologies for the frontend, but relies on a Python backend.

I chose to experiment with one possible approach towards running Kolibri on ChromeOS: using modern web technologies to run the existing architecture largely as-is.

This experiment appealed to me as a way of gaining some hands-on experience with WebAssembly as part of the solution, an interesting technology that can vastly expand the capabilities of websites. In this case, I used Pyodide, a technology that lets you run Python code directly within the web browser.

Within the allocated week, I managed to get Kolibri up and running in this shape, at least to the point of the setup wizard. It was rewarding to see all this fit together. However, I did not have enough time to make it into a usable app, and there are some key challenges left largely unaddressed such as how we handle the storage of educational content.

I was also able to uncover some technical considerations that question the overall feasibility of this idea. Ultimately, while WebAssembly & Pyodide have truly achieved something magical in that Python code can be run in the web browser, there are fundamental challenges in bringing the web and Python worlds together and making them interoperate smoothly. To support complex apps like this without requiring rearchitecting Kolibri, Pyodide needs to bring these worlds together seamlessly and there are numerous complications in making that happen.

Code & technical notes
Click to see the experiment in action

I really enjoyed being able to focus some time on this experimental project and it was fun observing my peers also learning new things via their own Endless Orange Week projects at the same time.