Turning Smartphones into Computers

An accidental discovery

At home, I make use of a small gadget on my desk, to which I have connected a keyboard, a mouse, a HDMI display, and my laptop charger. When I get home, I quickly plug this box into my laptop. At this point, my laptop transforms itself into a desktop computer – seamlessly making use of the keyboard, mouse and display. It charges the laptop battery at the same time. This is known as a USB-C/Thunderbolt hub, it is a modern day version of the docking station and it is very convenient.

One evening, with my smartphone battery running low, I decided to plug the hub into my phone instead of my laptop, hoping it would charge like my laptop does. Indeed, the smartphone battery started charging. But I was stunned to see that my smartphone immediately took advantage of this connectivity to present a PC-like desktop environment on my monitor, as you can see in the photo at the top of the article.

A smartphone. Connected to, essentially, a TV, plus keyboard and mouse. The result: a PC-like experience. This is Endless’s original idea, turned reality a decade later.

The Samsung DeX experience

My smartphone is a Samsung Galaxy S10, and the functionality I’d stumbled across here is known as Samsung DeX. And it’s rather well done.

Getting started with DeX is quick and convenient. No configuration is needed – you literally just plug in the hub and then you have the desktop on screen within seconds.

And this is not just your phone showing up on a bigger screen. It’s a re-imagination of your smartphone’s user experience, carefully adapted for more of a desktop setup. At Endless we comprehend the effort required to produce a compelling and delightful desktop experience, and it’s clear that plenty of thought went into DeX’s presentation and interaction models.

It’s a re-imagination of your smartphone’s user experience, carefully adapted for more of a desktop setup.

It was really exciting to see this and I immediately knew what I had to do next: I shut down my laptop, put it on the shelf, and moved to using DeX for my professional and personal PC needs for a couple of days as an experiment. Here are my main observations and reflections.

App-based desktop convenience

Like many other digital citizens, I have taken steps to synchronize my smartphone with my laptop. I have accounts and apps set up so that I can read my email or check my calendar on both devices. I am logged into WhatsApp on my laptop so that I can access messages sent to my phone. I use online banking on my laptop but also have installed my bank’s app on my phone for occasional use there. This has taken some effort as I’ve essentially had to set up multiple services on two distinct devices.

Now, under the lens of my smartphone fully replacing my PC, one of my first big realizations was that all my stuff is already there, in full, in the DeX setup. My existing smartphone apps can be run in this environment and they access the accounts and data already on my phone. With a single device serving as both the portable smartphone and also as the home PC, there’s no longer any need to set things up twice. Convenient and refreshing.

An even more exciting realization here is that DeX promotes the app-based desktop concept.

One of our innovative steps in Endless OS is to build the user experience around a central concept of apps, much like a smartphone. This is in contrast to a traditional PC interface, where users typically spend almost all of their time within the web browser. An even more exciting realization here is that DeX promotes the app-based desktop concept. As a PC user I’ve developed the habit of gravitating towards doing everything in the web browser, but after a short period of adaptation, under DeX I find myself preferring to use the individual dedicated apps for reading my email, booking things on the calendar, checking the map, logging into my bank account, etc. Similarly to the way we use smartphones, this feels logical and organized.

Productive results

With no setup required, and my apps, accounts and data already present within the environment, I was able to get my DeX-driven working day underway smoothly. That included my usual tasks of writing documents, reading/writing emails, web browsing, communication on Slack, video conferences via Google Meet, etc. In the evenings I did some online shopping, instant messaging, and also enjoyed being able to use some smartphone-only apps as part of my home PC setup.

Overall the experiment was both uneventful and remarkable. Uneventful in that I was able to get a bunch of work done. And remarkable, because my usual designated workhorse (my laptop) sat to one side gathering dust. My smartphone replaced my laptop. Performance was good, I had all the tools I needed, and my productivity was at its normal level.

That’s not to say there weren’t imperfections. Getting down to the details:

Some UI elements want you to swipe (e.g. dismissing notifications) and this does not feel natural with a mouse.
Big emails and documents hang for a few seconds, before being notified that grammar checking has been disabled because of too much text.
At one point, the Alt-Tab window switcher got stuck on screen and I couldn’t find any way to recover other than rebooting the smartphone.
Regular phone notifications are shown on-screen but it took me some time to spot where that was happening.

There were some other niggles related to the apps themselves which have been designed for a smartphone, where they are used one-at-a-time with touchscreen interactions only. Under DeX, they are displayed as large, resizable windows and are driven by the keyboard and mouse. When running apps in this ‘foreign’ environment, I hit some pain points such as Google’s Calendar app being more limited than the web version, and I couldn’t find a way to include images in mails sent in the Gmail app. Some apps require you to do a pinching motion, which you can’t do on a mouse. And some apps, like Slack, don’t make good use of the huge amount of space now available, continuing to behave as if running on a small screen (unlike the behavior of their PC & web versions).

These technicalities do not seem insurmountable though. And Google’s Chrome app shows just how well this can work. On my phone, the Chrome app runs in its regular minimalistic mode, but when running that same app on the big screen, it impressively transforms itself to look and behave like the PC desktop version of Chrome.

Software freedom

Endless is a strong proponent of Free and Open Source software being the right base for sustainable, scalable access to PCs. And as exciting as it feels to put my laptop to one side and switch to a PC experience produced by my smartphone, I faced a sinking feeling in this area: this is the first time in around 20 years that I’m not using an open source platform for the tasks I do on a PC. That feels like an unfortunate step backwards. While built on some open technologies, you cannot really consider the Samsung Galaxy Android OS to be a FOSS project, and the DeX functionality seems to be entirely proprietary.

That said, this does highlight an inconsistency in ideals. Why do FOSS proponents such as myself feel strongly about the need for open technologies on PC desktops, while simultaneously holding comparatively closed mainstream smartphones in our pockets?

Closing remarks

At the start of my 2-day experiment of using DeX instead of my PC, I had the feeling that it would fail. I had the perception that my laptop was an essential part of my work, I figured that a handheld smartphone would not be able to offer enough performance to replace a PC, and I expected the DeX experience to be sufficiently rough to have me craving the return to my laptop.

I was wrong on all these counts. I remain astounded that this experiment went as smoothly as it did. The DeX experience even surprised me with its smooth realization of the app-based PC desktop experience and the convenience of already holding all my apps and data. I did face minor inconveniences which left me considering if I should grab my laptop, but these could just be ordinary switching costs. The technical issues could likely be solved by developers without much effort. The overall experience seems so close to being a drop-in PC replacement.

From an Endless perspective, it’s hugely exciting progress in two of our focus areas. Firstly, the implication that a smartphone can replace a PC is hugely powerful, because smartphones are more affordable and much more widespread than PCs within the underserved communities we seek to benefit. Secondly, it delivers the app-based desktop experience in the PC-like environment, which is the same direction of travel that we have within Endless OS. The downside is that it’s a step backwards in terms of software freedom, but I am hopeful that open source implementations could emerge in future.

My biggest question: Why is this game-changing feature so hidden and unknown? Despite having so much relevance for our work at Endless, and with two DeX-capable smartphones at home, it still took 3 years for me to come across this – purely by accident.

Now that I’m aware of it, I see user discussions around DeX going back to 2018, and I see similar solutions such as Motorola Ready For and Huawei Easy Projection. There’s even the NexDock which uses DeX or similar to convert your smartphone into a laptop.

This feature delivers a compelling user experience, to the point where I will continue actively using it while reconsidering the importance of a traditional PC going forward.

Imperfections aside, this feature delivers a compelling user experience, to the point where I will continue actively using it while reconsidering the importance of a traditional PC going forward. As Matt Dalio writes in his accompanying article on this topic, we now need this feature to mature to the point where it is common in low-cost smartphones, and at that point, it could bring great positive impact to emerging markets where PC adoption is still in its infancy.

Chromebook app building discoveries

An Engineering Team in transition

Originally published at Endless OS Foundation

Endless previously operated as the for-profit entity Endless Mobile for around 7 years, before completing the transition to the nonprofit Endless OS Foundation in April 2020.

This transition did not involve a change of mission – even as a for-profit we were built around the social impact goals of empowering the world through technology – but the strategy and the approach to making our mark changed significantly.

It took us a good amount of head-scratching and going back to the drawing board to really grasp what it means to run a streamlined nonprofit organisation, especially while carrying around years of history from a different structure. I’d like to share some of the challenges faced by our engineering team during this transition, and show how we have adapted for our new organisational shape.

Breaking away from the for-profit model.

Previously, as a for-profit, we sought to build a thriving social enterprise where our success as a business would provide plentiful resources to further grow the impact and reach of our work. This did have an impact on our execution. For example, it entered us into the competitive platform game, where our business was reliant on people adopting Endless’ own software in order to bring possibilities for commercial growth (for example, by offering premium features or extra educational content at very low cost).

This led to our earlier focus on building Endless OS, our own Linux-based PC operating system, which provided exclusive access to our own education solutions and content. In some circumstances though, this all-inclusive product approach presented potential adopters with an awkward “us or them” question, where they could adopt either Endless’ platform or an alternative from another provider. Having to choose one solution would exclude them from receiving the benefits offered by the others. Ultimately, this aspect of our business model was holding back our ability to deliver social impact in some cases.

Our commercial business model was holding back our ability to deliver social impact in some cases. Now engineering impact is maximized when value is created that extends beyond our own platform.

Focusing our work on our mission & strategy.

Now as a nonprofit, our focus is not on achieving commercial success to sustain future growth. Instead, our prospects for success depend on delivering on our mission around the world, creating the maximum amount of impact for social good.

It took us a couple of rounds of discussions to grasp what this means in a comprehensive sense, with such conversations leading us to an entirely new strategy. In essence, we realised that it is no longer necessary for us to “win” at the platform game; in fact Endless’ impact can be maximized if the value created can extend beyond our own platform to a much wider audience.

With that in mind, we derived new initiatives around the 4 digital equity barriers that we wish to overcome:

Devices, where our innovations around making laptops more affordable increases access to PC ownership.
Connectivity, where we join forces with Kolibri and Flathub to deliver offline learning solutions.
Literacy, or the ability to use your PC to learn, where we collaborate with the GNOME ecosystem around improving free and open access to desktop computing.
Engagement, where our work increases the quantity and quality of free educational material available for offline use to everyone.

Our revised, nonprofit strategy opens up significant new opportunities for impact that we would have been unlikely to consider in our previous commercial structure. While we continue to believe in Free and Open Source personal computing as the most scalable way to extend the benefits of education & learning on PCs into all corners of the world, we can now also seek to benefit existing audiences of alternative platforms. One such example is our recent Endless Key initiative which specifically targets students with access to a Windows PC. Under our old for-profit mindset, our solutions would have been focused exclusively on our own platform instead of Windows.

The “discovery” channel of our Endless Key for offline Windows learners.

Now as a nonprofit, our team’s focus is on delivering our mission around the world via our work, creating the maximum amount of output for social good.

Community & transparency as an ingredient for growth of social impact.

Endless has always been a strong proponent of building software in open source community settings, but there was formerly a need to make some elements private in order to retain control over business opportunities. Now, as an impact-focused nonprofit, we have little reason to keep anything private. On the contrary, we want to enable and encourage others to take, adapt and build upon our work in any way that brings benefit to the world.

To go about this, we are working to provide full, open access to Endless OS’s code and surrounding infrastructure. As one step here, we recently released the Endless OS Image Builder which is the tool that we have used for years to offer tailor-made, custom versions of Endless OS for specific deployment contexts.

We’re also seeking to increase our transparency. On a technical level, we’ve taken steps here such as opening our Debian package repository and providing access to the latest, bleeding-edge Endless OS development version. We are currently looking at improving communication about our development direction and providing public access to our bug & task tracker.

We hope that our efforts towards increased transparency and access to our technologies will continue to support a growing community where we enjoy receiving independent impact deployment reports, development contributions, highly-skilled technical assistance, user guides, and more.

As an impact-focused nonprofit, we have little reason to keep anything private. On the contrary, we want to enable and encourage others to take, adapt and build upon our work in any way that brings benefit to the world.

Increasing engineering efficiency & streamlining product focus.

Being an effective nonprofit includes sustaining our team & efforts in the most efficient way possible, to maximize the resources that can be directly focused on executing our core mission.

Through our previous activities in a commercial context, where we had adapted and refined our software for use in specific environments and locations, we had accumulated a lot of miscellaneous features. In the engineering world, the existence of features carry a continued maintenance overhead, even if the aim is just to retain them in their existing state without developing further improvements.

At a fundamental level, we build upon a Free and Open Source software base, developed and supported by large worldwide communities. In some cases, our activities had led to us making modifications to some of these components. Just like product features, these deviations were generating a maintenance burden. We knew we had to reconsider the amount of effort put in these areas, and prioritizing our team’s time toward our new strategy became the lens for our re-evaluation.

We have maximized the engineering resources that directly link to executing our core mission.

With our efforts refocused around the specific impact areas detailed above, we took a step back to look at everything we had built, and decided to reduce and remove a significant amount of features and deviations that were not critical to achieving our mission.

We reduced our software to a simpler and leaner form, leaving only the key features that have a direct connection to our strategy and initiatives. This included removing integration with Android smartphones, our prototype offline news-feed for discovery, and our limited solution for managing websites and web links.
We dropped most of our modifications to Open Source software components, such as support for obscure hardware platforms no longer of relevance and our own printer driver manager.
We revised our processes for increased efficiency, including greatly simplifying the Endless OS build process to use Debian’s software repository directly, rather than building thousands of components.
We consolidated and reduced our server infrastructure and services, and for new initiatives we have been calling upon serverless solutions instead, which require less maintenance.

This far-reaching streamlining of our efforts turned out to be more work than anticipated, but the team recognised the sustainability gains at play, embraced the task, and everyone is feeling pleased with the results.

Collaboration and transparency with the larger open source community is a critical strategy at Endless.

Direct collaboration with upstream partners.

Combining the threads of maximizing impact, increasing sustainability and streamlining the software, we made some bold strategic shifts to work more closely with related technologies instead of creating our own solutions to some problems.

The most visible example is our handling of educational content. During the platform game years, Endless built an elaborate cloud-based system for ingesting content and generating offline content apps, exclusive to Endless OS. However, with our new strategy in mind, we made the difficult decision to turn that system off entirely and dedicate our educational content efforts towards Kolibri, a learning system produced by our partner Learning Equality.

Our users frequently praise the usability of our software, which we built from years of research amongst first-time PC users in emerging markets. To reduce the engineering cost of sustaining usability features, we have re-engaged the GNOME desktop computing community and contributed to usability research and redesign efforts that became part of GNOME 40. In the long term we hope to improve the GNOME desktop usability to the point where it is no longer necessary for Endless to maintain its own design; this would be a win for both sustainability and impact.

The Endless OS desktop, praised for its ease-of-use.

Above:

Combining the threads of increasing sustainability and streamlining software, we made some strategic shifts to work closely with related technologies and organizations — instead of creating our own solutions.

Getting closer to our users.

While transitioning to launch the Endless OS Foundation, we also identified the need to retain and improve our strong connections to our end users and the partners who take our work out into the world.

We experimented by creating a Deployment team within our Engineering division, responsible for the communication and technologies needed to support our partners. However, we later realised that this structure was creating distance between engineers and users, and moved to a structure where the responsibility for ensuring our software’s relevance for our users is something that explicitly belongs to each and every engineering team member.

Frequent user testing was one of the original pillars of Endless’ design and engineering efforts, and this must be kept amongst our regular engineering activities. We recently performed PC desktop usability testing on multiple continents, and tested our new Endless Key design and content with students across the US. Our engineers join these testing sessions, which aids them to relate better to our end users and make optimal engineering decisions.

Our software’s relevance for our users is a goal that explicitly belongs to each and every engineering team member.

There’s still work to be done after our team’s successful transition.

After years of operation under a different business structure, we faced many challenges in the transition to a nonprofit. However, after several demanding strategic discussions, we’ve been able to make significant changes to the structure and activities of our Engineering group directed towards greater social impact. These changes brought exciting progress through in 2021 and I’m excited about our team’s continued growth along these lines as we look towards a highly impactful future in the coming years.

Kolibri entirely in the web browser

Earlier this month, I participated in Endless Orange Week, a program where the entire Endless team engages in projects designed to grow our collective learning related to our skills, work and mission, while our ordinary work responsibilities and projects were put on pause.

I used the opportunity to do some experimentation around Kolibri, Endless’s chosen solution to provide offline access to educational content.

Especially with the proliferation of Chromebooks distributed by schools during the pandemic, there’s a clear opportunity for Kolibri to increase its impact by becoming available as a ChromeOS app. Kolibri can already be used on ChromeOS when there is a Kolibri server available over the network, however we are now focusing more on the independent learner case, providing access to educational content in limited-connectivity settings. We would like to make the entire Kolibri solution available to run on ChromeOS directly.

However, Kolibri has an advanced architecture due to its position as a highly featured learning environment, and that makes it challenging to bundle in a way that is compatible with constraints under which ChromeOS apps must operate. On a technical basis, Kolibri uses web technologies for the frontend, but relies on a Python backend.

I chose to experiment with one possible approach towards running Kolibri on ChromeOS: using modern web technologies to run the existing architecture largely as-is.

This experiment appealed to me as a way of gaining some hands-on experience with WebAssembly as part of the solution, an interesting technology that can vastly expand the capabilities of websites. In this case, I used Pyodide, a technology that lets you run Python code directly within the web browser.

Within the allocated week, I managed to get Kolibri up and running in this shape, at least to the point of the setup wizard. It was rewarding to see all this fit together. However, I did not have enough time to make it into a usable app, and there are some key challenges left largely unaddressed such as how we handle the storage of educational content.

I was also able to uncover some technical considerations that question the overall feasibility of this idea. Ultimately, while WebAssembly & Pyodide have truly achieved something magical in that Python code can be run in the web browser, there are fundamental challenges in bringing the web and Python worlds together and making them interoperate smoothly. To support complex apps like this without requiring rearchitecting Kolibri, Pyodide needs to bring these worlds together seamlessly and there are numerous complications in making that happen.

Code & technical notes
Click to see the experiment in action

I really enjoyed being able to focus some time on this experimental project and it was fun observing my peers also learning new things via their own Endless Orange Week projects at the same time.