steinbro.github.io

Let's snap Spiel

2026-08-09T00:00:00+00:00

Spiel</a> is a proposed replacement for the aging speech-dispatcher service on Linux. It’s essentially the glue between applications that need to generate speech, like screen readers, and text-to-speech engines like Piper</a> that can generate it.</p>

One selling point of Spiel over speech-dispatcher is that its architecture is better suited with modern sandboxed apps like snaps. But as far as I can tell, Spiel’s distribution has been limited to flatpaks. Let’s fix that.</p>

Using snap interfaces</h3>

Spiel’s architecture puts TTS engines in their own processes, called speech providers, that communicate with client applications via D-Bus. When new speech providers are installed, any application using Spiel libraries can find it automatically using D-Bus service discovery.</p>

Snaps support this kind of interface through plugs and slots</a>. The speech provider snap defines a D-Bus service, and the snap’s executable entrypoint is configured to activate based on that connection. This means the speech provider daemon will only run when an application triggers it via a D-Bus message.</p>

# speech-provider-piper snap</span></span>
slots</span>:</span></span>
  speech-provider</span>:</span></span>
    interface</span>:</span> dbus</span></span>
    bus</span>:</span> session</span></span>
    name</span>:</span> ai.piper.Speech.Provider</span></span>
</span>
apps</span>:</span></span>
  speech-provider-piper</span>:</span></span>
    command</span>:</span> usr/local/bin/speech-provider-piper</span></span>
    daemon</span>:</span> simple</span></span>
    daemon-scope</span>:</span> user</span></span>
    activates-on</span>:</span> [</span>speech-provider</span>]</span></span></code></pre>
A client application like Orca would define a corresponding plug with the same service name:</p>
# orca snap</span></span>
plugs</span>:</span></span>
  speech-provider-piper</span>:</span></span>
    interface</span>:</span> dbus</span></span>
    bus</span>:</span> session</span></span>
    name</span>:</span> ai.piper.Speech.Provider</span></span></code></pre>
Once both snaps are installed, users can hook up the application to the speech provider manually:</p>
$ snap connect orca:speech-provider-piper speech-provider-piper:speech-provider</span></span></code></pre>Defining voice packs</h3>
There’s a third layer of packaging required here, beyond the speech providers and client applications: voices. TTS engines typically support many languages, with potentially many voice options for each. For a modern TTS engine like Piper, an installed voice can use hundreds of megabytes of disk space for the underlying AI model. It’s therefore best to let users choose which subset of voices they’d like to install.</p>
Snaps have a couple different concepts related to packaging content separately from an application. Content snaps</a> are published independently from the snaps they support, potentially even by a different author, and use interfaces to share resources. They’re most valuable when the packaged content is used by multiple snaps, such as GNOME runtimes</a>. Since the Piper voice models will only be used by the speech provider snap, this isn’t the case here.</p>
On the other hand, components</a> are designed to package optional parts of a single application. This is the technique used by other snaps that have selectively-installable ML models, such as gemma4</a>. Components are also much faster to build than separate content snaps, because they are all built in the same build container as the snap they support. Since our Piper speech provider will have dozens of supported languages and locales, this can make a big difference here.</p>
Defining components involves simply including a top-level components</code> key and a part that writes the desired data into the component’s directory. For Piper voices, we download the voice model and its metadata individually, because cloning the whole repository of voices would incur several gigabytes of network traffic and disk usage.</p>
# speech-provider-piper snap</span></span>
components</span>:</span></span>
  voices-en-gb-alan-medium</span>:</span></span>
    type</span>:</span> standard</span></span>
    summary</span>:</span> en-GB alan voice for Piper</span></span>
    description</span>:</span> en-GB alan voice for Piper</span></span>
    version</span>:</span> '0.1'</span></span>
</span>
parts</span>:</span></span>
  voices-en-gb-alan-medium</span>:</span></span>
    plugin</span>:</span> dump</span></span>
    source</span>:</span> ./component-base</span></span>
    build-packages</span>:</span></span>
      -</span> curl</span></span>
      -</span> ca-certificates</span></span>
    override-build</span>:</span> |</span></span>
      craftctl default</span></span>
      set -eu</span></span>
      mkdir -p "$CRAFT_PART_INSTALL/en_GB-alan-medium"</span></span>
      curl -L --fail --retry 3 \</span></span>
        -o "$CRAFT_PART_INSTALL/en_GB-alan-medium/en_GB-alan-medium.onnx" \</span></span>
        "https://huggingface.co/rhasspy/piper-voices/resolve/main/en/en_GB/alan/medium/en_GB-alan-medium.onnx"</span></span>
      curl -L --fail --retry 3 \</span></span>
        -o "$CRAFT_PART_INSTALL/en_GB-alan-medium/en_GB-alan-medium.onnx.json" \</span></span>
        "https://huggingface.co/rhasspy/piper-voices/resolve/main/en/en_GB/alan/medium/en_GB-alan-medium.onnx.json"</span></span>
    organize</span>:</span></span>
      en_GB-alan-medium/en_GB-alan-medium.onnx</span>:</span> (component/voices-en-gb-alan-medium)/en_GB-alan-medium.onnx</span></span>
      en_GB-alan-medium/en_GB-alan-medium.onnx.json</span>:</span> (component/voices-en-gb-alan-medium)/en_GB-alan-medium.onnx.json</span></span></code></pre>Next steps</h3>
The Orca screen reader has experimental support for Spiel</a>, and it could take advantage of these packages on systems that support snaps. Orca itself wouldn’t need to be packaged as a snap to do so, though having an Orca snap would make installation convenient. However, Orca’s unusual interaction patterns could make it tricky to run as a sandboxed app.</p>
You can find snapcraft.yaml build configuration files and testing scripts on GitHub</a>. Try it out, and add a few more languages and voices to the Piper speech provider while you’re at it.</p>



Audio annotations for the real world
2026-06-19T00:00:00+00:00
Augmenting reality with standard maps</h3>
Cartography dates back to a time when the most efficient way to describe a place was to draw it on paper. In more recent years, we’ve had the ability to layer map information on top of the real world, thanks to our smartphones. Your phone knows what’s around you because it knows your exact GPS location together with information about all nearby venues, at least those that have been captured in online maps.</p>
I remember this idea showing up as a feature on Windows Phone over a decade ago, through an app called Nokia City Lens. You could hold up your phone to a city street, and it would add floating labels on top of all the places it knew about.</p>

(source: WIRED</a>)</p>
Less well-publicized was a Microsoft research project called Soundscape, which  did this reality augmentation as audio rather than video. Instead of drawing labels on top of a camera view, your phone speaks markers as spatial audio. Even with basic headphones, you hear the labels as if they emanate from different places and directions.</p>
This turns out to be a pretty handy navigational aid, especially if you can’t easily read signs. Walking down the street with earbuds in and phone in your pocket, you hear the names of places around you as you approach them, with a decent sense of exactly where they are.</p>

    
    </iframe>
</div>Community-driven mobility tools</h3>
Nifty as this technology is, there wasn’t much commercial potential, and Microsoft discontinued Soundscape</a> in 2023. To satisfy the dedicated fanbase among low-vision users, they first dumped the source code</a> on GitHub, under an open-source license. After some initial concern as to whether anyone would step up to run the servers and republish the app, no fewer than three different groups picked it up. Each has found their own niche to serve.</p>
A solo developer published VoiceVista</a>, a closed-source fork that adds all sorts of bells and whistles, though it’s hard to tell exactly what’s been added due to limited documentation. Separately, a Scottish nonprofit published a long-requested Android version</a>, which was no small feat. And finally, there is Soundscape Community</a>, the initiative I helped get off the ground and spent several years as a volunteer developer. They’ve established an open source community of contributors and users, and have been finding new uses and collaborations in adaptive sports, among others</p>
What makes Soundscape special</h3>
It’s hard to pinpoint what makes the Soundscape app feel so empowering, compared to audio turn-by-turn directions that are commonplace. Perhaps it’s the strong sense of agency: Soundscape is not telling you where to go – you follow whatever path you want, and it dutifully speaks up about what’s around you, when it’s most relevant.</p>
Soundscape is also unusual assistive tech because it’s providing a visual aid without using any visual input. It relies entirely on GPS position and compass heading, not the camera. This makes it feel a bit like a spatial superpower. It can tell you the name of a park that has no marker, or the name of a street that has no sign. This makes it a natural complement to smart glasses that can describe a visual scene.</p>


An unobtrusive text-to-speech workflow in GNOME
2026-06-06T00:00:00+00:00
With assistive tech, software ergonomics matter. When the user relies on a capability all day, every day, it doesn’t just need to work; it needs to be frictionless. When that feature is text-to-speech, it should be available instantly, for any text on the screen. The operating system, or more specifically the desktop environment, can step in to provide it system-wide, across all applications. For Linux, this functionality (as with much else on Linux) is possible, with some tinkering.</p>
On-demand speech on other operating systems</h3>
macOS has had “Speak Selection” functionality</a> for ages. When actively speaking, the system shows a floating controller for playback and speech rate adjustment.
</p>
Microsoft similarly introduced speech features into Magnifier</a> in Windows 10. Clicking the “Read from here” button starts reading text aloud from wherever the user specifies.
</p>
The Reader View in Firefox</a> has text-to-speech built in as well.
</p>
Screen readers also have some features that are geared towards partially-sighted users. Both NVDA on Windows and VoiceOver on macOS support a mode that reads text under the mouse pointer. But on Linux, Orca’s Mouse Review feature works inconsistently under Wayland</a> the modern display system for Linux.</p>
Prior art on Linux</h3>
A 2021 blog post</a> demonstrates how to set up a keyboard shortcut to trigger text-to-speech on most any Linux desktop environment. The impressively short shell script simply grabs the highlighted text from the clipboard’s primary selection buffer, and feeds that directly into the espeak synthesizer.</p>
This works great much of the time, but you’ll run into many unhandled edge cases if you use it extensively. For example. text containing Unicode mathematical characters, a common social media trick for adding styling, can turn into a stream of arcane code point names. “𝐉𝐚𝐯𝐚𝐒𝐜𝐫𝐢𝐩𝐭” becomes “letter 1D409, letter 1D41A, …” Emoji fare even worse, often disappearing entirely from the spoken output.</p>
My favorite failure mode is when the script encounters text surrounded by double square brackets. The espeak synthesizer interprets what’s inside as raw phonetic data. The result is text that is a very exotic flavor of garbled. [[ If you happen to be using it now, this example will demonstrate exactly what I  mean,. ]]</p>
A native extension for GNOME</h3>
I’ve published a GNOME shell extension</a> that builds on the same basic approach to capturing and speaking selected text, but makes it a bit more robust to text encountered in the wild.</p>
</p>
The most visible improvement is its integration with the GNOME desktop, but the bigger change comes from inserting Speech Dispatcher between your text and the speech engine. This long-standing Linux service handles many of the failure modes we encountered earlier: it translates Unicode characters into something meaningful, announces emoji names instead of silently dropping them, and filters out control sequences that can trip up speech synthesizers.</p>
Because Speech Dispatcher operates as a system-wide service, it also improves the overall listening experience. Other applications won’t try to speak over the extension, and you can change your preferred voice or speaking rate without needing to edit the values hard-coded into the original script.</p>
There are still some limitations. Most significantly, the extension still relies on the primary selection buffer, a legacy feature of the Linux desktop that isn’t supported universally. For example, applications that are based on the Flutter toolkit, including the Ubuntu App Center and FluffyChat, don’t populate this buffer</a> when the user highlights text. That means the extension will ignore what you’ve highlighted in your Flutter-based app, and instead speak the last text highlighted in some other non-Flutter-based application.</p>
Nevertheless, give it a try and let me know if you find it helpful!</p>


Adapting magnifier controls for mouse-centric websites
2026-05-25T00:00:00+00:00
The problem</h3>
Modern websites love using hover effects. Pausing the mouse pointer over various parts of the page can trigger preview cards, floating menus, or something else entirely.</p>
For most users, this is fine and sometimes even helpful. If it’s unwanted, the user can just move the mouse elsewhere. But it poses an accessibility challenge for full-screen magnification users. Their visible area of the page, known as a viewport, will be much smaller than what most other users have.</p>
</p>
Many hover-heavy interfaces are built around a pair of assumptions:</p>

Moving the mouse over something means the user wants to interact with it more deeply.</li>
Users can comfortably look at one part of the screen while manipulating another.</li>
</ul>
For full-screen magnification users, neither assumption really holds. Moving the mouse is often just a way to pan the viewport or keep content visible, not an intentional request for more UI. And because only a small portion of the screen is visible at once, users frequently cannot see the area they are interacting with and the area where the resulting information appears at the same time.</p>
Obscured content</h4>
At high zoom levels, the visible portion of the screen is tiny, and the magnified area usually follows the mouse pointer. So when a website decides to reveal extra information on hover, that information can end up covering the very thing the user was trying to look at.</p>
</p>
The obvious solution, moving the mouse elsewhere, doesn’t really work. Moving the pointer also moves the zoomed viewport, which can push the content completely off-screen.</p>
Important information outside the viewport</h4>
In some cases, the problem is spatial rather than transient. An interactive graph, for example, might show numerical values in a fixed corner of the screen while the user moves the mouse across the graph. Under full-screen magnification, the user may be able to see the graph or the readout, but not both at the same time. Moving the mouse to explore the graph also moves the zoomed viewport, making it difficult or impossible to keep the changing values visible.</p>

  
  Your browser does not support the video tag.
</video>
The root problem is that, for magnification users, mouse movement is doing two completely different jobs at the same time. It’s being used both to indicate what they want to interact with, and to control which part of the screen they can see.</p>
The solution</h3>
Fortunately, these problems don’t require changes to every offending site. We can introduce some small tweaks on the user side to decouple pointer interaction from panning the viewport.</p>
In macOS, this kind of functionality is built-in. One of the “advanced options” of the full-screen magnifier</a> is “Detach zoom view from pointer.” When the user holds down the Control and Command keys, the mouse pointer is effectively frozen in place. Moving the mouse moves the zoomed area, but the focused application doesn’t register any mouse movement.</p>
Freezing the pointer</h4>
With some AI assistance, I’ve replicated this as a “freeze pointer” user script</a>, which can be installed in a Web browser through an extension like Tampermonkey</a>.</p>
Below, as illustrated earlier, the user is moving the mouse across a graph while the exact values appear in a fixed location outside the magnified area. With the frozen-pointer user script, however, the user can hold down the Control key to temporarily freeze the hover position on a single data point. This enables them to pan the magnified viewport until the readout is visible again, without changing which data point is selected under the mouse pointer.</p>

  
  Your browser does not support the video tag.
</video>
Shielding the page from interaction</h4>
I also created a second user script that temporarily adds an “invisible wall” between the pointer and the web page</a>. This takes a more aggressive, but also more reliable, approach: instead of freezing the current hover state in place, it prevents the page from receiving pointer interactions altogether while the modifier key is held. The downside is that it disables hover behavior entirely rather than preserving the currently displayed item or tooltip, so it’s less helpful in cases like the interactive graph.</p>

steinbro.github.io

Let's snap Spiel

Audio annotations for the real world

An unobtrusive text-to-speech workflow in GNOME

Adapting magnifier controls for mouse-centric websites

The solution</h3> Fortunately, these problems don’t require changes to every offending site. We can introduce some small tweaks on the user side to decouple pointer interaction from panning the viewport.</p>