Creation of Sidero Labs Omni System Extension

Introduction

I recently faced the task of deploying a Flutter application using Kubernetes on a Raspberry Pi 4, with the goal of displaying the application through the device’s available HDMI ports. Initially, this seemed pretty simple and straightforward, but when you start to dig, you dig up more interesting challenges.

Problem statement

Enable GPU access inside a Kubernetes pod running on a lightweight Kubernetes Linux distribution by incorporating and activating the GPU driver during installation, ensuring the GPU is usable on the host and within the pod.

Solution

Challenge 1 – GPU access within a Kubernetes pod

One particularly notable challenge was enabling GPU access within a Kubernetes pod. At first glance, this appeared to be a relatively simple task: identify the GPU’s path on the host system, mount it into the pod, and thereby allow the Flutter application to leverage the GPU’s capabilities to output through the Raspberry Pi 4’s HDMI port.

Through the investigation and testing process, I discovered that achieving this required configuring the pod to run in a privileged mode, a setting that grants elevated permissions to interact with the host’s hardware. Specifically, I needed to mount the /dev/dri directory, which corresponds to the Direct Rendering Infrastructure (DRI) on Linux systems and provides access to the GPU’s rendering capabilities, as a volume into the pod. This step was critical to exposing the GPU hardware to the containerized environment.

This approach was proven to work with Ubuntu and Kubernetes, where GPU driver support is robust. So on the Kubernetes side, the deployment is already sorted out.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: flutter-deployment
...
spec:
  ...
  spec:
    containers:
    - name: flutter-app
      ...
      image: registry.com/flutter-app:v1
      securityContext:
        privileged: true # Required for GPU and hardware access
      volumeMounts:
      - mountPath: /dev/dri # GPU rendering device
        name: dri
      # - mountPath: /dev/input # Input devices (touchscreen, mouse, etc.)
      # name: input
      ...
    volumes:
    - name: dri
      hostPath:
        path: /dev/dri # Host GPU device
    # - name: input
    # hostPath:
    # path: /dev/input # Host input devices

Challenge 2 – GPU access on Talos Linux

The next hurdle involved deploying the same Flutter application using Talos Linux from Sidero Labs, our preferred Kubernetes distribution. We favor Talos Linux for its streamlined, Kubernetes-centric design and minimal footprint. However, this lean approach introduced an obstacle: Talos Linux on the Raspberry Pi 4 currently lacks support for enabling the Broadcom VideoCore VI GPU. As a result, the /dev/dri for GPU access is unavailable, preventing the Flutter app from utilizing the GPU to render output through the HDMI ports.

We manage it with Sidero Labs Omni, a SaaS platform that simplifies cluster provisioning and upgrades, offering tailored Talos images for the Raspberry Pi 4. It is very easy to create a bootable Talos Linux installation specific for the Raspberry Pi 4 from within Sidero Labs Omni:

Upon booting the Talos Linux installation and onboarding it to Sidero Labs Omni, it became evident that /dev/dri was absent, indicating the GPU driver wasn’t enabled by default. I turned to Talos System Extensions to source a driver for the Broadcom VideoCore VI GPU on the Raspberry Pi 4, but no compatible driver was available.

Modifying the Linux kernel

The Raspberry Pi 4 features a Broadcom VideoCore VI GPU, which relies on specific Linux kernel drivers for hardware acceleration. According to the Gentoo Wiki, this GPU is supported by two key drivers in the Linux kernel: vc4 and v3d. The Linux kernel documentation explains that the vc4 driver handles both 2D and 3D rendering for the VideoCore IV and VI GPUs, interfacing with the DRI for display output (e.g., via HDMI). Meanwhile, the v3d driver, as detailed in its source code, focuses on 3D acceleration for newer Broadcom GPUs, including the VideoCore VI. Both drivers are part of the mainline Linux kernel, with their implementations found in the vc4 and v3d directories.

The vc4 driver’s configuration is further clarified in the kernel’s Kconfig file, which notes that it supports HDMI output and integrates with the kernel’s Direct Rendering Manager (DRM) subsystem. A Raspberry Pi forum thread also confirmed that enabling these drivers is a common requirement for GPU access on Raspberry Pi systems, often necessitating custom kernel builds. Additionally, kernel logs shared in a GitHub gist illustrate how v3d and vc4 initialize when properly configured, providing a useful reference for debugging.

Since Talos Linux is built from a custom Linux kernel, I turned to the Sidero Labs pkgs repository, which houses containerized packages used to construct Talos images. I forked this repository to explore and modify the setup for my needs. Among these packages is the kernel package, with its configuration for the ARM64 architecture specific to the Raspberry Pi 4 located at kernel/build/config-arm64. The ARM64 configuration is critical here, as the Raspberry Pi 4 uses a 64-bit ARM processor, requiring drivers compiled for this architecture. Upon researching I found that they have a file called vc4graphics.cfg within the Yocto Project. Reviewing the config file revealed that I needed to enable v3d, vc4 and snd components:

Note: Not all changes are shown, view the pull request for more details.

# kernel/build/config-arm64
CONFIG_DRM_V3D=m
CONFIG_DRM_VC4=m
...
CONFIG_SOUND=m
CONFIG_SND=m
...

These settings indicate that both v3d, vc4 and snd are configured as loadable kernel modules (m).

Then I compiled a custom kernel with the necessary modules enabled. I ran the following command to build the kernel for the ARM64 platform and push the resulting container image to my registry (in this case, GitHub Container Registry):

make kernel REGISTRY=ghcr.io USERNAME= TAG= PUSH=true PLATFORM=linux/arm64

This process was not quick, the kernel took approximately 5 hours on my system! Once completed, the kernel container image was successfully pushed to ghcr.io//kernel, ready for use in a custom Talos image. Inspecting the build output, I confirmed that the necessary kernel modules were included, such as the v3d and vc4 modules:

# v3d/vc4 Kernel module
/usr/lib/modules//kernel/drivers/gpu/drm/v3d/v3d.ko
/usr/lib/modules//kernel/drivers/gpu/drm/vc4/vc4.ko

Create System Extension

With the custom kernel built and the vc4 and v3d drivers included as kernel modules, the next step was to make these drivers available to Talos Linux on the Raspberry Pi 4. This is where Talos System Extensions come into play. System extensions are a powerful feature of Talos Linux, allowing users to extend the base operating system with additional drivers, tools, or services without modifying the core image. These extensions are packaged as container images and integrated into the Talos root filesystem at boot time, making them an ideal solution for adding GPU support.

To get started, I explored the official Sidero Labs extensions repository, which hosts a collection of pre-built system extensions for Talos. This repository includes extensions for various hardware drivers, such as amdgpu and i915 for AMD and Intel GPUs, as well as utilities like nfsd for NFS support. Each extension follows a standardized structure, consisting of a manifest.yaml file defining metadata and compatibility, and a rootfs directory containing the files (e.g., kernel modules or binaries) to be mounted into the Talos system and a pkgs.yaml file defining the build process.

Since no existing extension supported the Broadcom VideoCore VI GPU, I decided to create my own. I forked the siderolabs/extensions repository to my own fork and set out to build a custom extension for the vc4 and v3d drivers. My approach was inspired by existing extensions, particularly the nfsd extension for its simplicity in handling kernel modules, and the drm-related extensions like amdgpu for their focus on GPU drivers.

In my fork, I added a new directory under extensions/drm/ called vc4, where I defined the new system extension. The structure mirrors other extensions in the repository:

– manifest.yaml: This file provides metadata about the extension, including its name, version, author, and compatibility with Talos versions. I modeled it after the nfsd extension, ensuring it specified compatibility with the Talos version I was targeting (e.g., >= v1.10.0). Here’s an example of what I included:

version: v1alpha1
metadata:
  name: vc4
  version: "$VERSION"
  author: Gluo NV
  description: |
    This system extension provides kernel modules for Broadcom VideoCore GPU.
  compatibility:
    talos:
      version: ">= v1.10.0"

– pkg.yaml: Following the bldr tool’s conventions (used by Sidero Labs to build extensions), I created a pkg.yaml file to define the build process. I took cues from the nfsd extension’s pkg.yaml, specifying the source files and target directory structure.
– files/modules.txt: Defining the modules like v3d, vc4, … to include within this system extension.

With the extension structure in place, I built and pushed the extension as a container image to my GitHub Container Registry. From the root of my forked repository, I ran the following command:

make vc4 REGISTRY=ghcr.io USERNAME= TAG= PUSH=true PKG_KERNEL=ghcr//kernel: PLATFORM=linux/arm64

I’ve submitted a pull request to the Sidero Labs extensions repository, adding the vc4 system extension to enable Broadcom VideoCore VI GPU support for Talos Linux on the Raspberry Pi 4.

Raspberry Pi 4 specific – config.txt

The config.txt file is a critical configuration file used by the Raspberry Pi to define system settings during boot. Unlike traditional PCs that rely on a BIOS or UEFI, the Raspberry Pi uses this plain-text file, located in the boot partition of the SD card, to configure hardware parameters such as memory allocation, display settings, and device overlays before the operating system loads. This approach allows the Pi to adapt its firmware and kernel behavior to specific hardware needs, making it a flexible solution for embedded systems.

To ensure the Broadcom VideoCore VI GPU functions correctly on the Raspberry Pi 4 with Talos Linux, specific configurations must be applied to the config.txt file that a Pi uses to initialize its hardware. Since Talos Linux leverages system extensions and a custom kernel, these options need to be removed and appended as Extra overlay options (advanced) in the image creation. Here’s what I added:

configTxt: |
  gpu_mem=128
  kernel=u-boot.bin
  arm_64bit=1
  arm_boost=1
  enable_uart=1
  dtoverlay=disable-bt
  dtoverlay=disable-wifi
  avoid_warnings=2
  dtoverlay=vc4-kms-v3d,noaudio

– gpu_mem=128: Allocates 128 MB of memory to the GPU, ensuring it has enough resources for rendering tasks like outputting to HDMI.
– avoid_warnings=2: Was a requirement for the GPU to function.
– dtoverlay=vc4-kms-v3d: Loads the vc4-kms-v3d overlay, which enables the VideoCore IV and VI GPU drivers with Full KMS (Kernel Mode Setting) support, crucial for integrating with the vc4 and v3d kernel modules.

Create boot medium with config.txt options and vc4 system extension enabled

Using Sidero Labs Image Factory and Imager, I created a bootable Talos image for the Raspberry Pi 4 (taking the extra config.txt options into account), enabling the vc4 system extension, now merged into mainline, via the “System Extensions” option. I flashed the generated disk image onto an SD card, booted the Pi, and confirmed /dev/dri was available. Created a single node cluster and deployed the Flutter app. My Flutter app then ran with the GPU over HDMI!

Image Factory Schematic:

# rpi_generic.yaml
overlay:
  name: rpi_generic
  image: siderolabs/sbc-raspberrypi
  options:
    configTxt: |
      gpu_mem=128
      kernel=u-boot.bin
      arm_64bit=1
      arm_boost=1
      enable_uart=1
      dtoverlay=disable-bt
      dtoverlay=disable-wifi
      avoid_warnings=2
      dtoverlay=vc4-kms-v3d,noaudio
customization:
  systemExtensions:
    officialExtensions:
      - siderolabs/vc4

To ensure this solution is accessible to others, I submitted a pull request (Documentation PR) to the Talos documentation, detailing the steps to enable GPU support on the Raspberry Pi 4. The PR includes the Image Factory schematic and configuration specifics, making it easier for users to replicate this setup. This contribution enhances the official Talos docs, bridging the gap for Raspberry Pi users seeking GPU functionality.

Conclusion

Enabling GPU access for a Flutter app on a Raspberry Pi 4 with Talos Linux seemed challenging but became manageable by tackling it step-by-step. Unlike Ubuntu, Talos lean design required extra work to add Broadcom VideoCore VI support via a custom kernel and system extension. This modular solution showcased Talos flexibility, teaching me valuable lessons in kernel configuration and system extensibility for Kubernetes and embedded projects.

Menu