Last year, he demoed turning your router into a full-fledged computer.

This year, it’s the Google Nexus One phone as a full computer — with attached external keyboard, mouse, display, and more.

The Nexus One demo is using the udlfb Linux kernel module to talk with the DisplayLink device, and it will work with any DisplayLink device, including Plugable’s.

Note the version of udlfb in the 2.6.34 staging tree unfortunately didn’t work for Sven. So he’s now using the latest udlfb from http://git.plugable.com/, which will likely get merged in for kernel 2.6.36.

Also, he used a dual headed cable to get enough power. A powered hub or a docking station/terminal like the UD-160-A won’t need that — it supplies its own power from AC – all the hardware needed is in the one package.

It’s exciting to have all this open source work come together in interesting demos like this.

There was a question recently why udlfb doesn’t use the same compression technique as the Windows drivers. Among other reasons, one is that the RL compression used by udlfb scales down to devices like the ones Sven has been working on — it’s as light as possible on CPU load, while getting decent compression.

Stepping back, it’s clear Apple (and now Microsoft) are making a mistake by limiting the hardware ecosystem around their devices. Android and the other Linux variants have an opportunity here — and considering the Apple juggernaut, they definitely need every advantage.

Sven’s demos show how powerful these scenarios can be. The hardware is ready. Devices like Plugable’s are designed with these scenarios in mind. Now we need to get the software refined and included in standard distributions, so normal consumers can take advantage of all the possibilities and benefits here.

We’re very excited about the Google funded project to refine USB multiseat on Linux, with the winning proposal from Lucas Nascimento Ferreira at the Federal University of Parana in Brazil.

In addition to providing mentoring, Plugable is also providing donated hardware for this project. Two Plugable Universal Docking stations with recent enhancements for use as a terminal, were just picked up by Lucas in Brazil today. By next month, we expect to have updated versions of the UGA-125-HUB for terminal use to send down.

For those interested in learning more and potentially following Lucas’ work:

• Google’s SoC 2010 Timeline
• Lucas’ winning USB Multiseat Proposal

And we’ll post periodic status updates here.

Linux hasn’t had that support — it would try to set the highest mode the monitor is capable of, often resulting in a black screen. Especially common for the DL-125 chip, with its mode limits of 1440×900/1280×1024.

That’s a shame because the DL-125 chip is a smart choice in many cases – by limiting itself to those lower modes, it stays more consistently within the limits of the USB 2.0 bus, resulting in more consistent performance.

So coinciding with the launch of Plugable’s DisplayLink DL-125 based products (UGA-125 and UGA-125-HUB), changes have been implemented to bring Linux roughly up to the level of Windows and Mac in this area for DisplayLink devices. This also helps devices like the UD-160-A when running on monitors greater than its limit of 1920×1080.

The kernel framebuffer driver udlfb has been enhanced to read the resolution limit from the firmware descriptors of the device, and adhere to it.

On the X server side, we needed a driver which would limit itself to the resulting reduced mode list. Unfortunately, the existing displaylink X server reads EDID directly, and assumes the adapter can do whatever the monitor can do.

We’ve been wanting to get rid of the need for a displaylink-specific X server, and the standard xf86-video-fbdev driver runs with the best existing mode, rather than trying to set a higher one in EDID. So this was a good trigger for converting over.

So xf86-video-fbdev has been enhanced with X Damage protocol support, ported from Roberto’s displaylink driver. This is still a little in-flux from an interface perspective, but from a functional perspective it’s done and fully performant.

So it’s now possible to run with a modified generic fbdev driver, which talks to udlfb, with full performance and without needing defio (although there’s also some good news in the defio space, which will be posted about later).

You can grab the latest udlfb kernel module with a “git clone http://git.plugable.com/webdav/udlfb”. Compile with “make && sudo make install && sudo depmod -a”

And you can grab the latest modified xf86-video-fbdev with a “git clone http://git.plugable.com/webdav/xf86-video-fbdev”. Compile with “./autogen.sh && make && sudo make install”

You’ll need a very recent xorg-macros version (1.4), which in package “sudo apt-get install xutils-dev”

To use the new X server, you must turn on the new “ReportDamage” option to fbdev. Modify your existing xorg conf like this:

Section "Device"
  Identifier "dl"
  Driver "fbdev"
  Option "ReportDamage" "true"
  Option "fbdev" "/dev/fb0"
EndSection

And you should be all set to go. This new X server should work with the existing udlfb in the staging tree of kernel 2.6.31+ for now, as it’s re-using the same original ioctl. But may require modeset changes that are only in 2.6.34+.

The main focus is USB multiseat, and the details are on the xorg wiki for SoC 2010

With all the pieces that are just coming together now, there is a potential here to do a project with huge impact, without a massive amount of engineering. There is already a very solid proposal coming from a student in Brazil who has previously been involved with the MDM multiseat project.

Google’s deadline for applications is now just a few days away: April 9th.

If there’s interest from other parties, we’ll get everyone talking – or there are other related proposals that we might be able to get in at the last minute. Foremost among those is conversion of the DisplayLink USB driver udlfb and matching X server from a fbdev driver to the KMS model.

Here’s the basics of the USB multiseat opportunity:

USB Multiseat Refinement

Linux Multiseat setups have potential to significantly reduce the cost of computing, but can be hard to configure. Some progress has been made on USB multiseat, where all components of the “terminal’ (display, keyboard, mouse, and more) are on USB, so configuration can be fully plug and play – you can just assume that all devices on the same USB hub constitute a terminal.

Some early prototypes of this are working (see http://plugable.com/2009/11/16/setting-up-usb-multiseat-with-displaylink-on-linux-gdm-up-to-2-20/). The underlying kernel drivers and X servers are largely in place.

But recent changes to the X Server, ConsoleKit, and other components open the possibility for a cleaner implementation.

This SoC project would constitute the refinement/creation of configuration scripts to enable a standard Linux or *nix computer to automatically launch additional seats when a USB terminal is plugged in

* udev rules to detect hubs/devices which should be collectively treated as terminals
* udev attributes to label the set of devices with a common seat id
* udev triggers for on-demand generation of the appropriate Xorg config files, to allow seats to coexist with the primary display/devices.
* ConsoleKit scripts to launch independent GDM/X sessions for each USB terminal seat
* InputClass rules to cause the primary X session to ignore multiseat-assigned devices, and the appropriate seat to use them
* udev rules and X init scripts to grant access to audio, storage, and other devices to the person logged into the matching seat

The one-sentence goal of this project: To make USB multiseat fully plug and play for the end-user, and ready for any distro to safely and cleanly drop in at any time.

Know any aspiring software engineering students that might be interested? Check SoC info from Google and the SoC guidelines from xorg.

There will still be some bugs and issues to resolve — but this release has most of the features, specifically related to input handling and udev-based dynamic configuration, of a more solid USB multiseat solution.

In short:

• udev rules, which already can detect a grouping of USB devices that constitute a terminal, can now tell X to ignore or use particular devices more easily through udev attributes and more dynamic/independent xorg.conf.d scripts.
• udev is now the default configuration mechanism (the torch has been passed from the less flexible hal)

So the challenges to explore are now more in the area of ConsoleKit integration and whether we can use these new capabilities support not just pure-USB multiseat setups, but also a mix of seats with PCIe attached graphics, and with fully USB seats.

You can read more about the new 1.8 configuration features which multiseat will use at Peter Hutterer’s blog.

And read more about the RC itself at Phoronix

Add the following line to the bottom of the /lib/udev/rules.d/50-usbseat.rules file created per the previous instructions.

KERNEL=="control*", SUBSYSTEM=="sound", SUBSYSTEMS=="usb", PROGRAM="/bin/cat /sys/%p/../../../../../devnum", SYMLINK+="usbseat/%c/sound"

Then create a new /etc/X11/Xsession.d/50usbseat file which will be run at Xsession create time, with the following contents

oldIFS=$IFS
IFS=:
set $DISPLAY
IFS=.
set $2
SEAT_ID=$1
LN=`ls -al /dev/usbseat/$SEAT_ID/sound`
IFS=C
set $LN
CARD_ID=$2
export ALSA_CARD=$2
export ALSA_PCM_CARD=$2
IFS=$oldIFS

Each of the users who might need access to the USB devices needs to be added to the ‘audio’ group. On Ubuntu 9.04, this can be done with these commands to backup and then modify the groups (replace MY_USERNAME, of course) …

sudo cp /etc/group /etc/group_backup
sudo chmod a-wx /etc/group_backup
sudo adduser MY_USERNAME audio

See Ubuntu Sound TroubleShooting for details on that step.

Now, as you connect UD-160-A terminals, a new X instance and GDM login will pop up as before, but also each of them will have /dev/usbseat/%SEAT_ID%/sound linking to their sound device, and the ALSA_CARD environment variable for all processes off of that X session, set to the matching sound card ID. For apps which support ALSA/Pulse (like most browsers, flash, etc.), audio will now come out the correct terminal — all in a completely plug-and-play fashion.

If you’re wondering what the strange IFS stuff is in the above script, it’s bash’s built-in Internal Field Separator variable, which is an easy way to split strings without having to launch a separate sed or awk process.

Note, as before, these instructions are specific to and tested on an older version of Ubuntu: 9.04, and may need to be ported to other distros until the distros themselves integrate these scripts.

Thanks to Alexander Todorov’s earlier work on multiseat sound support, which demonstrated how to match the USB audio devices in udev, and which ALSA_ environment variables to set. Alexander reported some problems reliably matching the audio devices, but with these scripts (with limited testing so far), things are working as expected.

DisplayLink’s new thin client materials also provide a bunch of additional detail, with good bullet points on why simple USB connectivity provides real value for this scenario.

Companies like Sun, nComputing, Miniframe, and Userful have sold products for Windows and Unix in this space for some time. The unix world has had various forms of multiseat capability forever — not to speak of traditional X terminals and all the thin client network protocols like RDP, ICA, and VNC on Windows and Unix, which people think of first when they think of terminals.

But when it comes to true multiseat (locally connected devices, treated as terminals), Linux has had it first, has lots of advantages in terms of licensing cost and performance, has gotten hundreds of thousands of seats in use — and yet hasn’t quite make it easy enough for end users for it to reach its full audience.

So these new product launches, now with Microsoft on board, definitely have the ability to raise more awareness of the potential of this scenario. It’s a challenge to the Unix and Linux crowd to leverage their built in advantages — and continue working on the shortcomings of Linux as a client desktop — in order to continue to provide compelling competition in this market.

Either way, the additional exposure is great for communicating the potential of multiseat.

Until this is in the distros, we have to assemble a few pieces ourselves to get everything installed for this scenario. Some major components (GDM/ConsoleKit) are changing the way this kind of thing will be done, so for now the instructions here are designed only for distributions with GDM 2.20 or earlier (Ubuntu < 9.10; Fedora < F12; Debian < 5.0; etc). The instructions have been tested most extensively on Ubuntu 9.04.

gdm –version

will show what version you have on your current system. Any tweaks to make things work your specific distro are definitely welcome in the comments.

The first step is hardware. You need a USB hub with a DisplayLink USB device and free USB ports for keyboard and mouse. You can get this in one package in the form of devices like the Plugable Universal Docking Station, or buy independent devices like the UGA-125-HUB bundle. USB Audio and other devices can be made to work also, but these instructions just cover basic display, keyboard, and mouse.

With that, you're ready to start configuring your Linux setup ...

1. DisplayLink framebuffer driver

udlfb is in the staging tree of Linux kernels 2.6.32 and later. We'll install the latest version here. First, make sure you have git installed with "sudo apt-get install git-core", and create a directory which will host all the source code you download with git (e.g. ~/git/).

sudo apt-get install module-assistant
sudo module-assistant prepare
git clone http://git.plugable.com/webdav/udlfb/
cd udlfb
make
sudo make install
sudo depmod -a

Now, when you plug in a DisplayLink device, you should see a "green screen" as the driver successfully loads and sets the graphics mode to match your monitor.

2. DisplayLink X server

This will get the X server installed, ready for use by later scripts.

sudo apt-get install pkg-config xorg-dev
cd ~git
git clone http://git.plugable.com/webdav/xf-video-udlfb/
cd xf-video-udlfb
./configure
make
sudo make install

We now need to create or modify a few scripts and configuration files. You'll need to use sudo to edit files in these system directories. You can cut/paste the text below, or download the files with "git clone http://git.plugable.com/webdav/misc-udlfb/" and copy each to the right location, and fix up ownership and permissions on the files.

3. udev script

Create a file called /lib/udev/rules.d/50-usbseat.rules owned by user root, with the following contents.

# set all DisplayLink devices to configuration 1
# see http://libdlo.freedesktop.org/wiki/DeviceQuirks for more info
ATTR{idVendor}=="17e9", ATTR{bConfigurationValue}=="2", RUN="/bin/echo 1 > /sys%p/bConfigurationValue"
 
# aliases for display, kbd, mouse attached to specific hubs
 
KERNEL=="fb*",SUBSYSTEMS=="usb",PROGRAM="/bin/cat /sys/%p/../../../devnum",SYMLINK+="usbseat/%c/display",RUN+="usbseat.sh %c"
KERNEL=="mouse*", SUBSYSTEMS=="usb", ATTRS{bInterfaceClass}=="03", ATTRS{bInterfaceProtocol}=="02", PROGRAM="/bin/cat /sys/%p/../../../../../devnum",SYMLINK+="usbseat/%c/mouse",RUN+="usbseat.sh %c"
KERNEL=="event*", SUBSYSTEM=="input", ATTRS{bInterfaceClass}=="03", ATTRS{bInterfaceProtocol}=="01",PROGRAM="/bin/cat /sys/%p/../../../../../devnum",SYMLINK+="usbseat/%c/keyboard",RUN+="usbseat.sh %c"
KERNEL=="control*", SUBSYSTEM=="sound", SUBSYSTEMS=="usb", PROGRAM="/bin/cat /sys/%p/../../../../../devnum", SYMLINK+="usbseat/%c/sound"
 
# Handle when keyboard and mouse are one more hub downstream. Relying on pnp order to have already set up mouse, keyboard on upstream hub if we're daisy-chaining
KERNEL=="event*", SUBSYSTEM=="input", ATTRS{bInterfaceClass}=="03", ATTRS{bInterfaceProtocol}=="01",PROGRAM="/bin/cat /sys/%p/../../../../../../devnum",SYMLINK+="usbseat/%c/keyboard",RUN+="usbseat.sh %c"
KERNEL=="mouse*", SUBSYSTEMS=="usb", ATTRS{bInterfaceClass}=="03", ATTRS{bInterfaceProtocol}=="02", PROGRAM="/bin/cat /sys/%p/../../../../../../devnum",SYMLINK+="usbseat/%c/mouse",RUN+="usbseat.sh %c"

The udev subsystem will run this script automatically each time a USB device is attached.

4. usbseat.sh script

Create a file /lib/udev/usbseat.sh owned by user root with the following contents. Make sure to "sudo chmod a+x" to make the file executable.

#!/bin/bash
# takes the "seat number" as parameter $1
# the seat number is the kernel device id of the hub the seat's devices are sitting off of
# called once for every usb device that MIGHT be part of a seat, when they arrive or remove 
 
if [[ !(-n `/bin/pidof gdm`) ]]; then
    exit 0
fi
 
seat_running=`/usr/bin/gdmdynamic -l | /bin/sed -n -e "/:$1,/p"`
 
# $ACTION environment variable is set by udev subsystem
case "$ACTION" in
	'remove')
		if [[ -n "{$seat_running}" ]]; then
			/usr/bin/gdmdynamic -v -d $1
		fi
		;;
	*)
                # A device which might be part of a seat has been added
 
		# if we already have a running seat for this #, exit
		if [[ -n "${seat_running}" ]]; then
			exit 0
		fi
 
		if [[ -e /dev/usbseat/$1/keyboard && -e /dev/usbseat/$1/mouse && -e /dev/usbseat/$1/display ]]; then
 
			# We have a newly complete seat. Start it.
			TMPFILE=`/bin/mktemp` || exit 1
			/bin/sed "s/%ID_SEAT%/$1/g" < /lib/udev/usbseat-xf86.conf.sed > $TMPFILE
			/usr/bin/gdmdynamic -v -t 2 -s 1 -a "$1=/usr/X11R6/bin/X -br :$1 -audit 0 -nolisten tcp vt07 -config $TMPFILE"
			/usr/bin/gdmdynamic -v -r $1
		fi
		;;
esac
 
exit 0

5. X config file

Create file /lib/udev/usbseat-xf86.conf.sed with contents

 
Section "ServerFlags"
	Option	"AutoEnableDevices"	"false"
	Option	"AutoAddDevices"	"false"
	Option  "DefaultLayout"		"seat"
	Option	"DontZoom"		"true"
	Option	"DontZap"		"true"
	Option	"AllowMouseOpenFail"	"yes"
EndSection
 
Section "Module"
	Load "ddc"
EndSection
 
Section "Files"                                                                                                              
    ModulePath      "/usr/lib/xorg/modules"
    ModulePath      "/usr/local/lib/xorg/modules"
EndSection
 
Section "Device"
	Identifier "dl"
	driver	   "displaylink"
	Option "fbdev"	"/dev/usbseat/%ID_SEAT%/display"
EndSection
 
Section "InputDevice"
	Identifier "keyboard"
	Driver	"evdev"
	Option	"CoreKeyboard"
	Option	"Device"	"/dev/usbseat/%ID_SEAT%/keyboard"
	Option	"XkbModel"	"evdev"
	Option	"XkbLayout"	"us"
EndSection
 
Section "InputDevice"
	Identifier "mouse"
	Driver	"mouse"
	Option	"CorePointer"
	Option	"Protocol" "ImPS/2"
	Option	"Device"	"/dev/usbseat/%ID_SEAT%/mouse"
        Option  "Buttons" "5"
	Option	"ZAxisMapping" "4 5"
EndSection
 
Section "Monitor"
	Identifier "monitor"
EndSection
 
Section "Screen"
	Identifier "screen"
	Device "dl"
	Monitor "monitor"
EndSection
 
Section "ServerLayout"
	Identifier "seat"
	Screen	0 "screen" 0 0 
	InputDevice "keyboard" "CoreKeyboard"
	InputDevice "mouse" "CorePointer"
EndSection

6. fbcon workaround

fbcon is a standard Linux kernel module, which aggressively assumes it can open any framebuffer device and take it over for use as a text terminal. Unfortunately, that's not what we want if we're going to be using that framebuffer to run its own X server. So we need to add a file /etc/modprobe.d/fbcon.conf to disable fbcon and leave our framebuffers free for other uses.

blacklist font
blacklist tileblit
blacklist bitblit
blacklist fbcon

This file will not take effect until you run

sudo update-initramfs -u

Now, when you reboot and run "lsmod" you should not see fbcon in the loaded modules. And in /sys/class/graphics you should see fb0, instead of fbcon.

7. xrandr workaround

The DisplayLink X server currently has limited RANDR support, but later versions of GDM assume better. So for the time being, a workaround is required to get GDM applications (including gdmlogin) to display properly within the actual screen area -- otherwise they tend to think the screen has a strange rotation, and display themselves completely off it.

Add these lines into /etc/gdm/Init/Default, right after the definition of gdmwhich()

XRANDR=`gdmwhich xrandr`
if [ "x$XRANDR" != "x" ]; then
  $XRANDR -o 0
fi

8. /etc/rc.local script

Add the following lines to your /etc/rc.local script to check for attached usb terminals that were attached (at boot), and udev found them before GDM was running.

oldIFS=$IFS
IFS=/
for seat in /dev/usbseat/*; do
	set $seat
	/lib/udev/usbseat.sh $4
done
IFS=$oldIFS

9. /etc/init.d/gdm patch

In recent versions of X, the system largely assumes that you'll only run one X server, and it will own all devices. So to support multiseat with multiple X servers easily, we need to have two configurations: normal (using only your primary graphics), and multiseat (where your primary graphics isn't used). We do this by detecting whether you have a USB terminal attached and configured, and if so coming up with a different gdm configuration.

Add these lines to your /etc/init.d/gdm script, just after the section to "Allow cdd to override the config" (around line 35).

# Allow usbseat to override the config
if [ -f /etc/gdm/gdm-usbseat.conf ]; then
	for usbseat in /dev/usbseat/*; do
		seatid=${usbseat##*/}
		if [ -e "/dev/usbseat/$seatid/keyboard" -a -e "/dev/usbseat/$seatid/mouse" -a -e "/dev/usbseat/$seatid/display" ]; then
			CONFIG_FILE="--config=/etc/gdm/gdm-usbseat.conf"
		fi
	done
fi

10. Create /etc/gdm/gdm-usbseat.conf

This is the alternative gdm.conf that will be used when a USB terminal is present at boot:

[daemon]
DynamicXServers=true
FlexibleXServers=0
Greeter=/usr/lib/gdm/gdmgreeter
 
[security]
 
[xdmcp]
 
[gui]
 
[greeter]
 
[chooser]
 
[debug]
 
[servers]
0=inactive

Now, when you boot with USB terminal(s) attached, graphical logins will come up on all of those, while your primary display will remain a text console.

See my video from Linux Plumbers 2009 for more thoughts and background on USB multiseat.

Unsolved Problems

Beyond reworking for the new GDM/ConsoleKit versions, and getting all this just "built in" to the distributions, there are at least several major problems yet:

input duplication

[Update Feb 19, 2010 - fixed by removing "-sharevts -novtswitch" from the X start line and substituting a specific vt "vt07". Listing above now has problem resolved]

Input to any of the terminals will be duplicated to your primary console. This has a bunch of very nasty effects, including often duplicating your login to both your session and the console, and possibly causing strange effects like ctrl-c within any terminal will often cause that whole X session to close. This seems to be http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=525736, which at the bottom reports a workaround involving faketty, which unfortunately didn't work for me. You can also see a short discussion of this bug at https://fedorahosted.org/pipermail/multiseat/2009-April/date.html There ought to be a simple solution here -- any comments on that welcome.

mouse scroll wheel doesn't work

[Update Feb 19, 2010 - fixed by changing Protocol from "auto" to "ImPS/2" in the xorg.conf]

This is a problem with some other ways of doing multiseat: https://fedorahosted.org/pipermail/multiseat/2009-February/000004.html Again, probably a small tweak might resolve this.

unplug terminal doesn't tear down (or reconnect)
Currently, when you physically disconnect a terminal, everything remains running. By default, each user can only be logged in once, which means that you must do gdmdynamic -l to find the zombie instance and sudo gdmdynamic -d $seat to kill it. But the behavior users would really like to see is disconnecting the terminal leaves all apps running, but logging in from that terminal (or any other) reconnects and just transparently starts where the user left off.

Comments on other distros (e.g. yum vs. apt-get), better solutions, and other problems are very much appreciated.

See Microsoft’s “Unlimited Potential” blog for more information from Microsoft.

“Unlimited” is over the top, but there’s certainly unfulfilled potential for multiseat (my talk on this), so raising awareness of the opportunity to lower the cost of computing this way is a good thing, whether you care about both Windows and Linux, or just one. Clusters of computing terminals that are cheaper to buy and much less expensive to maintain is a great thing for education and many other scenarios.

Let’s all make it happen.