Purely Functional Operating Systems

Apparently Peter Henderson’s 1982 paper “Purely Functional Operating Systems” is hard to find online. Some years ago, during my PhD research, I scanned the paper from a physical copy in the university library. Here’s the scan I made.

Henderson, Peter. “Purely Functional Operating Systems.” In Functional Programming and Its Applications, edited by J. Darlington, P. Henderson, and D. Turner, 177–92. Cambridge University Press, 1982.

~60x speed-up of Linux "perf"

I’ve recently been using cargo-flamegraph to profile syndicate-server.

The tool is great, but it uses perf to record and analyze profile data, and perf on Debian has a performance problem: when not linked against libbfd, it shells out to addr2line for every address it needs to look up. Thousands and thousands and thousands of incredibly short-lived processes.

Michał Sidor suggests building against libbfd, something that the Debian maintainers aren’t allowed to do.1 I tried the other approach, suggested by Steinar H. Gunderson, of patching perf to use a long-running addr2line process instead, sending queries to it over a pipe.

This works very well. What used to take endless minutes now takes a few seconds. It makes working with cargo-flamegraph much more pleasant!

With the unpatched, debian-default perf_5.10, running on about 10 seconds of activity in syndicate-server:

$ time /usr/bin/perf_5.10 script -i perf.data >/dev/null

real    12m51.499s
user    11m57.455s
sys     0m53.821s

With my patch:

$ time perf script -i perf.data >/dev/null

real    0m11.335s
user    0m11.047s
sys     0m0.309s

That’s sixty-eight times faster.

You can download the patch here.


  1. The problem is an unfortunate incompatibility of licenses: perf is GPLv2, not GPLv2+, and libbfd is GPLv3+. 

New System-layer Syndicate project

Part of a series: #squeak-phone

I’m proud to announce a new project on using the Syndicated Actor model and dataspaces to implement a system layer.


The project will be investigating the question

Could dataspaces be a suitable system layer foundation, perhaps replacing software like systemd and D-Bus?

Concretely, it will produce a prototype system layer implementation building on my existing “squeak-on-a-cellphone” prototyping work.

I’m very grateful for project sponsorship from the NLnet Foundation, as part of the NGI Zero PET programme.

New Syndicate website

I’ve just published a completely revamped and massively expanded Syndicate project website:



A big part of the change is to fix the confusing terminology used in the project. From now on, I’ll try to stick to the following:

  • Syndicated actors, and the Syndicated Actor model: Like regular actors, but with replicated state, i.e. assertions connecting peers rather than just messages.

  • The notion of a dataspace: A particular kind of behaviour of a syndicated actor: tracks assertions made to it, and reacts to assertions of interest by responding with previously-learned facts.

  • Structuring of a syndicated actor via conversational concurrency and facets: Programming actors, syndicated or not, gets to be a bit of a mess if you’re restricted to using a monolithic behaviour function. Facets let you split up an actor’s behaviour into reusable composable pieces. Conversational concurrency guides you in how to carve it up.

  • Language support for syndicated actors and conversational concurrency: the Syndicate DSL.

OnScreenKeyboardMorph: Smalltalk keyboard on a phone

Part of a series: #squeak-phone

Back in October 2020, I built an on-screen keyboard for Squeak Smalltalk. It ended up being about 230 lines of code (!) in total.

I’ve been using Smalltalk as the primary UI for my experimental cellphone software stack, and I needed a way to type input. Using VNC to develop on the phone works fine, but to work on the device itself - both for day-to-day phone tasks as well as developing the system - I need a proper on-device keyboard.

This video shows the keyboard in action. As you can see, it’s not perfect! At one point I tapped ctrl while typing, leading to an unexpected pop-up menu appearing.

But the basic idea is sound. (Aside: why aren’t there any Android keyboards that are laid out like a PC keyboard, with punctuation in the right place, etc.?)

The usual shortcuts like Alt-P for “print it”, Alt-D for “do it” and Alt-I for “inspect it” all work for evaluating snippets of Smalltalk interactively.

The keyboard integrates with my previously-written touchscreen support code, with the red/blue/yellow modifier buttons affecting touches, making them into simulated left-/middle-/right-clicks, and with keyboard focus changes auto-popping-up and -down the keyboard.

Simulating mouse clicks is a temporary state of affairs that lets me use the touchscreen reasonably naturally, making use of context menus and so on, without having to make invasive changes to the input pipeline of Squeak’s Morphic.

How it works

Class OnScreenKeyboardMorph synthesizes keyboard events and injects them into the Morphic world.

OnScreenKeyboardMorph >> emitKeystroke: aCharacter
    | evt |
    evt := KeyboardEvent new
        setType: #keystroke
        buttons: modifiers
        position: 0@0
        keyValue: aCharacter asciiValue
        hand: ActiveHand
        stamp: Time millisecondClockValue.
    self resetModifiers.
    ActiveHand handleEvent: evt.

You can have arbitrarily many keyboards instantiated, but there’s a global one living in a flap at the bottom of the screen. That’s the blue tab at the bottom left of the screen you can see in the video.

OnScreenKeyboardMorph class >> rebuildFlap
    | f k |
    self flap ifNotNil: [:old |
        Flaps removeFlapTab: old keepInList: false.
        ActiveWorld reformulateUpdatingMenus].

    k := self new.
    f := FlapTab new referent: k beSticky.
    f setName: self flapId edge: #bottom color: Color blue lighter.
    k beFlap: true.

    Flaps addGlobalFlap: f.
    ActiveWorld addGlobalFlaps.
    ActiveWorld reformulateUpdatingMenus.

I had already written code to read from /dev/input/eventN, tracking each multitouch contact separately. A subclass of HandMorph overrides newKeyboardFocus: to pop the keyboard up and down as the keyboard focus comes and goes:

LinuxInputHandMorph >> newKeyboardFocus: aMorphOrNil
        ifNil: [OnScreenKeyboardMorph hideFlap]
        ifNotNil: [(OnScreenKeyboardMorph future: 200) raiseFlap].
    ^ super newKeyboardFocus: aMorphOrNil.

Each key is represented by an OnScreenKeyMorph. It tracks keyboard shift state by registering as a dependent of the OnScreenKeyboardMorph:

OnScreenKeyMorph >> keyboard: anOnScreenKeyboardMorph
    keyboard := anOnScreenKeyboardMorph.
    keyboard addDependent: self.

OnScreenKeyMorph >> update: aParameter
    aParameter = #keyboardShiftState ifTrue: [^ self recomputeLabel].
    super update: aParameter.

The important part, of course, is the mouse event handler for OnScreenKeyMorph:

OnScreenKeyMorph >> mouseDown: evt
        ifNil: [keyboard emitKeystroke: self activeLabel first]
        ifNotNil: [keyboard perform: selector]

The only other interesting part is the keyboard initialization routine, which builds the layout:

OnScreenKeyboardMorph >> initialize
    super initialize.
    self borderWidth: 0.
    self extent: 180 points @ 120 points.
    self layoutPolicy: TableLayout new.
    self wantsHaloFromClick: false.

    font := (TextStyle named: #Roboto) fontOfPointSize: 18.
    modifiers := 0.
    stickyModifiers := 0.
    buttons := Dictionary new.

        '`~ 1! 2@ 3# 4$ 5% 6^ 7& 8* 9( 0) -_ =+' substrings, {'<-' -> #backspace}.
        {'->' -> #tab}, 'qQ wW eE rR tT yY uU iI oO pP [{ ]}' substrings.
        {'esc' -> #escape}, 'aA sS dD fF gG hH jJ kK lL ;: ''" \|' substrings, {#cr}.
        {#shift}, 'zZ xX cC vV bB nN mM ,< .> /?' substrings.
        {#ctrl. #space. #alt}.
        {'|<-' -> #home. '<' -> #arrowLeft. 'v' -> #arrowDown. '^' -> #arrowUp. '>' -> #arrowRight. '->|' -> #end.}.
    } do: [:r | self addMorphBack: (self makeRow: (r collect: [:item | self makeButton: item]))].
    self addMorphBack:
        (self makeRow:
                OnScreenMouseButtonMorph new bit: 4.
                OnScreenMouseButtonMorph new bit: 2.
                OnScreenMouseButtonMorph new bit: 1.

The rest is book-keeping and simple delegation methods, like this:

OnScreenKeyboardMorph >> shift
    self toggleButton: 8

OnScreenKeyboardMorph >> space
    self emitKeystroke: Character space

and so on.

I haven’t released the whole package yet, because it’s all very much in flux, but if you’re interested, you can take a look at the changeset for the Keyboard-related code here: LinuxIO-Morphic-KeyboardMorph.st

TypeScript: Messages from Interfaces and back

UPDATE: Full code available at this gist (also embedded below).

Say you have the following TypeScript interface I that you want to invoke remotely by passing messages of type M; or that you receive messages of type M and want to handle them using an object of type I:

interface I {
    m1(a: string, b: number): boolean;
    m2(): void;
    m3(n: number): void;
    m4(x: [string, string]): { k: string, j: string };
    m5(a: string, b: string[]): number;

type M = { selector: "m1", args: [string, number], callback: (result: boolean) => void }
       | { selector: "m2", args: [], callback: (result: void) => void }
       | { selector: "m3", args: [number], callback: (result: void) => void }
       | { selector: "m4", args: [[string, string]], callback: (result: { k: string; j: string }) => void }
       | { selector: "m5", args: [string, string[]], callback: (result: number) => void }

Keeping things type-safe looks really tedious! There’s obviously a connection between I and M. Can we avoid writing them by hand?

Can we derive M from I? Can we derive I from M?

The answer to all of these questions is yes!1

TL;DR TypeScript lets us define generic types Messages and Methods such that M = Messages<I> and I = Methods<M>. Read on for the details.

Interface ⟶ Messages

Let’s start with what, for me, has been the common case: given an interface type, automatically produce the type of messages that can be sent to implementors of the interface.

First, how do we want to represent messages?

type Message<Selector extends ValidSelector, Args extends any[], Result> =
    Args extends never[]
    ? { selector: Selector, args: [], callback: (result: Result) => void }
    : { selector: Selector, args: Args, callback: (result: Result) => void }

type ValidSelector = string | number | symbol

I’ve taken a leaf out of Smalltalk’s book, and made a message include a selector, the name of the method the message intends to invoke, and some args, the provided arguments to the method. The Args extends never[] check is to help type inference deduce the empty argument tuple: without it, the type system won’t complain about missing arguments.

I’ve also added a callback to Message. The technique I describe here can be further extended to “asynchronous” or callbackless settings with minor modifications.

The next definition, of type Messages<I>, is where the magic happens. It expands to a union of Messages representing the methods defined in I:

type Messages<I> = MessagesProduct<I>[keyof I]
type MessagesProduct<I> = {
    [K in keyof I]: (I[K] extends (...args: infer P) => infer Q ? Message<K, P, Q> : never);

And that’s it! Here’s how it works:

  • MessagesProduct is a mapped type that describes a modified interface, where all (and only) the method properties of interface I are rewritten to have a Message as their type, but keeping the same key;

  • then, the ...[keyof I] part in the definition of Messages uses index types to set up a union type built from all the value types (“indexed access operator”) associated with all the keys in I (“index type query operator”).

Messages ⟶ Interface

Going in the other direction is simpler:

type Methods<M extends { selector: ValidSelector }> = {
    [S in M['selector']]: (
        M extends Message<S, infer P, infer R> ? (...args: P) => R :

It’s a mapped type, again, that maps union members that have Message type to an appropriate function signature. It takes advantage of TypeScript’s automatic distributivity: a union of products gets rewritten to be a product of unions. Then, in the conditional type M extends Message<...> ? ..., it projects out just exactly the member of interest again.2

This time we use the mapped type as-is instead of re-projecting it into a union using indexed access like we did with MessagesProduct above.

Type-safe interpretation of messages

Now we have types for our interfaces, and types for the messages that match them, can we write a type-safe generic perform function? Yes, we can!

function perform<I extends Methods<M>,
                 S extends ValidSelector,
                 M extends Message<S, any, any>>(i: I, m: M): void
    m.callback(i[m.selector](... m.args));

An example

Given the above definition for I, actually using Messages<I> produces the following type definition3 (according to the IDE that I use):

type M = Message<"m1", [a: string, b: number], boolean>
       | Message<"m2", [], void>
       | Message<"m3", [n: number], void>
       | Message<"m4", [x: [string, string]], { k: string; j: string }>
       | Message<"m5", [a: string, b: string[]], number>

Conversely, given the M from the top of the file, we get the following for Methods<M>:

type I = {
    m1: (a: string, b: number) => boolean;
    m2: () => void;
    m3: (n: number) => void;
    m4: (x: [string, string]) => { k: string; j: string };
    m5: (a: string, b: string[]) => number;

Roundtripping works too: both Methods<Messages<I>> and Messages<Methods<M>> give what you expect.

TypeScript is really cool

It’s a fully-fledged, ergonomic realization of the research started by Sam Tobin-Hochstadt, who invented Occurrence Typing, the technology at the heart of TypeScript.

Then, building on the language itself, emacs with tide, flycheck, and company makes for a very pleasant IDE.4

Congratulations to Sam, whose ideas really have worked out amazingly well, and to the TypeScript team for producing such a polished and pleasant language.

Appendix: Full code implementing this idea

This module implements the idea described in this article, extended with the notion of EventMessages, which don’t have a callback.

// This Tuple type (and tuple() function) is a hack to induce
// TypeScript to infer tuple types rather than array types. (Source:
// https://github.com/microsoft/TypeScript/issues/27179#issuecomment-422606990)
// Without it, [123, 'hi', true] will often get the type (string |
// number | boolean)[] instead of [number, string, boolean].
export type Tuple = any[] | [any];
export const tuple = <A extends Tuple>(... args: A) => args;

// Type ValidSelector captures TypeScript's notion of a valid object
// property name.
export type ValidSelector = string | number | symbol;

export type EventMessage<Selector extends ValidSelector, Args extends any[]> =
    { selector: Selector, args: Args };

export type RequestMessage<Selector extends ValidSelector, Args extends any[], Result extends Exclude<any, void>> =
    { selector: Selector, args: Args, callback: (result: Result) => void };

export type Message<Selector extends ValidSelector, Args extends any[], Result> =
    void extends Result ? EventMessage<Selector, Args> : RequestMessage<Selector, Args, Result>;

// Function message() is needed for similar reasons to tuple() above:
// to help TypeScript infer the correct literal type for the selector
// (as well as the arguments).
export const message = <S extends ValidSelector, A extends Tuple, R>(m: Message<S, A, R>) => m;

type MessagesProduct<I, ContextArgs extends any[]> = {
    [K in keyof I]: (I[K] extends (...args: [...ContextArgs, ...infer P]) => infer Q
        ? Message<K, P, Q>
        : never);

export type Messages<I, ContextArgs extends any[] = []> = MessagesProduct<I, ContextArgs>[keyof I];

export type Methods<M extends { selector: ValidSelector }, ContextArgs extends any[] = []> = {
    [S in M['selector']]: (
        M extends RequestMessage<S, infer P, infer R>
            ? (void extends R ? never : (...args: [...ContextArgs, ...P]) => R)
            : (M extends EventMessage<S, infer P>
                ? (...args: [...ContextArgs, ...P]) => void
               : never));

export function perform<I extends Methods<M, ContextArgs>,
                        S extends ValidSelector,
                        M extends RequestMessage<S, Tuple, any>,
                        ContextArgs extends any[] = []>
    (i: I, m: M, ...ctxt: ContextArgs): (M extends RequestMessage<S, Tuple, infer R> ? R : never);
export function perform<I extends Methods<M, ContextArgs>,
                        S extends ValidSelector,
                        M extends EventMessage<S, Tuple>,
                        ContextArgs extends any[] = []>
    (i: I, m: M, ...ctxt: ContextArgs): void;
export function perform<I extends Methods<M, ContextArgs>,
                        S extends ValidSelector,
                        M extends RequestMessage<S, Tuple, any>,
                        ContextArgs extends any[] = []>
    (i: I, m: M, ...ctxt: ContextArgs): any
    const r = i[m.selector](...ctxt, ... m.args);
    return r;

  1. Well, at least in TypeScript v4.x, anyway. I don’t know about earlier versions. 

  2. Actually I’ll admit to not being quite sure that this is what’s really going on here. TypeScript’s unions feel a bit murky: there’s been more than one occasion I’ve been surprised at what a union-of-products has been automatically “simplified” (?) into. 

  3. Hey, what’s going on with those named tuple slots? I would have expected a tuple type like [number, string] not to be able to have names attached to the slots, but it turns out I’m wrong and the compiler at least internally propagates names in some circumstances! It even re-uses them if you convert a Messages<I> back into an interface, Methods<Messages<I>>… 

  4. Here’s my .emacs TypeScript setup, based on the examples in the tide manual:

    (defun setup-tide-mode ()
      (flycheck-mode +1)
      (setq flycheck-check-syntax-automatically '(save mode-enabled))
      (eldoc-mode +1)
      (tide-hl-identifier-mode +1)
      (company-mode +1)
      (local-set-key (kbd "TAB") #'company-indent-or-complete-common)
      (local-set-key (kbd "C-<return>") #'tide-fix))
    (setq company-tooltip-align-annotations t)
    (add-hook 'typescript-mode-hook #'setup-tide-mode)

Crashing Squeak Smalltalk is easy—or is it?

Recently on HN, rbanffy brought up a form of the old chestnut about crashing the image by simply executing true become: false.

It turns out it’s no longer true!

In the latest Squeak,

True := False.

doesn’t work – the compiler complains that you can’t assign into a read-only variable. So let’s try this:

Smalltalk at: #True put: False.

But now the metaprogramming system complains you’re trying to modify a read-only binding! So we view source on ClassBinding»value:, and see that a resumable exception is being used to guard the modification, so let’s explicitly signal that we really want to modify that binding:

[Smalltalk at: #True put: False]
  on: AttemptToWriteReadOnlyGlobal
  do: [:ex | ex resume: true].

Finally! Now, evaluating True yields False.

But the image keeps running! Use of the literal class True seems to be rare enough that things are OK for at least several minutes after the change.

Doing this, however, definitely should immediately torpedo things:

true become: false.

… huh. It didn’t work. It used to! Again, on this current Squeak version, we see a different behaviour. This time, it says Cannot execute #elementsExchangeIdentityWith: on read-only object #(false).

So we’ll have to try harder:

true becomeForward: false.

That doesn’t work either! Same error as for #become:.

Welp, I’m actually all out of ways to crash this modern image in analogous ways to the easy pitfalls of images of yesteryear…

P.S. Running the following command,

ln -sf /dev/zero /lib/x86_64-linux-gnu/libc.so.6

analogous to True := False, definitely does have an analogous effect on a similarly-“alive” Unix image :-) . I just tried it on a scratch VM; the results are pretty intimidating!

Developing with Squeak on a Cellphone

Part of a series: #squeak-phone

One lovely thing about working in Smalltalk is the effortlessness of development.

I started off developing the code on my desktop machine, and occasionally testing it on the phone itself. I just rsync the image and changes files back and forth. This lets me pick up exactly where I left off on the other device each time I move over.

However, developing on the phone was challenging because of the lack of a keyboard (though I’ll post soon about an on-screen keyboard I’ve written). So I installed RFB (from here) into my image on the desktop, and tested it. Then I saved the image and rsynced it to the phone as usual, and presto, I can develop and test interactively on the phone itself:

Using VNC to develop on the phone itself
Using VNC to develop on the phone itself

There were a couple of things I had to do to get this to work:

  • Use this version of RFBServer: http://source.squeak.org/ss/

  • Change AllowTcpForwarding no to yes in /etc/ssh/sshd_config on the phone and then service sshd restart

  • Use ssh -L 5900:localhost:5900 pm to log into the phone (that’s the green-screen transcript in the background in the picture above)

BitSyntax for Smalltalk

Part of a series: #squeak-phone

Hand-written binary parsing/unparsing sucks

As I’ve been working on a mobile Smalltalk system, I’ve found myself needing to decode and encode a number of complex telephony packet formats1 such as the following, an incoming SMS delivery message containing an SMS-DELIVER TPDU in GSM 03.40 format, containing seven-bit (!) GSM 03.38-encoded text:

02 01 ffff
01 28 07911356131313f3
04 0b911316325476f8 000002909021044480 0ec67219644e83cc6f90b9de0e01

It turns out there are a plethora of such binary formats needed to get a working cellphone.

I started off hand-rolling them, but it quickly became too much, so I borrowed liberally stole from Erlang, and implemented BitSyntax for Smalltalk. (After all, I am already using Erlang-influenced actors for the Smalltalk system daemons!)

I’ve done this before, for Racket, and there are plenty of other similar projects for e.g. JavaScript and OCaml.

Every language needs a BitSyntax, it seems!

What does BitSyntax do?

The BitSyntax package includes a BitSyntaxCompiler class which interprets BitSyntaxSpecification objects, producing reasonably efficient Smalltalk for decoding and encoding binary structures, mapping from bytes to instance variables and back again.

The interface to the compiled code is simple. After compiling a BitSyntaxSpecification for the data format above, we can analyze the example message straightforwardly:

parsedMessage := SmsIncoming loadFrom: (ByteArray fromHex:
    '02 01 ffff
     01 28 07911356131313f3
     04 0b911316325476f8 000002909021044480 0ec67219644e83cc6f90b9de0e01')

and, if we wish, serialize it again:

serializedBytes := ByteArray streamContents: [:w | parsedMessage saveTo: w]

How does it work?

Syntax specifications are built using an embedded domain-specific language (EDSL).

For example, for the above data format, we would supply the following spec for class SmsIncoming:

        (1 byte >> #msgType),
        (1 byte >> #type),
        (2 bytesLE >> #simIndex),
        (1 byte >> #id),
        ((1 byte storeTemp: #payloadLength expr: 'payload size'), 'payloadLength' bytes)
            >>> #payload <<<
                (SmsAddress codecCountingOctets >> #smscAddress),
                (SmsPdu codecIncoming >> #tpdu))

along with appropriate specs for SmsAddress and SmsPdu (omitted for space reasons here) and the following for the SmsPdu subclass SmsPduDeliver:

        (1 bit boolean >> #replyPath),
        (1 bit boolean >> #userDataHeaderIndicator),
        (1 bit boolean >> #statusReportIndication),
        (2 bits),
        (1 bit boolean >> #moreMessagesToSend),
        (2 bits = 0),

        (SmsAddress codecCountingSemiOctets >> #originatingAddress),
        (1 byte >> #protocolIdentifier),
        (1 byte >> #dataCodingScheme),
        ((7 bytes
                transformLoad: [:v | 'self class decodeSmscTimestamp: ', v]
                save: [:v | 'self class encodeSmscTimestamp: ', v])
            >> #serviceCentreTimeStamp),
        (((1 byte >> #itemCount)
            transformLoad: [:v | 'self userDataOctetsFor: ', v]
            save: [:v | 'itemCount'])
                storeTemp: #userDataLength expr: 'userData size'),
        (#userDataLength bytes >> #userData)

These are non-trivial examples; the simple cases are simple, and the complex cases are usually possible to express without having to write code by hand. The EDSL is extensible, so more combinators and parser types can be easily added as the need arises.

How do I get it?

Load it into an up-to-date trunk Squeak image:

(Installer squeaksource project: 'BitSyntax')
    install: 'BitSyntax-Core';      "the compiler and EDSL"
    install: 'BitSyntax-Examples';  "non-trivial examples"
    install: 'BitSyntax-Help'.      "user guide and reference"

You can also visit the project page directly.

The package BitSyntax-Help contains an extensive manual written for Squeak’s built-in documentation system.


  1. Telephony packet formats are particularly squirrelly in places. Seven-bit text encoding? Really? Multiple ways to encode phone numbers. Lengths sometimes in octets, sometimes in half-octets, sometimes in septets (!) with padding implicit. Occasional eight-bit data shoehorned into a septet-based section of a message. Bit fields everywhere. Everything is an acronym, cross-referenced to yet another document. Looking at the 3GPP and GSM specs gave me flashbacks to the last time I worked in telephony, nearly 20 years ago… 

PSA: Check your Acer C720 battery!

Today I noticed that the keyboard on my 2013-vintage Acer C720 chromebook was visibly curved, being pushed up from below by the battery, which had swollen enormously.

This can be really dangerous, so I took the back off the machine and removed the battery. Here’s what it looks like:

Swollen C720 battery

Swollen C720 battery

Swollen C720 battery


Changing the battery is extremely quick and easy, and replacement batteries can be had for about USD40/EUR40 delivered. I’ve already ordered a replacement.