Interprocess Communication in L4Re

There are several layers of interfaces and mechanisms for interprocess communication (IPC) in L4Re, alongside various fundamental concepts.

Messages

IPC operations define messages using a tag abstraction with the following properties:

A label to identify the kind (or protocol) of the message (distinct from the gate label)
Flags controlling the behaviour of the call operation caused by the communication of the message
Words and items providing the message content, as described below

Messages employ "virtual registers" found in the user-level thread control block (UTCB):

"Words" for sending are placed in message registers and are copied to the receiver; these are typically used for simple values
"Items" for sending are also placed in message registers; these denote special values
Details of items to be received are specified in buffer registers as "receive items"

For items to be successfully sent, the receiver must declare the expected items in its buffer registers. So, a call from a thread that sends an item must be received by a thread declaring an expected item:

If a thread, having made a call, expects items in return, it too must declare these items in the buffer registers:

Failure to declare expected items will cause an "overflow" or "sent message cut" error. Thus, the necessary steps are as follows:

Reserve a capability slot for any expected capability
Declare the expected capability using the slot
Perform a wait operation
Test for any items providing a capability
Obtain any capability
Start again (by providing a new slot for any expected capability)

A limitation exists in combining words and items. Since sent items are not counted but sent words are, words must be placed in the message registers before any items. Received items are copied into corresponding positions in the receiver's message registers.

Unless the buffer descriptor register has been adjusted, expected items are declared in the buffer registers starting at register 0, with each item corresponding to the items as they are encountered after the words in the message registers.

The buffer descriptor register indicates the positions of items in the buffer registers as well as various flags. The Fiasco.OC kernel code has more details about the register structure:

src/abi/l4_buf_desc.cpp

The following layout applies to the buffer descriptor register in Fiasco.OC:

Bits
31..24	23..15	14..10	9..5	4..0
Flags		First capability item	First I/O buffer item	First memory buffer item

It would seem that a common value for the register is zero, indicating that items are present from position zero onwards.

Details of the tag abstraction (l4_msgtag_t) and associated functions can be found here:

pkg/l4re-core/l4sys/include/types.h

Items and Flexpages

The flexpage is an abstraction used to describe memory, capabilities and other things. Items employ flexpage values along with additional details:

The "base" or "snd_base" of an item describes the nature of the item
The "data" is the actual flexpage value describing the item

It appears possible to just specify the appropriate metadata in the "base". For example:

mr[0] = 0 | L4_ITEM_MAP | L4_ITEM_CONT;

Here, the superfluous 0 merely indicates that no base capability is specified.

Long IPC

According to L4 Microkernels: The Lessons from 20 Years of Research and Deployment, "long IPC" is no longer supported in modern L4 implementations including Fiasco.OC. This is presumably the form of IPC described in various educational materials where the kernel copies data from the sender directly into the recipient's address space. Alternatives to this include explicit sharing of memory between sender and recipient.

IPC Gates and Labels

When binding a thread to an existing IPC gate (using l4_rcv_ep_bind_thread) or doing so when creating the gate (l4_factory_create_gate), a label can be associated with the gate which can then be inspected upon receiving messages. This label allows the gate through which the message arrived to be identified.

Consequently, gates may be associated with objects or structures that can be used to handle incoming messages by providing the address of such an object or structure as the label. Since the label is defined by the recipient and preserved by the gate, invalid addresses cannot be presented as labels by message senders.

The lowest two bits of a label is reserved for rights flags. Thus, addresses need to be aligned to four-byte boundaries, which is already typical on various architectures.

l4_ipc

The l4_ipc functions, defined here:

pkg/l4re-core/l4sys/include/ipc.h

These expose operations acting more directly with capabilities, message tags and the UTCB than other IPC interface layers. Only very limited support for updating the virtual registers is provided, this in the form of the l4_sndfpage_add and l4_sndfpage_add_u functions which add "items" for flexpage values to the message registers.

Flexpage definitions can be found here:

pkg/l4re-core/l4sys/include/__l4_fpage.h

The principal function is the l4_ipc function itself, with various other functions employing it to perform specialised IPC operations.

ipc_stream

A "stream" abstraction is provided for IPC clients and servers for the modification of the virtual registers.

Server-Side IPC Framework

The server-side IPC framework provides abstractions for server-like objects interpreting and dispatching IPC messages, with registry objects provided to dispatch messages to servers based on message labels.

Useful things to note:

Interfaces are defined by deriving from parameterised forms of L4::Kobject_t, with the parameters being the interface class itself, L4::Kobject, protocol information, and any "demands" made on capability slots by received items
Interfaces may be used by clients to access operations, but the inherited details seem really to be employed by servers implementing the interfaces

Kernel Activities

Message Items

Message items are handled in the Thread::transfer_msg_items method found here:

src/kern/thread-ipc.cpp

Each item is read from the sending UTCB:

src/abi/l4_types.cpp

According to the type of the item (Memory, Io or Obj, with the latter referring to general capabilities), the appropriate buffer register region is selected for iteration, such regions being defined by the buffer descriptor register:

src/abi/l4_buf_desc.cpp

Where buffer registers do not describe expected items that match the incoming items, errors are raised. Missing descriptions cause an "overflow" error.

Flexpages, being items, are defined here:

src/abi/l4_fpage.cpp