Welcome to the blog series "From The Board Up" by Michael Brown. In this series we create a QNX image from scratch and build upon it stepwise, all the while looking deep under the hood to understand what the system is doing at each step.

With QNX for a decade now, Michael works on the QNX kernel and has in-depth knowledge and experience with embedded systems and system architecture.

Welcome back for post #6! In the previous post we:

• started to dig into our helloworld program, but
• decided to re-implement it using the POSIX function write(), and then
• discussed a bit about CPU-specific "system calls", and how to write kernel calls atop system calls, and
• talked a bit about QNX's message-based inter-process communication (IPC),

but, we didn't get into how write() actually gets the data to /dev/text.

Bear with me for a second here. Before we finally get to that, we need to build our way up from message-based IPC, which even on its own is a very useful thing!

Message Protocols

Once you have a message-based IPC mechanism, how you coordinate interactions between two processes is (almost) completely up to you.

We stated last time that the QNX kernel does not "look at" / interpret / make any decisions based on the contents of messages, so you are free to implement whatever message protocol(s) you want.

Great freedom to make custom choices also means variety, which means complexity. Therefore, most people fall into a small set of approaches to message protocols.

Message Multiplexing and Demultiplexing

Since you can do whatever you want, you could, in theory, decide to have a honkin huge number of connections between two processes that want to communicate: one connection for each thing you want to talk about.

However, the usual approach is to have a single connection, and then define a message protocol which supports a fixed set of message types, each message type having its own definition of data to be passed / returned.

This protocol might say something like:

• The first N bytes of the message indicate the message type, and
• the contents after that depend on the message type.
• If the message is message type 1, there will be 64 bytes, of which the first 4 bytes are ... blah blah blah
• Then yadda yadda yadda for the other supported message types.

You can imagine that if you were to do such a thing, then someone receiving messages might have some code that looks like this:

    // Receive message into a buffer known as 'msg'
    // First 2 bytes indicate the message type
    // After that, it depends.
    switch (msg.type) {
        case MESSAGE_TYPE_1: return handle_message_type_1(msg);
        case MESSAGE_TYPE_2: return handle_message_type_2(msg);
        ...

Or, instead of a switch, have an array of function pointers, or an associative array, or whatever. Your protocol. Your code. Whatever.

Regardless, you're basically demultiplexing the data within a message, i.e.:

• receiving a message of unknown type,
• figuring out what the specific message type is, then
• handing the message off to a message-type-specific handler.

And once you've handled the message, i.e.

• performed whatever action/service is required, and
• replied,

... you're back to waiting for the next message.

Add a while (1) loop, and you got yourself a message pump, pumping received messages through to the appropriate message handlers:

    // Our message pump
    while (1) {
        // receive message
        // look up appropriate handler
        // pass message to appropriate handler
    }

A message pump, aka an event loop, aka an イベントループ, is a fairly common pattern. It's certainly well known to those who fought with Windows back in the day.

Since it is a common pattern with message-based IPC, QNX provides a simple software framework (in the C library) known as the "dispatch framework" (because it dispatches received messages to the appropriate handler).

You tell the framework about your handlers, which messages they handle, the sizes of the messages, and then you make a simple message pump.

The "only" requirement for a message-based protocol that uses this framework is that the first 2 bytes of the message must indicate the message type. Any data in the message after that, that's all you and your protocol.

(Yes, there are other requirements, but, for our purposes, that's close enough for jazz.)

Hey, You Said You'd Explain write()!

Oh, right. Once you have:

• message-based IPC,
• a framework for message dispatching, and
• the freedom to implement whatever message-based protocol you want,

... it seems natural that QNX then said "Wait a second! We can implement POSIX file functionality using some QNX-specific message-based IPC protocol!"

And that's what happened.

For example, in the case of write(), the protocol requires sending a message of type _IO_WRITE, i.e. a message with message type 258 (0x0102), and "some more data".

The data to send with that message is described by a C struct, and if we look at that struct in iomsg.h, we'll see something like this:

struct _io_write {
    uint16_t  type;
    uint16_t  combine_len;
    uint32_t  nbytes;
    uint32_t  xtype;
    uint32_t  zero;
    /* unsigned char data[nbytes]; */
};

We can see that type is the first field: i.e. the message struct incorporates the message type as part of its definition. After type (and some other fields we'll ignore for now), it has a comment indicating that the actual contents of the message to be written will follow immediately after the struct.

Meta: I just checked the source revision log, and the definition of this structure predates flexible array members, which were added in C99. I'm not sure when the GCC "array of length zero" extension to the C language was added. In general, older code also sometimes uses the "array with a length of 1" idiom, and then you would purposely exceed the bounds of the array. But, that's undefined behaviour (UB) that may or may not work for you.

(Aside) Possible Approaches To "Variable Number Of Bytes To Follow"

struct foo1 {
    uint16_t num_bytes;
    uint8_t  data[]; // C99 flexible array member
};

struct foo2 {
    uint16_t num_bytes;
    uint8_t  data[0]; // GCC array-of-length-zero extension
};

struct foo3 {
    uint16_t num_bytes;
    uint8_t  data[1]; // Old skool UB-but-IF-it-works-ATM-then-sure-whatever.
}

// Data follows structure
struct foo4 {
    uint16_t num_bytes;
}

Beware: Each has its quirks (sizeof, padding etc.).

So to recap...

• write() calls MsgSendv() to send a message of type _IO_WRITE, and
• the message contains:
  • struct _io_write, with .type set to _IO_WRITE,
  • [and other things set], followed by
  • the data to be written

What happens when the message is received is up to whoever receives the message and handles it.

Operations On POSIX Files

There are a lot of things you can do with a POSIX file:

• open a file,
• read some bytes,
• seek to a specific offset within the file / array of bytes,
• write some bytes,
• close a file,
• and so on, in that fashion.

Since they each have a different set of parameters, it makes sense to have a message type and message structure defined for each. Since there are going to be a lot of message types, maybe a framework to help with this?

That dispatch framework looks pretty handy. But, it sure would be nice to have even more functionality in a framework to handle these messages for operations on POSIX files, maybe?

That's where the resource manager framework comes in.

"resmgr"s and "iofunc"

A "resource manager" (aka resmgr, pr. "rezz mugger") is something that uses QNX's POSIX file-system message protocol / the resource manager framework to:

• expose a file (or files, or files and dirs) to the system,
  • i.e. make it known to QNX – and thereby all processes in the system – at some path
• respond to operations on POSIX files / POSIX file-system messages.

Since there are a lot of messages for this file-based I/O, and it might be helpful to have a lot of useful defaults, in the spirit of "Can't someone else do it?", a lot of useful defaults are provided with the "iofunc" functionality.

For example, the helper function iofunc_write_verify() will examine the message passed with an _IO_WRITE message to see if, given the way the file descriptor was created, the person sending the message may perform a write. For example, if someone opens a file for reading, and then tries to write to it, someone has to check for this situation and say "No way, friendo."

File Descriptor

Yes, I've talked about QNX's message-based IPC with its channels and connections, and then we jumped up to "write() calls MsgSendv() to send a message of type _IO_WRITE", but there's still a gap here. If we look at the signatures of the two functions:

ssize_t write(int fd, const void* buff, size_t nbytes);

long MsgSendv(int coid, const iov_t* siov, size_t sparts, const iov_t* riov, size_t rparts);

write() uses a POSIX file descriptor, but MsgSendv() uses a coid, aka a connection id. To quote the great Canadian philosopher Kevin Blackmore, "Same t'ing."

Now, I can already hear hackles being raised by some readers at QNX who will – quite rightly – point out they they are NOT! the same thing:

• You can use POSIXy messages (e.g. _IO_WRITE, i.e. 0x0102) on a coid that is a file descriptor, but,
• you should not use POSIXy messages on a coid that is not a file descriptor because it's using some other message protocol.

In other words, a file descriptor is a connection to a channel that has a resource manager receiving and supporting POSIXy messages.

Yes, and we partition the number space for coids into file descriptors vs. "side channels", etc. etc. (But, not yet, b'y.)

That being said, from the QNX kernel's perspective, when it comes to messages, a coid can be used to send messages, and receive replies to those messages. The QNX kernel does not care what those messages – or their contents – mean to you; the same functional guarantees are provided, regardless of content.

Big Picture Recap

So to wrap up all of our learnings into one list:

• QNX kernel supports message-based IPC.
• QNX defines a QNX-specific protocol for operations on POSIX files that is based on message-based IPC.
• A POSIX file descriptor (fd) is a connection id (coid), but, not all coids are fds.
• The QNX C library provides implementations of write(), read(), etc. that use
  • message-based IPC, and
  • the QNX-specific protocol for operations on POSIX files.
• Resource managers running on QNX
  • tell the QNX kernel that they want to make a file (or files) (and directories) "available to the system"
    • i.e. available at a specific path (e.g. /dev/text)
  • respond to the QNX-specific message types for POSIX functions from
  • clients (aka processes with connections)

And in the case of our "Hello, world!" example:

• During kernel initialization:
  • A QNX resource manager said to QNX "I'm /dev/text".
  • The QNX kernel open()ed the file /dev/text for standard in, out, and error to get 3 file descriptors.
• During processing of the initialization script:
  • helloworld was created, and it was passed the file descriptors for standard in, out, and error.
• When helloworld ran:
  • it called puts() (in the original version) which
  • called write() which
  • called MsgSendv()
    • on the file descriptor for standard out
      • which is a connection to the resource manager for /dev/text
    • with the message type _IO_WRITE with
    • the struct _io_write followed by
    • the bytes for the string "Hello, world!\n".
• This caused the QNX-kernel-provided resource manager behind /dev/text to
  • receive the message, then
  • invoke the message handler for message type _IO_WRITE, which
  • wrote the characters of the message after the struct (i.e. the bytes to be written) to the startup-provided debug callout.
• The startup-provided debug callout wrote the bytes to the 8250 UART.

More simply put:

helloworld -> puts() -> write() -> MsgSendv() _IO_WRITE message -> /dev/text resource manager -> startup-provided callout -> 8250 UART

Startup-Kernel Interface

Just to cover one last piece of magic: How does the kernel resource manager for /dev/text pass a character to the startup-provided code that writes to the 8250 UART?

During initialization, the startup passes to the QNX kernel a pointer to a structure that contains, amongst many other things, a function pointer that the kernel can call to cause a single character to be "displayed" – whatever that means to startup – for debugging purposes.

This structure that startup provides is known as the "system page" because it

• describes the system, and
• provides system-specific functionality

... for the QNX kernel.

There are several aircraft carriers full of interesting information here, and we'll look at that later.

Coming Up...

• Looking at IPC a little more closely
• More about the Startup-Kernel Interface
• More about the kernel initialization and architecture
• Improving our interactive program, and the utilities we've written