Welcome to the blog series "From The Board Up" by Michael Brown. In this series we create a QNX image from scratch and build upon it stepwise, all the while looking deep under the hood to understand what the system is doing at each step.

With QNX for a decade now, Michael works on the QNX kernel and has in-depth knowledge and experience with embedded systems and system architecture.

Erstwhile in this series, we talked about how helloworld using printf() in our minimal QNX configuration results in some characters being written to the 8250 UART:

helloworld -> puts() -> write() -> MsgSendv() _IO_WRITE message -> /dev/text resource manager -> startup-provided callout -> 8250 UART

In this post, I'd like to look a little more at the message-based inter-process communication (IPC) that the QNX kernel provides via kernel calls.

Channels and Connections

We saw how you can have simply a connection to a channel:

... but, it can be as complex as you need it to be.

You can have several clients connected to the same channel.

You can even have several connections to the same channel.

Or a little bit of everything.

Name and Fame

Let's identify (some of) the kernel calls that are used for this message-based IPC.

There are many variations to these, and a lot more functionality (e.g. pulses), but those are bigger topics, and all explained well in the user documentation. For now, this is enough for us to be dangerous.

Typical Interaction

Let's look at these functions individually.

MsgReceive()

MsgReceive() is your way of telling the QNX kernel: "I'm ready to receive messages from any of the connections to this channel."

Like most of the message-based IPC QNX kernel calls, MsgReceive() is a blocking call, i.e. once you call the function, you don't return from the function until you receive something.

The function returns an identifier, a "receive id" to help you distinguish who sent the message, so that you can use that identifier later to reply.

This is helpful because you can actually have many threads calling MsgReceive() on the same channel. When a message is sent, one thread is chosen to receive it, unblocked, and the rest remain blocked and ready to receive any other messages. You can have many messages received and being worked on at the same time, so, how to identify which message is being replied to when replying? The receive id, aka "rcvid".

It's a handle, a token, ticket, whatever, that the kernel gives you so that you can say later "Hey, QNX kernel, whoever this is, here's their reply back."

MsgSend()

Why is this kernel call called MsgSend()? Because it sends messages, Avi.

You tell the kernel 4 things:

• I want to send
• this message (array of bytes)
• over this connection, and
• I want the reply to be put here.

And, because this too is a blocking call, you're also saying "And don't come back until there's a reply!"

MsgReply()

This actually serves two purposes:

• the data for a reply, and
• the status to return to someone sending a message.

To understand this, let's go back a bit and look at the signature of MsgSend():

long MsgSend(int          coid,                     // connection id
             const void * message_to_send,          // pointer to buffer with message to send
             size_t       size_of_message_to_send,  // how many bytes to send
             void*        reply,                    // pointer to buffer into which the reply is to be written
             size_t       max_size_of_reply)        // size of the reply buffer

The important bit for now is that the return type of the function MsgSend() is a long. With MsgReply() you can say what value you want that long to be: i.e. that's the status to return:

int MsgReply( rcvid_t      rcvid,           // identifies the message received
              long         status,          // What you want MsgSend to return
              const void * data_to_reply,   // pointer to reply buffer
              size_t       size_of_reply );

Receiving Large Messages

I glossed over the details earlier when we discussed MsgReceive(), but, let's dig into the details of receiving a message.

First off, the signature of the function:

rcvid_t MsgReceive( int    chid,
                    void*  msg,                 // Pointer to buffer where the function can store the received data
                    size_t msg_size,            // Length of buffer
                    struct _msg_info * info );

The chid is the channel id. Makes sense because we receive a message on a channel from a connection to that channel.

A message being sent has to be received / written to somewhere, so msg makes sense. msg_size seems like important information. We tell the QNX kernel how big our buffer is so that it doesn't write outside that buffer.

But, we talked about how you can define protocols for message-based IPC, and how for each type there is a different / variable amount of data for the message.

This raises a few questions:

• How big do we have to make our receive buffer?
• What if that number is huge?
• If the receive buffer is too small for a message, does the kernel throw the rest away?

The quick answers are:

• Just big enough.
• Not a problem.
• No.

... because all of these questions are addressed with MsgRead(): given a receive id, read part of – or all of – the received message.

This naturally leads to the question: How do I know how much was actually sent? Answer: struct _msg_info provides that information about a message.

You can then use that info and MsgRead() to consume the message as you see best.

One might then think, "Oh, so when I receive a message, I can specify a msg_size of 0 and then read the entire message with MsgRead()?" You could, but, that's not how it's typically done. (We'll talk about this in the next post when we address pulses.)

To recap, you typically either:

• call MsgReceive() with a pointer to a buffer big enough for all the messages you expect to receive, or
• call MsgReceive() with a pointer to a buffer big enough for some messages, but
  • if there's more, use MsgRead() to read the rest.

You can call MsgRead() as often as you want. The message is available until you reply.

Replying With Large Messages

Ok, we've handled receiving large messages, but, what happens if you have a large reply?

If we reply to a message with MsgReply(), does that mean you have to have everything together before you reply?

That's one approach, yes. However, what if, to get the reply together, the server chooses to farm work out to a bunch of helper threads that will each get part of the reply together?

To help with that -- and similar challenges -- there's another approach:

• get part of the reply, any part, and write that part of the reply
• get a different part of the reply, any part, and write that part of the reply
• etc. etc. etc.
• until the entire reply is written
• Tell the kernel we're done with the message by
  • calling MsgReply() with
  • a buffer of 0 bytes, (or the last trickle of bytes, whatever), and
  • provide a status.

And that is what MsgWrite() , i.e. write all or part of a reply, is all about:

Making Room For Large Replies

Well, I imagine as part of your message-based protocol, you requested a number of bytes, so, it's pretty easy to know before sending the message how big the reply to expect is. Prepare accordingly.

Or, you can structure your protocol to break up a read of a huge amount of data into a set of smaller reads.

Or, you can structure your protocol to request a huge amount of data, and dictate that it shall be fine to receive a subset of that. POSIX certainly thinks that way:

• "The read() function shall attempt to read nbyte bytes from..."
• "The value returned may be less than nbyte ... if ... fewer than nbyte bytes [are] immediately available for reading."

All because working with one large contiguous buffer can be a bit of a pain in more ways than one.

A Contiguous Buffer Can Be A Bit Of A Pain

Instead of one huge contiguous buffer, can't I just have several smaller buffers that are each, individually, contiguous?

Hmm. I suppose, if you wanted that, you'd need something to describe

• the start address (aka base address) of each (contiguous) buffer, and
• the size of each buffer, and
• the number of buffers.

Fortunately, this is a pretty old idea, and has been baked into POSIX. For example, readv() lets you read into a vector of buffers, i.e. an array that describes a set of individually contiguous buffers.

The function looks like this:

ssize_t readv(int fildes, const struct iovec* iov, int iovcnt);

where iovec is "I/O vector", and where, on QNX, a struct iovec looks like this:

struct iovec {
    union {
        void*        iov_base;         // Base address of a buffer you can read from or write to
        const void * iov_base_const;   // Base address of a buffer you can read from
    };
    size_t iov_len;                    // Length of buffer, in bytes.
};

printf() / write() / MsgSendv()

Just to bring this back to where we started, we discovered that

• printf() was (eventually) going to call write()
• write() calls MsgSendv()

There's that 'v', and the docs for MsgSendv() say the signature is

long MsgSendv( int          coid,
               const iov_t* send_iov,
               size_t       send_num_parts,
               const iov_t* receive_iov,
               size_t       receive_num_parts );

Ah! It's using IOV. But write() is given a pointer to a single contiguous buffer. Why use MsgSendv() instead of MsgSend()? Well, there's nothing saying you can't have an IOV with just one buffer. "Just One Buffer", is "Just A Special Case" of an IOV. So, that fine.

And, I won't delve further into that Pandora's box, other than to vaguely allude to the multiplexing of similar kernel calls, while letting myself be distracted with, ooo, let's say, ... pulses!

Coming Up...

But, what if you don't want to block? What if you don't want a reply?

That's for next time when we talk about pulses.

Once we've covered pulses, we can really get into the various APIs the QNX kernel exposes to the the World Of User Code, because they're based on kernel calls, messages, and pulses.