Welcome to the blog series "From The Board Up" by Michael Brown. In this series we create a QNX image from scratch and build upon it stepwise, all the while looking deep under the hood to understand what the system is doing at each step.

With QNX for a decade now, Michael works on the QNX kernel and has in-depth knowledge and experience with embedded systems and system architecture.

Welcome back for post #5 in the series! In previous articles we managed to:

• compile our helloworld program
• get it running on QNX 8.0
• see that stdout is using a file descriptor on /dev/text
  • which is provided by a resource manager in the kernel's process
  • that uses (in our configuration) the 8250 UART provided by startup
• create ls to look around at files (and use stdout), and
• create nkiss for a bit of interactivity (input and output)

But, through all of that we didn't get into how printf() actually gets the data to /dev/text.

printf()?

If you recall, the source for our helloworld is:

#include <stdio.h>
#include <stdlib.h>

int main(void)
{
    printf("Hello, world!\n");

    return EXIT_SUCCESS;
}

Let's look inside the helloworld file and see exactly what the compiler and QNX C library do to make printf() happen.

Since helloworld is an ELF, we can use the objdump utility for ELF files to disassemble the contents of the executable code within the ELF. If we remove a bunch of stuff we don't care about, we see this:

x86_64> objdump -d ./helloworld
...
0000000000000741 <main>:
 741:   55                      push   %rbp
 742:   48 89 e5                mov    %rsp,%rbp
 745:   48 8d 05 18 00 00 00    lea    0x18(%rip),%rax        # 764 <_fini+0x9>
 74c:   48 89 c7                mov    %rax,%rdi
 74f:   e8 7c fe ff ff          callq  5d0 <puts@plt>
 754:   b8 00 00 00 00          mov    $0x0,%eax
 759:   5d                      pop    %rbp
 75a:   c3                      retq

No printf?!

Without digging into the details, we can quickly see that the compiler optimized the call to printf() by replacing it with a call to puts(). That kinda makes sense because we:

• are not providing any special formatting (e.g. %d) for printf() to interpret, and,
• have a '\n' at the end of our string

... and that is what puts() is really good at.

I'll skip this, but, if we did go spelunking around the ELF, we'd see the string passed to puts() is actually Hello, world!, without the newline. But, this modified string passed to puts() gets us where we intend to go, so, that's fair.

Nicely done, optimizer!

I too will optimize, because I want to keep this simple and unfortunately neither printf() nor puts() is "simple" because C streams have unbuffered vs. fully buffered vs. line buffered, and we'd have to do a lot of digging, and this isn't about that. However, we can make an educated optimization and skip all that foofaraw.

QNX is POSIXy, and we discussed last time how stdout is using a file descriptor, STDOUT_FILENO, i.e. 1. So, let's simplify by cutting to the chase: let's use the POSIX function write() in our helloworld to write our important message to standard out:

#include <stdlib.h>
#include <string.h>
#include <unistd.h>

int main(void)
{
    char const * const msg = "Hello, world!\n";
    write(STDOUT_FILENO, msg, strlen(msg));

    return EXIT_SUCCESS;
}

Build this and take another gander inside the ELF with objdump to look at main(), and we see

0000000000000801 <main>:
...
 841:	e8 2a fe ff ff       	callq  670 <write@plt>

We can see that our code is indeed calling write(). We'll ignore the Procedure Linkage Table (PLT) (an ldqnx-related topic for another time).

To see details of the implementation of write(), we have to look inside the C Library, libc.so, and, keeping our eyes on the prize, we can spy something very interesting:

000000000002ce30 <write>:
...
   2cea2:       e8 29 8d ff ff          callq  25bd0 <MsgSendv@plt>

write() is calling MsgSendv(), which QNX's documentation says will "Send a message to a channel". According to the "Classification" section, this is a QNX thing, so let's detour a bit to understand this QNX thing of messages and channels.

Channels, Connections, and Messages, oh my!

There's lots of material in the QNX user docs about channels and messages, but, let's see if we can distill it down to an elevator pitch:

• The QNX kernel provides inter-process communication (IPC).
• A process can ask the kernel to create a channel.
• A process can ask the kernel to create a connection to a channel.
•  A process can ask the kernel to pass messages (data) via a connection to a channel.
•  A process can ask the kernel to receive messages sent to a channel.

Well, that wasn't really that hard. The devil's in the details, but that's the gist of it. Now that we understand the essence of message-based IPC, a couple more things to note:

• The QNX kernel does not care what is in a message. Like Milton and the cake, "Just pass." It does not semantically examine the contents at all. Passing a message is all about copying the message data.
• Message sending is synchronous. i.e. you are blocked until whoever is on the other side of the connection returns an answer (aka a reply).
• For those who don't like to wait, you can send a very small amount of data asynchronously. But, there's no free lunch. You can send a small amount of data, and, you're not blocked, but, you also get no reply. This is known as sending a "pulse" of data.

You Said You'd Explain write()!

We're working our way up to that. And by 'up' I mean up the software stack.

The kernel provides the functionality for channels, connections, messages, and pulses. i.e. these things are services provided by the kernel, so there must be some way for user code to request these services from the kernel. And there is.

But first, let's be clear about what functionality CPUs support, and how some software such as, say, a QNX kernel, may make use of it.

Asking The Kernel For Services

This is a big topic. Two big topics, actually:

• trust, aka privilege level
• spatial isolation.

Each deserves its own discussion, so I'll come back to these in later articles.

Just so we can stay focussed on write() and IPC for now, I'll say this:

• The CPU always knows how much to trust the code that is currently executing.
• Based on the current amount of "trust" in the code that's running, the CPU may or may not allow certain instructions to execute.
• CPUs provide a way for untrusted code to ask more trusted code to do something on its behalf.

Why this distinction in trust of code? Because the best-laid schemes o' mice an' men gang aft agley. Sometimes we and our fellow-mortals make small mistakes. And sometimes big mistakes.

System Calls

Long story short: user code is untrusted, and therefore limited in what it can and may do. However, it can ask the kernel – as code that is more trusted – to do work on its behalf; the kernel may or may not oblige, depending on current conditions and system configuration.

For user code to make a request of kernel code, it executes a special CPU instruction known as the "system call" instruction / opcode: syscall.

This opcode causes a few things to happen, the most relevant of which are:

• The CPU changes state: from "I'm running untrusted code" to "I'm running trusted code".
• The CPU jumps to a different location where there is some (trusted) code.

The CPU (H/W) itself will automatically cause the transition from untrusted mode to trusted mode, but how does the CPU know where to jump and that there's trusted code there? That is Yet Another Thing the kernel has to do as part of its initialization. "Hey, CPU, if someone executes syscall, jump to this location for me and I'll take it from there!"

Kernel Calls

If that's a system call / syscall, what's a "kernel call"?

A kernel call is just a protocol that makes uses of the system call. There's a QNX-specific protocol atop system calls for kernel calls. Ditto for Linux, Windows, FreeBSD, etc. If you write your own OS, you can make up your own! There are pros and cons for each implementation choice. It depends what you want to prioritize.

One parameter of the protocol (typically) is "what number is this?", where the number indicates the service you're requesting.

Shifting our attention back to the topic at hand: the number for MsgSendv() is 11. In hex, that's 0xB. So let's see how QNX uses the system call and the number 11 to implement the kernel call MsgSendv().

MsgSendv()

If we look inside the C library for the implementation of the kernel call MsgSendv()...

... so anyway, if we look inside the C library for the implementation of MsgSendv(), we'll see this:

000000000004fb90 <MsgSendv>:
   4fb90:       49 89 ca                mov    %rcx,%r10
   4fb93:       b8 0b 00 00 00          mov    $0xb,%eax
   4fb98:       0f 05                   syscall
   4fb9a:       c3                      retq
   4fb9b:       48 89 c7                mov    %rax,%rdi
   4fb9e:       48 8d 0d 00 00 00 00    lea    0x0(%rip),%rcx
   4fba5:       49 bb d3 94 05 00 00    movabs $0x594d3,%r11
   4fbac:       00 00 00
   4fbaf:       48 b8 68 1c 00 00 00    movabs $0x1c68,%rax
   4fbb6:       00 00 00
   4fbb9:       4c 01 d9                add    %r11,%rcx
   4fbbc:       48 8b 04 01             mov    (%rcx,%rax,1),%rax
   4fbc0:       ff e0                   jmpq   *%rax

There is the system call made with the Intel opcode syscall, and you can see just before it the number 11 ($0xb) being moved into a register (%eax).

The rest of the stuff before and after the syscall is for a larger discussion on another day, but, I'll briefly mention that the stuff after the system call has to do with setting errno.

So, that covers how the kernel provides:

• kernel calls
• kernel calls for message-based IPC (channels, connections, messages, pulses).

... but we still haven't gotten into how write() works!

And that's where we start building upon message-based IPC, and start thinking about how one would used the message-based IPC to go about creating some infrastructure for POSIX-based file functionality.

Coming Up...

We still haven't explained printing, beyond "Something to do with sending a message", but we're getting there.

Coming up in future posts:

• Using QNX's message-based IPC.
• An IPC-based protocol for POSIX file systems.
• More about our interactive program and utilities, and how they can be improved to put the "System" in Operating System.
• And more!