Welcome to the blog series "From The Board Up" by Michael Brown. In this series we create a QNX image from scratch and build upon it stepwise, all the while looking deep under the hood to understand what the system is doing at each step.

With QNX for a decade now, Michael works on the QNX kernel and has in-depth knowledge and experience with embedded systems and system architecture.

Welcome back to the series! In the first two posts we managed to:

• describe a computer for QEMU to emulate;
• describe an image for mkifs to create to run within that computer
• which boots QNX, and
• displays the message "Hello, stranger!"
• through the 8250 UART (being emulated by QEMU)
• to QEMU's standard output.

Next, in a nod to the book, "The C Programming Language", written by Brian Kernighan & Dennis Ritchie (aka K&R), let's create a simple "Hello, world!" program and get it running on this minimal QNX system.

The Code

First, let's see the code:

#include <stdio.h>
#include <stdlib.h>

int main(void)
{
    printf("Hello, world!\n");

    return EXIT_SUCCESS;
}

This differs slightly from what's in older versions/drafts of the book, which had:

main()
{
    printf("hello, world\n");
}

Our version:

• tells the preprocessor to process 2 header files so that:
  • the preprocessor knows about the macro EXIT_SUCCESS
  • the compiler knows about the function printf and its signature (i.e. sees the declaration of printf),
• uses one of the two possible signatures mentioned in the C language specification for main.
  • The other one is int main(int argc, char* argv[]);, but, we're ignoring arguments for now.
  • (Yes, the spec also says "or in some implementation-defined manner", but, let's stay on the beaten path for now.)

An Aside on Functions

A function foo declared this way can be called with any number of parameters:

int foo() { return 42; }

Say we have a file foo.c:

int foo()      { return 42; }
int main(void) { return foo(5.0, "Mahlzeit!", 0x1D0AB1DE); }

... and then compile it, being extra picky with warnings:

x86_64 $ gcc -Wall -Wextra -c foo.c

All good?! No warnings!?!

This might be surprising, but, this goes back to the early days of C, pre-standardization (ANSI, 1989) when the only reference for C was the book "The C Programming Language". And that's why this book-based de facto standard is referred to as "K&R C". And, that's why a function with this style/format is known as a "K&R function". For example:

int add(a,b)
    int a;
    int b;
{
    return a + b;
}

However, if we use (void) to be clear that this function wants NOTHING passed to it, like so:

int foo (void) { return 42; }
int main(void) { return foo(5.0, "Mahlzeit!", 0x1D0AB1DE); }

... then we get:

x86_64 $ gcc -Wall -Wextra -c foo.c
foo.c: In function ‘main’:
foo.c:2:25: error: too many arguments to function ‘foo’
 int main(void) { return foo(5.0, "Mahlzeit!", 0x1D0AB1DE); }
                         ^~~
foo.c:1:5: note: declared here
 int foo(void)  { return 42; }

Yes, C++ lets you use () and it knows you mean "I accept nothing". This is a C thing. That being said, this changes in C23 (aka ISO/IEC 9899:2024, released in October 2024) to be more in line with C++ / move away from K&R C.

Compiling

I'm developing on a host system that is not QNX (Ubuntu) and I will therefore have to "cross-compile" the code: using a compiler that runs on Ubuntu that is configured to create binaries for a different OS (QNX).

If we open up a terminal, we can issue the command:

qcc -o helloworld helloworld.c

qcc is the QNX C compiler (qcc), which is based on the GNU C compiler suite, configured to have a few QNX-specific defaults. (I'm betting the QNX Tools Team has a lot more to say about that – are you interested in a post about this?) With this command, we're saying "compile the file helloworld.c and output a file named helloworld (-o helloworld)".

I've cheated a bit here by taking advantage of the fact that qcc will default to compiling to Intel 64-bit when you're running on an Intel 64-bit system, which I am. I could have been more explicit, and ended up with the same result:

qcc -Vgcc_ntox86_64 -o helloworld helloworld.c

What did the compiler create for us as a result of the compilation? We can use the POSIX-defined utility named file. This long-existing utility will poke around the insides of a file and look for tell-tale signatures of certain file formats. Let's see what it says about the file our qcc compiler created:

x86_64 $ file ./helloworld
helloworld: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /usr/lib/ldqnx-64.so.2, BuildID[md5/uuid]=5beed135becdcac1a6fffabba73ea189, with debug_info, not stripped
x86_64 $

The important info here is that it's saying the file is (likely) an ELF file, i.e. it adheres to the Executable and Linkable Format (aka ELF) Specification.

Interesting, but, let's see this thing run!

Running Hello, world!

To run it on our QNX system, we clearly have to do 2 things:

• get the file onto the computer being emulated by QEMU, and
• tell QNX to run it after the kernel is initialized.

How do we get it on that computer to make it accessible to QNX? What does it even mean to "be accessible to QNX"?

Uh oh, a Circular Dependency

Last time, we discussed the "initialization script" that is processed after the kernel is initialized, and how it's typically used to execute programs. But, all we did is use the display_msg command, which is a built-in command, i.e. the implementation of display_msg is in the kernel code; there were no programs / ELF executables involved at all.

On a "normal" computer system, the executables are typically going to be on a hard disk drive (HDD), solid-state drive (SSD), or whatever your choice of data-containing device is.

How can we get access to the executables on the HDD/SSD/whatever? We'll need to run some software that can make the contents of that HDD/SSD/whatever "accessible to the system". We need a device driver (some software) that:

• knows how the data is formatted on the HDD/SSD/whatever,
• knows how to present the data to the OS such that the data:
  • can be discovered by programs running on the OS, and
  • can be accessed.

How can programs discover the data? That's where files, directories, and paths come in. A string like /home/mtb/foo/bar.txt is just a way of uniquely identifying where to find some data. It's a key. It's up to the OS to take that string/key/path and figure out which device driver knows how to get to the data/value/file. Then, it's up to the device driver to make the contents accessible.

Ok, great. That means we need to:

• put the file helloworld on an HDD/SSD/whatever, then
• run a device driver so we can
• get access to the file helloworld on that HDD/SSD/whatever at some path.

But, on QNX, a device driver is just another program that has to be run (kinda, more about that next time), so, now we're back to the problem of: How do we run an executable?

The way QNX breaks this circular dependency is to have the QNX kernel itself provide

• a very simple read-only file system, and
• the device driver for that file system, so that the very simple file system is accessible to the system (i.e. all running software) at some path.

Remember when we said HDD/SSD/whatever? In this case, 'whatever' is a file system that has all its data in plain old RAM, so no funky hardware involved. Very simple, very easy.

Part of the kernel's initialization is to get the device driver for this file system up and running.

If you can place inside this very simple read-only file system the drivers needed to access HDDs/SSDs/whatever else, then, when the kernel is loaded into the target and initialized, you'll have access to the simple file system, and therefore have access to everything you need to get the rest of the system up and running.

The easiest/best place to put the contents of this file system are inside the image that contains the kernel. This way both the kernel and this file system are loaded into RAM by startup.

You can tell mkifs (make an image with a file system) to add files to this file system, and then mkifs will bundle them into the image file. At runtime, the kernel initializes a device driver that exposes those files.

In QNX parlance, a "device driver" is a kind of "resource manager". There may or may not be an actual "device", but a resource manager makes one or more things (files, directories) available to the system in the path namespace. But, that's a much bigger topic for later.

Modify The Build File To Add Files To IFS In QNX Image

Let's modify the build file we created last time to include helloworld so we can run it.

Last time we had this:

[virtual=x86_64,multiboot] boot = {
    startup-x86 -D 8250
    procnto-smp-instr
}

[+script] init = {
    display_msg "Hello, stranger!"
}

... so now we'll change it to this:

[virtual=x86_64,multiboot] boot = {
    startup-x86 -D 8250
    procnto-smp-instr
}

[+script] init = {
   helloworld
}

/home/yourusername/helloworld/x86_64/helloworld

... where that last line is the absolute path to the helloworld we created above on our host development system.

As we discussed last time, we run mkifs to create a new image.ifs, and then run QEMU. Give that a go, and the output (ignoring some other debugging info) is:

Unable to start "helloworld" (2)

I'm gonna guess that that (2) is an error number that can be found in errno.h! According to errno.h in the SDP, error 2 is ENOENT: "No such file or directory".

What? Did mkifs not add helloworld to the image file system? Let's check by using the dumpifs utility to see what's inside the image file system inside the image file image.ifs:

x86_64 $ dumpifs ./image.ifs
  Offset     Size    Entry Name
  400000      200        0 *.boot
  400200      100     ---- Startup-header flags1=0x21 flags2=0 paddr_bias=0
  400300    2f048   402008 startup.*
  42f348       5c     ---- Image-header mountpoint=/
  42f3a4      108     ---- Image-directory
    ----     ----     ---- Root-dirent
  430000    e9000     ---- proc/boot/procnto-smp-instr
    ----     ----     ---- proc
    ----     ----     ---- proc/boot
  519000     2000     ---- proc/boot/helloworld
  42f4ac       28     ---- proc/boot/init
  51b000       44     ---- Image-trailer

So mkifs did put helloworld in there, and it put it at /proc/boot/helloworld.

Maybe QNX can't find helloworld? Let's try telling QNX how to find helloworld using the full/absolute path to the file:

[virtual=x86_64,multiboot] boot = {
    startup-x86 -D 8250
    procnto-smp-instr
}

[+script] init = {
   /proc/boot/helloworld
}

/home/mtb/helloworld/x86_64/helloworld

Rebuild and run it again to get:

Unable to start "/proc/boot/helloworld" (83)

Well, that's a kind of progress, I guess. A new error number! According to errno.h, 83 is ELIBACC, "Can't access shared library". What shared library does helloworld need, and how do we even ask?

We saw before that helloworld is an ELF file. Are there any ELF utilities that can tell us more about it? We're in luck: the readelf utility is one way of getting more info.

Using the -d option, we can see the information of the "dynamic section" of the ELF. More specifically, we can see if the ELF needs any shared libraries to be loaded at runtime/dynamically.

x86_64 $ readelf -d ./helloworld | grep NEED
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]

Oh! libc. Of course an executable created with C needs the C language execution library.

But, what's the other thing, this libgcc thingie? This library is known as the "GCC low-level runtime library" and is also provided with the QNX SDP. If we look inside we see it has quite a few things in it

lib $ readelf -s libgcc_s.so | grep GLOBAL | grep FUNC | wc -l
305

It has 305 functions! This library includes compiler-provided functions. A simple example is:

int __popcountsi2(unsigned long a);

... which returns the number of bits set in the parameter a. Since qcc is based on gcc/GCC, that dependency makes sense.

Ok, fine, let's update our build file:

[virtual=x86_64,multiboot] boot = {
    startup-x86 -D 8250
    procnto-smp-instr
}

[+script] init = {
   /proc/boot/helloworld
}
libc.so.6
libgcc_s.so
/home/mtb/helloworld/x86_64/helloworld

and the output is, TADA!

Unable to start "/proc/boot/helloworld" (83)

It still gives the same error?!

Hm.

Remember when we asked the file utility about helloworld? It actually gave us a hint as to what we're missing.

If we look at the contents of helloworld again, specifically the program headers, we will see (and I've punted a bunch of stuff):

x86_64 $ readelf --program-headers ./helloworld

Program Headers:
  INTERP         0x0000000000000200 0x0000000000000200 0x0000000000000200
                 0x0000000000000017 0x0000000000000017  R      0x1
      [Requesting program interpreter: /usr/lib/ldqnx-64.so.2]

The compiler put into the helloworld ELF a note to the operating system/QNX saying, "If you want run this, the file ldqnx-64.so.2 knows know to INTERPret the contents of this file to get it up and running in a process."

Let's include that file too.

[virtual=x86_64,multiboot] boot = {
    startup-x86 -D 8250
    procnto-smp-instr
}

[+script] init = {
   /proc/boot/helloworld
}

libc.so.6
libgcc_s.so
ldqnx-64.so.2
/home/mtb/helloworld/x86_64/helloworld

And then? Tada, maybe?

Unable to start "/proc/boot/helloworld" (83)

Same error.

Oh, wait. Paths. The INTERP stuff inside helloworld is saying that ldqnx-64.so.6 must be loaded specifically at the absolute path /usr/lib/ldqnx-64.so.2. But, this is a minimal system, and we only have files in the IFS available, and mkifs put ldqnx-64.so.6 in /proc/boot:

x86_64 $ dumpifs ./image.ifs
  Offset     Size    Entry Name
...
  519000    3a000     ---- proc/boot/ldqnx-64.so.2

We need to somehow tell QNX that if someone asks about /usr/lib/ldqnx-64.so.2, point them to /proc/boot/ldqnx-64.so.2. i.e. we have tell QNX to create a symbolic link. I know the POSIX ln utility can create symbolic links, but, how can we run an ln program when we can't even run helloworld?

Fortunately, there is a way to solve this circular dependency in the initialization script:

[virtual=x86_64,multiboot] boot = {
    startup-x86 -D 8250
    procnto-smp-instr
}

[+script] init = {
   procmgr_symlink /proc/boot/ldqnx-64.so.2 /usr/lib/ldqnx-64.so.2
   /proc/boot/helloworld
}

libc.so.6
libgcc_s.so
ldqnx-64.so.2
/home/mtb/helloworld/x86_64/helloworld

procmgr_symlink is – like display_msg – a built-in command that the QNX kernel provides. This is basically saying, "Hey, QNX! If someone asks about /usr/lib/ldqnx-64.so.2, just redirect them to /proc/boot/ldqnx-64.so.2" (and more).

Make the image file, run QEMU, and...

Hello, world!

Success! Finally! 🎉

An Aside on mkifs Shortcuts

For completeness, I'll mention that there's an mkifs feature:

[autoso=add]

... which will cause mkifs to search the binaries to be added to the IFS the same way we did above to find what's NEEDed, and automatically add those libraries to the IFS.

[virtual=multiboot] boot = {
    startup-x86 -D8250
    procnto-smp-instr
}

[+script] init = {
    procmgr_symlink /proc/boot/ldqnx-64.so.2 /usr/lib/ldqnx-64.so.2
    /proc/boot/helloworld
}

[autoso=add]
/home/mtb/helloworld/x86_64/helloworld

If we try running this, it doesn't work because mkifs only looks at the NEEDED entries in the ELF; it does not look at the INTERPreters required. mkifs assumes you've added ldqnx, because you won't get far without it.

[virtual=multiboot] boot = {
    startup-x86 -D8250
    procnto-smp-instr
}

[+script] init = {
    procmgr_symlink /proc/boot/ldqnx-64.so.2 /usr/lib/ldqnx-64.so.2
    /proc/boot/helloworld
}

ldqnx-64.so.2
[autoso=add]
/home/mtb/helloworld/x86_64/helloworld

And..

Hello, world!

You can use another feature of autoso, list:

[autoso=list]

... to have mkifs list the dependencies, so you can

• know what they are, and
• explicitly add them yourself.

Example output:

Missing shared libraries:
proc/boot/libc.so.6=/home/mtb/sdp/target/qnx/x86_64/lib/libc.so.6
proc/boot/libgcc_s.so.1=/home/mtb/sdp/target/qnx/x86_64/lib/libgcc_s.so.1

By default, mkifs chooses the path /proc/boot as the "mount point" for the image file system, but, if you want to, you can change that with the prefix and mount attributes.

Mr. Dressup's Tickle Trunk always seemed to have everything you need, so, let's use /tickletrunk as the mount point for the IFS and make the necessary adjustments.

[prefix=""]
[mount="/tickletrunk"]

[virtual=multiboot] boot = {
    startup-x86 -D8250
    procnto-smp-instr
}

[+script] init = {
    procmgr_symlink /tickletrunk/ldqnx-64.so.2 /usr/lib/ldqnx-64.so.2
    /tickletrunk/helloworld
}

ldqnx-64.so.2
libc.so.6
libgcc_s.so

/home/mtb/helloworld/x86_64/helloworld

And let's see the insides of the IFS:

x86_64 $ dumpifs ./image.ifs
  Offset     Size    Entry Name
  400000      200        0 *.boot
  400200      100     ---- Startup-header flags1=0x21 flags2=0 paddr_bias=0
  400300    30048   402008 startup.*
  430348       5c     ---- Image-header mountpoint=/tickletrunk
  4303b0      1d4     ---- Image-directory
    ----     ----     ---- Root-dirent
  431000    e9000     ---- procnto-smp-instr
  61f000     2000     ---- helloworld
  430584       9c     ---- init
    ----        d     ---- ldqnx-64.so -> ldqnx-64.so.2
  51a000    3a000     ---- ldqnx-64.so.2
    ----        9     ---- libc.so -> libc.so.6
  554000    ad000     ---- libc.so.6
    ----        d     ---- libgcc_s.so -> libgcc_s.so.1
  601000    1e000     ---- libgcc_s.so.1
  621000       44     ---- Image-trailer

Does it work? You bet.

Hello, world!

Coming Up...

Now that we can run something we made with a C compiler, we'll look around this minimal system we have to see what's available by writing our own version of the ls utility and using it to inspect what files are available. Yes, we could use the one provided by the SDP, but what fun is that?

And, I alluded last time to the "kernel process" when talking about the idle threads, and said the kernel runs a device driver / resource manager for the IFS. We'll finally get into that a bit.

Make sure to subscribe to get the next post!