Welcome to the blog series "From The Board Up" by Michael Brown. In this series we create a QNX image from scratch and build upon it stepwise, all the while looking deep under the hood to understand what the system is doing at each step.

With QNX for a decade now, Michael works on the QNX kernel and has in-depth knowledge and experience with embedded systems and system architecture.

Virtual Addresses, Again?

Just a quick follow up and detour on virtual addresses.

We said last time that, with QNX 8, we use a 64-bit virtual address space, and therefore a virtual address can be anywhere from 0x0000'0000'0000'0000 to 0xFFFF'FFFF'FFFF'FFFF.

Yes, in theory, I can create a pointer with any of the 18 quintillion possible values:

// seed the random number generator
srandom(ClockCycles());

// random() only returns a 31-bit value
// https://pubs.opengroup.org/onlinepubs/9799919799/functions/initstate.html
// https://www.qnx.com/developers/docs/8.0/com.qnx.doc.neutrino.lib_ref/topic/r/random.html?hl=random
const uint64_t v0 =  ((uint64_t)random())         << (0 * 31); // bottom 31 bits
const uint64_t v1 =  ((uint64_t)random())         << (1 * 31); //
const uint64_t v2 = (((uint64_t)random()) & 0b11) << (2 * 31); // top 2 bits

const uint64_t value = v2 | v1 | v0;

printf("0x%16.16" PRIX64 "\n", value);

I ran this and got this:

0x5B5070847D663186

Is that a valid virtual address? i.e. can I ever have a pointer to this virtual address that will ever let me successfully read from or write to this address? (or fetch an instruction from it)

No.

Why not?

Because money.

Can you find a computer currently for sale that supports 2^64 bytes of memory? i.e.

>>> (1 << 64) / 1024 / 1024 / 1024 / 1024 / 1024 / 1024
16.0

16              kilo   mega   giga   tera   peta   exa

Is there a computer anywhere that lists 16 EB of RAM? i.e. 17,179,869,184 GB of RAM?

Nope.

Therefore, people who design CPUs said "64 bits is nice, but, nobody can use them all right now. I'm not going to waste precious silicon supporting every. single. bit."

Fewer mm^2 per die means higher yield per wafer means higher profit$$$.

And so they said "64-bits, but." Specifically, "But some of the bits must all be 0, or all be 1." How many bits? Depends on your system, and how you configure it.

Since we're using QEMU for 64-bit Intel, let's see what the Intel Software Developer's Manual (SDM) says. In SDM Volume 3, it says that "bits 63:47 of the address are identical". (Note: In case you're following along with the SDM at home, this is for the MMU configuration that QNX 8 uses, which is 4-level paging)

Let's look at those bits and see what's allowed:

6666 5555 5555 5544 4444 4444 3333 3333 3322 2222 2222 1111 1111 1100 0000 0000  Bit position
3210 9876 5432 1098 7654 3210 9876 5432 1098 7654 3210 9876 5432 1098 7654 3210

For all 1:
1111 1111 1111 1111 1xxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx  63:47 identical 1

1111 1111 1111 1111 1111 1111 1111 1111 1111 1111 1111 1111 1111 1111 1111 1111  Maximum
1111 1111 1111 1111 1000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000  Minimum

For all 0:
0000 0000 0000 0000 0xxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx  63:47 identical 0

0000 0000 0000 0000 0111 1111 1111 1111 1111 1111 1111 1111 1111 1111 1111 1111  Maximum
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000  Minimum

In summary, there are 2 ranges of virtual addresses that are valid:

• [0xFFFF'8000'0000'0000, 0xFFFF'FFFF'FFFF'FFFF], and
• [0x0000'0000'0000'0000, 0x0000'7FFF'FFFF'FFFF].

Each range is

>>> (0x00007FFFFFFFFFFF + 1) / 1024 / 1024 / 1024 / 1024
128.0

128                            kilo   mega   giga   tera

128 TB.

So, 2 ranges at 128 TB each means the MMU supports up to 256 TB of useful virtual address space.

I think the biggest box I've ever seen for sale on a web site was a server-class jobbie that supported up to 18 TB of RAM. Word around the campfire is that there were boxes back in '19 with 768 CPUs and 22 TB of RAM.

i.e. even this "restricted" virtual address space of 128 TB means there's still several metric lake freighters of future-proofing built in.

Back to our random address, 0x5B5070847D663186. We can see it's not within either of the ranges, because the uppermost 17 bits are neither all 0 nor all 1.

Or, to use the terminology that both Intel and ARM use, the random virtual address we created is "non-canonical". Any virtual address within the two valid ranges is "canonical", or "in canonical form."

For every access performed by a CPU, the MMU will look at the upper 17 bits: if they're not all 0, or they're not all 1, then there's no further work to do because it's a non-canonical address; tell the CPU to handle this bogus virtual address. Intel documentation refers to this as a "canonicality violation".

QNX and Canonical

With 2 canonical regions of 128 TB, how best to use them? If you don't support more than 128 TB, you don't even really need to use both. You could just use one.

But, it's nice to have in-your-face-obvious separation, and there are CPU features that assume use of both, so, QNX uses:

• "all 1s" canonical for kernel code and data, and
• "all 0s" canonical for user code and data.

We saw the latter in action in the the previous article with the virtual addresses we were getting back from mmap(). e.g.

Virtual  address: 0x000000273C77A000

Therefore, around QNX, we (well, I) colloquially refer to:

• "all 1s" as "kernel canonical", and
• "all 0s" as "user canonical".

Can we see the QNX kernel using kernel canonical? Where can we find some virtual addresses the kernel uses? Answer: The system page.

Remember way in the early articles we said that the startup provides a callout for the QNX kernel that allows the QNX kernel to write a character to a UART? That callout code should be in the kernel canonical range.

Let's use pidin to look at the system page's callouts:

/dev # pidin syspage=callout
Header size=0x000000f0, Total Size=0x00000980, Version=2.0, #Cpu=1, Type=4352
Section:callout offset:0x000000f0 size:0x00000068
  reboot:ffff80800000822c watchdog:0000000000000000
  0) display:ffff80800000827a poll:ffff8080000082ba break:ffff8080000082fd
  1) display:0000000000000000 poll:0000000000000000 break:0000000000000000

There's the display_char callout code at 0xffff80800000827a. The ffff8 tells us it's canonical, and for QNX, that's kernel canonical. Ditto for the others, except those which are 0, i.e. NULL, which indicates "not in use".

Yes, But.

The hardware says we have 128 TB of user canonical virtual address available, so, are you saying I can fire up QNX on a system that has 128 TB and mmap() 128 TB?

1. Good luck finding such a machine.
2. Good luck affording to create such a machine.
3. No. Now we're getting into QNX's limits.

And, as is the case with software, results may vary from release to release. What I'll say here is subject to change. It's what I get in my system right now with the version I'm running (QNX 8).

Alright then, let's talk QNX's limits.

Physical Address Space

When it comes to the physical address space, QNX 8 has a limit of 16 TB.

16 TB ought to be enough for anybody.

Virtual Address Space - User Canonical

For each process's virtual address space, the user canonical is restricted by hardware to a range of 128 TB.

If that's the case, I should see pointers being returned by mmap() anywhere between 0x0000'0000'0000'0000 and 0x0000'7FFF'FFFF'FFFF, right?

No, because there's yet another limit. QNX's virtual memory manager (VMM) has a limit on the upper range of user canonical: 0x0000'0080'0000'0000.

>>> 0x8000000000 / 1024 / 1024 / 1024
512.0              kilo   mega   giga

512 GB.

Why the limitation? First, while it is a limit, is it an actual limitation? Second, as with all designs, it's a trade-off between capability, efficiency, and performance. The trade-off is aimed at the majority of expected use cases. If/When the 512 GB "limit" becomes an issue, we'll increase it.

Alright, so that should mean that any address returned by mmap() value could be in the range [0x0000'0000'0000'0000, 0x0000'0080'0000'0000], right?

No.

Movin' On Up

Turns out there's already some stuff up at the top of the range. If we run this code:

    printf("_syspage_ptr = 0x%16.16" PRIX64 "\n", (uintptr_t)_syspage_ptr);

we'll get this:

_syspage_ptr = 0x0000007FFFFFD000

QNX gives every process a (read-only) mapping to the System Page up at the top of the user canonical. The QNX C library, as part of its initialization, initializes the variable _syspage_ptr with the base address of that mapping. Kinda makes sense to put it out of the way.

Don't Bring Me Down

At the bottom of the range, virtual address 0x0000'0000'0000'0000, is kind of special.

In (most versions of) the C specification, it says:

An integer constant expression with the value 0,
or such an expression cast to type void *,
is called a null pointer constant.
If a null pointer constant is converted to a pointer type,
the resulting pointer, called a null pointer,
is guaranteed to compare unequal to a pointer to any object or function.

with a footnote "The macro NULL is defined in <stddef.h> (and other headers) as a null pointer constant"

Aside: No, I'm not getting into the "A pointer is not an integer" thing. I agree: It's not. But, for our purposes here:

• it can be considered a uintptr_t, and
• there's a 1:1 correspondence between a pointer and a vaddr.

Several functions in the standard C library return a null pointer, e.g. malloc(), "The malloc function returns either a null pointer or a pointer to the allocated space."

The C specification also says:

If an object that has static or thread storage duration is not initialized explicitly, then:
— if it has pointer type, it is initialized to a null pointer;

This leads to the common idiom of "If this pointer is a null pointer, I haven't initialized things yet."

There's also the idiom of "If this pointer is a null pointer, use the defaults." e.g. pthread_create() takes a pointer to a pthread_attr_t, attr, such that "If attr is NULL, the default attributes shall be used." And, there's the small variation, "Not available", as we saw above with some of the System Page callouts.

A null pointer, by definition, will not refer to any object or function, so, dereferencing it is a good sign that a mistake is being made. We have two choices:

• treat address 0 like any other virtual address: it's fair game.
• endeavour to ensure that accesses of address 0 cause some specific action.

In the former case, if there's no mapping, an access will cause a SIGSEGV, but if there is a mapping, dereferencing the null pointer, i.e. reading from it or writing to it, will lead to behaviour that can best be mathematically described as "I don't know." Could be good. Could be bad. Who can tell? But my money is on "Not good."

For the latter, we can force any access to generate a signal, SIGSEGV. The default action for that signal is to "abnormally terminate" the process (kill it dead) (and write a core dump file if you're configured for that). That should pretty quickly stop things getting worse.

So, that's what we do. QNX will, by default, not return from mmap() a virtual address between [0, PAGE_SIZE-1], therefore, by default, dereferencing the null pointer / NULL will cause a SIGSEGV to be delivered to the process.

Me First And The Gimme Gimmes

Let's test out these limits by asking mmap() for some plain old RAM at a specific virtual address at the limits.

Looking again at the documentation for mmap(), specifically the first parameter, addr, it says that this parameter can be:

• NULL, meaning "I don't care where it goes in the virtual address space."
• if non-NULL, a hint for where you want it to go, but, if that address doesn't work, anything else is fine, as long as it doesn't overlap an existing mapping.
• if non-NULL and MAP_FIXED is set, then either
  • that virtual address is successfully used for the mapping, or
  • the mapping fails.

This MAP_FIXED flag sounds like a good option for us to try out the limits of user canonical because we can just try mapping at different specific/fixed virtual addresses!

But, there's a gotcha here! Well, a few gotchas mentioned in the "⚠️ CAUTION" box:

1. It requires an ability, PROCMGR_AID_MAP_FIXED. We haven't talked about abilities yet, but, with the configuration we're running with, we have all the abilities. This is great for us while we're playing around, but, you definitely do not want to release a system without abilities configured properly. Much bigger topic that I'm going to ignore/delay for now.
2. "Use MAP_FIXED with caution because it removes any existing mappings, making it easy for a process to corrupt its own address space." i.e. it's saying that if you're saying you want exactly this address in your virtual address space, then, if there's something already mapped there, throw that mapping away (i.e. munmap() it) so that this mapping can be created. That does sound dangerous, because everything in the virtual address space that is currently accessible is there because it's been mapped into the virtual address space, especially your program's code, and data.
3. There's also a caveat that if you're using MAP_FIXED then the virtual address you specify must be page-aligned, i.e. a multiple of the page size. We saw earlier that QNX is using a page size of 4096. This can be confirmed by querying sysconf(_SC_PAGE_SIZE). Let's do that.

The docs say that sysconf() returns a long. This code:

    const long page_size = sysconf(_SC_PAGESIZE);
    printf("sysconf(_SC_PAGESIZE) = %ld (0x%lX) \n", page_size, page_size);

prints this out

sysconf(_SC_PAGESIZE) = 4096 (0x1000)

There are those bottom 12 bits being 0.

For the sake of pedagogy and with a nod to US Navy Admiral Farragut's navigational skill, let's ignore the torpedoes cautions mentioned in the documentation and just create some mappings and see what happens.

Up Top

Well, the system page says it's at 0x0000007FFFFFD000. Can we get the page immediately below it, i.e. at 0x0000007FFFFFC000? With this code:

    const size_t size   = 8; // anything <= 4096
    const int prot      = PROT_READ | PROT_WRITE;
    const int flags     = MAP_PRIVATE | MAP_ANON | MAP_FIXED;
    void* const desired_vaddr = (void*)UINT64_C(0x0000007FFFFFC000);

    uint8_t * const p = mmap(desired_vaddr, size, prot, flags, NOFD, 0);
    assert(MAP_FAILED != p);

    printf(" p = 0x%p\n", p);
    *p = 42;                 // test writing
    printf("*p = %u\n", *p); // test reading

we get this:

 p = 0x7fffffc000
*p = 42

so that worked.

Down Below

I said just above that the first page is reserved so that dereferencing NULL will generate a SIGSEGV, so let's first see that happen with this code:

    uint8_t* const p = NULL;
    *p = 42;

and, I'm going to configure the QNX kernel to be verbose by changing the build file slightly:

[virtual=x86_64,multiboot] boot = {
    startup-x86 -D8250

    procnto-smp-instr -v
}

and when I run the program (canonical_limits) I get this:

Process 3 (canonical_limits) terminated SIGSEGV code=1 fltno=11 ip=0000001a5a6813ab mapaddr=00000000000013ab ref=0000000000000000

SIGSEGV. As expected.

What if I try accessing address 0x666? Not NULL, but still in the first page of the virtual address space.

Process 3 (canonical_limits) terminated SIGSEGV code=1 fltno=11 ip=00000029ae18f3ab mapaddr=00000000000013ab ref=0000000000000666

SIGSEGV. As expected.

Just to make sure we understand what happened here:

• Our code in our process tried to access address 0x666.
• The MMU looked at the address, and sees it's canonical.
• The MMU looked for a mapping, and found none.
• The MMU reported to the CPU: "I have no mapping for address 0x666!", i.e. raised an exception.
• The CPU transferred control to the QNX kernel's exception handler ("This code needs adult supervision!").
• The QNX kernel looked at what happened, and in response to this egregiosity, sent the signal SIGSEGV to our process, as one does when you're a UNIXy system.
• Our process, having not set up an action for this signal, the default action for this signal is to terminate the process ("abnormal termination").
• As there's no dumper utility configured in our image, QNX skips the generation of a core dump and jumps directly to terminating our process.
• Because of the -v option to procnto-smp-instr, the kernel provided some info to /dev/text (and therefore to our 8250 UART) about this abnormally terminated process.

Let's look at the info in that message:

• Process 3 (canonical_limits) terminated is pretty obvious.
• SIGSEGV Uh huh.
• code=1 For SIGSEGV, based on siginfo.h in the section for SIGSEGV, is SEGV_MAPERR "Address not mapped".
• fltno=11 Based on fault.h, fault number 11 is FLTPAGE.
• ip is the instruction pointer, i.e. where the instruction that caused the exception is/was.
• mapaddr is the offset of the faulting instruction within the binary that was mapped in with the code. Deeper topic with ELFs and LOAD segments.
• ref looks like this is the address we were trying to access (refer to / reference) when we got the SIGSEGV, 0x666!

One Way Or Another

Ok, but, what happens if I ask for demand! plain old RAM to be at virtual address 0:

    const size_t size   = 8;
    const int prot      = PROT_READ | PROT_WRITE;
    const int flags     = MAP_PRIVATE | MAP_ANON | MAP_FIXED;
    void* const desired_vaddr = NULL;

    uint8_t * const p = mmap(desired_vaddr, size, prot, flags, NOFD, 0);
    assert(MAP_FAILED != p)

    printf(" p = 0x%p\n", p);
    *p = 42;                 // test writing
    printf("*p = %u\n", *p); // test reading

And run that:

 p = 0x0
*p = 42

It worked!?

So, QNX does not entirely reserve [0, 4095] in the virtual address space. We don't know what your use cases are, so we don't stop you from explicitly asking for it. Maybe you have a perfectly good reason to do this, and are willing to accept the risk that dereferencing NULL will not lead to a SIGSEGV.

But, generally this falls under the category of stupid mmap() tricks. Just because you can, it doesn't mean you should.

This is but one reason why MAP_FIXED requires an ability. The yellow caution tape in the documentation around MAP_FIXED is there for a reason.

Remember how I said right up front I'd be doing some things that you shouldn't do in a production system? Yeah, this.

Mr. Big Stuff

So NULL is special?

Kinda, but no. Any access (read, write, fetch instruction) to any virtual address that does not have a mapping or is non-canonical will cause the MMU to report you to the CPU (generate an exception), and then the CPU will tell on you to the adult in the room, QNX, (invoke the exception handler) and then QNX will deliver a SIGSEGV to your process, as one does. By default, this leads to the abnormal termination of your process, with a core dump if so configured.

Which is a pretty safe setup, really. If you're accessing parts of the virtual address space that you didn't ask for, directly or indirectly, something has probably gone wrong. NULL is one of quintillions of possible wrong accesses, albeit with a slightly higher probability.

Let's remember the scale here. That diagram above of the 64-bit virtual address space is not to scale. Say the entire virtual address space is the length of the Trans-Canada Highway stretching from Beacon Hill Park in downtown Victoria, BC, to Cape Spear, Newfoundland, 5 time zones and 7500 km away. How big is each canonical section?

If Beacon Hill Park's Mile Zero Monument is NULL, and then we start walking, we'll walk to the statue of Terry Fox, walk that distance again, say to that tree at the top of the park, and now we're into non-canonical territory until the parking lot at Cape Spear.

That random address we generated, 0x5B5070847D663186, puts us somewhere near Winnipeg, maybe more like Kenora. i.e. a random address is extremely likely to SIGSEGV.

The question is: How often are rogue addresses truly random vs. say, off-by-one relative to a valid address?

Brother, Can you Spare A Dime?

Spare bits, you say?

Remember how in a previous post we parenthetically mentioned that the alignment requirements mean that all valid pointers have (at least 4 of) the bottom-most bits all 0? We also mentioned how some people use those guaranteed-zero bits to encode extra information in there. The only 'gotcha' being that you have to mask those bits out before you use them.

With this canonical stuff, where we know that a valid address is guaranteed to all 1s or 0s at the top, maybe can we use some of those "spare bits"??

Yes. Software can. And, hardware can too.

If you're interested in an example of this, and are interested in some further reading, I'll point you to the Armv8-A Architectural Reference Manual (ARM).

This glossy brochure is a light read, coming in at just under 15,000 pages. That's about 10 times the length of War and Peace., or, 43 times the number of pages in "Who Has Seen The Wind". Alas, the Armv8-A ARM has a confusing plot and lacks a sympathetic protagonist. 3 out of 4 stars.

If you search that hefty tome for FEAT_PAuth, aka "pointer authentication", it talks about how those "spare bits" can be used to hold a "pointer authentication code" (PAC) to protect against certain security risks. You may also want to look into the GCC aarch64-specific compiler flag -msign-return-address.

Coming up... Let's Get Physical!

This time we detoured into asking for a specific virtual address with the mmap() flag MAP_FIXED. Next time, we'll then look at the mmap() flag MAP_PHYS to ask for a specific physical address. (After a quick detour to look at an implementation of free().)