27 KiB
bob64 migration status
This document tracks the separate x86-64 OS target. bob16 and bob32 are
software-emulated custom instruction sets; the current bob32 kernel is not an
x86 kernel and cannot be promoted to x86-64 by changing C integer widths. The
new target starts as an x64 UEFI application. Its opt-in handoff now takes
ownership of paging, early memory allocation, display output and a minimal shell;
the existing emulator/build outputs stay separate.
Implemented stages
bob64/types.hdefines fixed-width scalar types and pointer/size types from compiler-provided x86-64 definitions, with compile-time width checks.bob64/boot.cbuilds as a PE32+ AMD64 EFI application at the standard removable-media pathEFI/BOOT/BOOTX64.EFI. The image includes a PE base relocation directory andDYNAMIC_BASE, so firmware may load it away from its preferred0x100000image base.- x64 UEFI transfers control in 64-bit long mode with its x64 ABI, stack and
page tables. The entry checks the maximum CPUID leaves before reading feature
bits, requires PAE and long-mode support, and confirms
EFER.LMAbefore it printsbob64!through UEFI text output.bob64/cpu.calso decodes NX support and the CPU's physical-address width for paging setup. The safe probe returns through firmware; the handoff variant switches to bob64-owned tables and exits Boot Services. - The boot entry reads the UEFI memory map through the x64 Boot Services table.
bob64/memory.cparses the firmware-provided descriptor stride and seeds a sorted, coalesced first-fit allocator of 4 KiB conventional physical pages, keeping physical addresses inu64, clipping ranges to the detected CPU physical-address width, and reserving the first MiB. Allocation records make invalid and double frees fail; the independent live-allocation ledger supports up to 16,384 ranges, while free extents have a separate 128-range limit. Boot reserves the replacement stack and table pool from this allocator, and the kernel continues allocating from the preserved state. - The x64 UEFI ABI is Microsoft's x64 ABI: integer/pointer arguments use
RCX,RDX,R8, andR9; integer/pointer returns useRAX;RSPis 16-byte aligned at call sites with 32 bytes of caller-provided shadow space. The compiler data model is LLP64 (char8,short16,int32,long32,long long64, pointer/size_t64). bob64 code should useu64/s64andusize/isizefor explicit widths rather than assuminglongis 64-bit. bob64/paging.cbuilds four-level x86-64 page tables with 4 KiB leaves. Allocate/access callbacks keep physical page addresses distinct from currently accessible pointers; the builder enforces the CPUID-reported physical-address width.bob64_page_map_rangevalidates canonical spans and maps byte ranges while preserving partial-page offsets.bob64_page_protecttightens leaf permissions andbob64_page_unmapreturns the removed physical frame and flags; callers invalidate the TLB for live edits. The host suite walks low and upper canonical virtual addresses and physical/table addresses above 4 GiB, and verifies writable pages can become read-only/NX, be unmapped, and be mapped again. The handoff activates the root after enablingEFER.NXEwhen supported; host tests cover mapping logic, while the privileged switch still needs UEFI runtime verification.bob64/bootstrap.ccomposes that mapper into a bootstrap identity address space for the loaded EFI image, a replacement stack, and a pre-reserved contiguous page-table pool. It marks stack/pool pages NX when supported and rejects overlapping ranges or an undersized pool. Tests verify translation and permissions for all three regions. The EFI entry obtains the loaded-image protocol and uses its full-width image base and size to prepare and verify identity mappings for the actual image, an 8-page replacement stack, and a 128-page table pool. The handoff also maps its memory-map buffer and supported GOP framebuffer. It builds the kernel GDT and exception IDT while the image is mapped. The default--bob64image still returns safely to firmware after preparation.bob64/descriptors.cbuilds kernel and DPL3 user code/data GDT entries, a full-width 64-bit TSS descriptor, the packed 10-byte GDTR/IDTR operand, and 16-byte x86-64 IDT gates with split 64-bit handler addresses. Boot sets TSSRSP0to the initial kernel stack, and the handoff loads the task register. After heap startup, the kernel changesRSP0to a separate supervisor-only privilege-transition stack so user interrupts will not overwrite the paused kernel call stack. Each shell-launched application now gets its own zeroed 16 KiB privilege stack; the previous TSS stack is restored before that allocation is released. Tests verify selector values, TSS offsets/base/limit, full-width stack replacement, alignment, DPL3 gates, IST, and gate type.bob64/interrupts.Scontains 64-bit stubs for CPU exception vectors 0-31, normalizing hardware error-code and no-error-code frames before saving all general-purpose registers and calling the Microsoft x64 C dispatcher. The common path restores registers and usesiretq.interrupts.cwires an IDT to those stubs and a fatal fallback, and prints full 64-bit register values over COM1 and the framebuffer before halting. The opt-in handoff loads the IDT. The DPL3 vector0x80enters a dedicated syscall stub; runtime exception and syscall delivery still need firmware/emulator testing.bob64/kernel.cis the first kernel-owned C entry point. It checks the boot-information structure, active CR3, memory-map range, and stack, then reports the 64-bit root and map details, checks the growable kernel heap, then starts the polled shell. The filesystem and shell are still bring-up services, not yet a general-purpose OS.bob64/handoff.Sprovides the Microsoft x64 ABI entry bridge: interrupts are disabled, the kernel GDT is loaded and CS reloaded, the IDT is loaded, NXE and CR0.WP are enabled when supported, the TSS is loaded, CR3 is switched, and execution moves to the replacement stack before calling the kernel. The EFI boot path callsExitBootServicesusing a freshly acquired map key, retrying a stale key up to three times.- The handoff is opt-in so the original EFI probe stays available:
build-tool --bob64-handoffemits a distinct image andbuild-tool --bob64-handoff-testruns host PE checks against it. Those checks cannot prove the privileged transition works on real firmware. - The handoff queries UEFI GOP and maps a supported RGB/BGR framebuffer into
the new address space as writable, non-executable, uncached memory. The
kernel's
bob64/console.cdraws a compact 5x7 text font, wraps and scrolls, and mirrors output to COM1. Unsupported or invalid GOP modes retain the serial console fallback. Host tests exercise format encoding, glyph output, and invalid framebuffer bounds; actual display output still needs UEFI testing. bob64/heap.cprovides a 16-byte-aligned, coalescing kernel heap withalloc,calloc, andfree. Kernel entry adopts the page allocator and page-table pool seeded by boot, then grows the heap by mapping physical 4 KiB pages into a 64 MiB high-half range at0xffff900000000000. New pages are zeroed before use. Kernel entry smoke-tests multi-page allocations, alignment, zeroing, writes, frees, and double-free rejection. Host tests cover splitting, coalescing, overflow, and invalid frees. This early heap is single-core and does not reclaim empty pages yet.bob64/keyboard.cdecodes PS/2 set-1 input, including shift, caps lock, editing keys, and Pause/extended sequences. IRQ1 now feeds a bounded 64-event queue used by the shell and app event service after the kernel remaps the PIC; the kernel keeps polling as a fallback if the controller is unavailable. IRQ12 feeds a bounded mouse-event queue through the same app event service; COM1 remains polled. Unit tests cover scan-code state, keyboard/mouse queue behavior, and shell command/edit behavior. A 100 Hz PIT IRQ now supplies a monotonic tick counter through the versioned syscall ABI. USB keyboards, history and completion remain future work.bob64/filesystem.cadds a flat, volatile binary-safe filesystem backed by the kernel heap. It supports 64-bit lengths, case-sensitive names, atomic file replacement, listing, reads, and deletion; names reject path separators and characters outside the safe filename set. The shell exposesls,cat NAME,write NAME TEXT, andrm NAME. Host tests cover replacement, invalid names, binary NUL round-trips, and shell workflows.bob64/snapshot.cdefines B64S version 1 as a pointer-free little-endian format with 64-bit total/data lengths and a CRC-32. Restore validates version, size, checksum, record bounds, names, and duplicate files before replacing the live filesystem. The shell'ssaveandrestorecommands keep one checkpoint in the kernel heap for the current boot session; it is lost on reboot. Host tests cover round-trip, empty files, replace-on-restore, truncation, unsupported versions, checksum errors, and unchanged state after rejected input. Disk persistence and directories remain future stages; the app filesystem ABI supports bounded whole-file operations and ABI v10 streaming handles with 64-bit seek positions and bounded reads/writes; hardware-backed persistence remains outstanding.bob64/exec.cdefines B64E version 1, a flat position-independent 64-bit executable image with a versioned ABI, exact payload length, 64-bit memory size and entry offset, page-aligned code/data boundary, payload CRC-32, and zero-required reserved fields. The shared parser rejects malformed/truncated images before returning a view; a contiguous copy helper preflights capacity and address overflow, copies the image, clears the memory-only tail, and preserves entry addresses above 4 GiB. Mapped application loading is provided separately byprocess.c. A synthetic B64E app now runs at CPL3, writesbob!through syscall v2, performs a bounded RAM-file round trip, and exits back to the saved kernel stack.bob64/abi.hstarts application ABI v1 on the same Microsoft x64 calling convention used by the UEFI kernel: integer/pointer arguments in RCX, RDX, R8, and R9; scalar status in RAX; 32-byte shadow space; 16-byte call-site stack alignment; and no red zone. Entry receives one pointer to a fixed 32-byte startup record containing a 64-bit argument count and argv pointer. The record and its strings are app-space data, so the contract does not pass kernel pointers. Tests check the record layout, entry call, nested calls, and stack-passed arguments. Syscall ABI v10 defines an ABI query, character and bounded buffer output, file reads and writes, pixel-display query/present, blocking keyboard and PS/2 mouse events, and process exit through interrupt vector0x80; user pointers are validated against the active process root before buffer reads or writes. Display apps submit a packed 0x00RRGGBB frame matching GOP dimensions; the kernel copies bounded chunks into the device surface without exposing its address. Tests cover the syscall and both GOP channel formats.apps/bob64_display.cexercises the app-facing wrapper and is packaged as a B64E image with a bounded 1024x768 static surface; runtime execution awaits UEFI verification.bob64/window.hports the reusable window-manager behavior to app-owned fixed-capacity 64-bit state: nonzero 64-bit handles, full-width context pointers, visibility, focus, z-order, hit testing, title-bar dragging, clamping, dirty redraws, and keyboard/mouse event routing to window-local coordinates. Host tests exercise overlap redraw, wheel routing, focus and pointer values above 4 GiB.apps/bob64_gui.cis the first native desktop on this manager. It opens a named RAM text file (ornotes.txtby default), lists RAM files in a separate window, supports selection/open/refresh and confirmed deletion, plus insertion, line breaks, cursor movement, click placement, scrolling, Ctrl+S save and an unsaved-exit prompt. It validates flat filenames and protects an existing file over the current 4 KiB syscall limit from accidental overwrite. The shell launches it withrun desktop.b64e [filename]; host tests cover the editor buffer, packaged UI strings, file ABI and window manager. The full existing bob GUI, a file picker and real UEFI interaction remain to be ported/verified.bob64/ABI.mdrecords the register-preservation rules, stack alignment and shadow-space contract, pointer ownership, and application entry shape. The compiler's aggregate behavior follows the Microsoft x64 ABI; aggregate returns are not part of the application-entry contract.bob64/app.hexposes syscall wrappers for native C apps without kernel-internal types. The smallbob64/libc.cruntime provides memory and string operations with 64-bit sizes.bob64/compiler.cis the first resident x86-64 code-generation stage. The shell exposescc bob.c [OUTPUT], which packages the accepted source directly as B64E;run app.b64elaunches it. The current syntax acceptsintorlong,long long, orint main(void), initializedint/long/long longlocals and assignments, boundedbob64_app_writecalls with string literals, and 64-bit integer return expressions with parentheses, unary minus, addition, subtraction, and multiplication. Its frame and entry wrapper follow the Microsoft x64 stack contract; syscall ABI v10 provides exit, graphics output, whole-file and streaming file I/O/list/delete, and input. The compiler recognizesusize/uintptr_tas unsigned 64-bit scalar types andisize/intptr_tas signed 64-bit scalar types in declarations and function signatures. These are built-in type names; typedefs and header inclusion remain unsupported.usize *,uintptr_t *,isize *,intptr_t *, andlong long *map to signed/unsigned wide-pointer types with eight-byte loads, stores, indexing and scaled pointer arithmetic. They work as local, parameter, return, global, and struct-field types; global pointers may target matching scalar objects. Fixed-size signed and unsigned 64-bit arrays now support local and global storage, initialization, indexing, mutation, array parameter decay, and eight-byte element addressing. Local arrays are limited to 16 elements by the fixed function frame; global arrays share the 4 KiB data page. Function declarations and definitions can returnchar *,short *,int *, and pointers to declared structs. Returned pointers keep all 64 bits; return expressions must match the declared pointee type. Returning a pointer throughlong long/usizeremains accepted for the compiler's existing raw-address tests. Host execution tests cover pointer identity through a prototype and a returned string literal used by the caller.bob64/process.cadds a callback-backed B64E loader core. It reparses the exact file before allocation, maps up to 16 MiB of app memory at0x0000004000000000, keeps code user-readable/executable and read-only, and maps data, BSS, and a 64 KiB stack writable/user/NX. It leaves an unmapped guard page below the stack, builds a 64-bit argv array and startup record in app memory, and prepares an ABI-aligned entry stack with shadow space. Failed loads roll back mappings and frames; unload removes each mapping before returning its physical frame. Host tests use physical frames above 4 GiB and verify argument contents, permissions, zero-filled BSS, guard placement, and rollback. The kernel wires these callbacks to its real page allocator and a private cloned root with a 34 MiB isolated app arena. At boot, the synthetic user app printsbob!and exits through the controlled syscall path. The shell'srun NAME [ARG ...]command now reads B64E files from the RAM filesystem and launches them with app-owned arguments;bob.b64eis seeded as an example. General app services and persistent storage remain ahead.build-tool --bob64builds only the new EFI image.build-tool --bob64-testalso validates the PE machine, PE32+ header, EFI application subsystem, 64-bit type sizes, synthetic pointer arithmetic above 4 GiB, and the presence of base relocations for firmware-selected load addresses. It also exercises four-level mapping, translation, permissions, canonical-address checks, physical-width limits in both allocator and page tables, multi-page range mapping, protect/unmap/remap behavior, duplicate-map rejection, bootstrap identity mappings, synthetic CPUID feature decoding, 64-bit GDT/IDT layouts, and exception-stub wiring.
Still required by the migration goal
- Boot the handoff image under x64 UEFI emulation. QEMU/OVMF verified the
long-mode entry,
ExitBootServices, owned CR3, kernel heap, framebuffer console, COM1 diagnostics, ring-3 smoke app, and shell. A runtime script repeats the boot and checks its serial milestones. - Exercise ring-3 syscalls and user exception recovery after loading the GDT/IDT and new CR3. A deliberate invalid opcode was diagnosed with the full register frame and returned to the kernel; the corrected smoke app now exits normally. The IDT now also contains PIT, PS/2 keyboard, and PS/2 mouse IRQ handlers; runtime IRQ-path verification remains pending below.
- Verify actual ring-3 text protection and data NX behavior. QEMU runtime probes attempt to write the read-only executable page and execute the NX data page; both receive the expected user page-fault error bits and return to the kernel. The kernel also checks that the stack guard is unmapped.
- Add dynamic page-table-pool growth, heap locking, and reclamation of empty heap pages. The bootstrap identity map favors bring-up simplicity.
- Verify GOP, COM1, PS/2 keyboard, and mouse input under UEFI emulation. QEMU showed mouse movement and a click selecting a file-list row; keyboard input launched/closed the desktop and created a third window.
- Verify the new IRQ1 keyboard and IRQ12 mouse paths under UEFI emulation. Host tests cover the IDT gates and bounded event queues; QEMU verification is pending.
- Add a 100 Hz PIT timer IRQ, tick counter, and user tick query. The timer tick path is covered by host syscall tests; runtime QEMU verification remains pending alongside keyboard/mouse IRQ checks.
- Add USB keyboard support and command history/completion. COM1 remains polled.
- Add persistent storage for B64S snapshots. ABI v10 now supplies bounded handle-based streaming reads/writes, 64-bit absolute seeks, create/truncate/ append modes, and per-app handle cleanup; physical persistence remains open.
- Port remaining device services and add USB keyboard support.
- Add typed pointer returns to the resident x86-64 compiler for
char *,short *,int *, and declared struct pointers. Tests cover prototype calls, string literal and struct pointer returns, and rejection of incompatible pointee types. - Add
voidhelper functions and scalarcharreturns to the resident compiler. Void helpers accept explicitreturn;or fall through, can be called as statements, and are rejected in value contexts. Scalarcharreturns sign-extend from eight bits; main remains required to return a value. - Add typed pointers to
usize/uintptr_tandisize/intptr_tvalues. Local/global/struct-field storage, parameters, returns, dereference, indexing, assignment, and 8-byte-scaled arithmetic are covered, including synthetic pointer values above 4 GiB. - Add fixed signed and unsigned 64-bit arrays with local/global storage,
initialization, indexed reads/writes, and array-parameter decay. Local arrays
are bounded to 16 elements and global data to the existing 4 KiB page.
File-scope
short,int,long,char, andlong longscalars plus bounded fixedshort/int/chararrays now use initialized writable B64E data and RIP-relative x64 access across functions. Globalshort */int */char *initializers may target previously declared arrays or string literals. Named structs define aligned scalar/pointer layouts and support local/global instances plus./->field loads/stores. Struct arrays, nested structs, typedefs, and multiple translation units remain unsupported. Limits are 8 struct types, 16 fields per type, 16 global objects, and the single 4 KiB data page. Local fixedshort/intarrays,short */int *locals, dereference/indexing, scaled pointer arithmetic and pointer differences, and local brace initializers are implemented.short(16-bit),int/long(32-bit), andlong long(64-bit) helper functions and matching prototypes before or aftermainsupport nested/forward calls with up to sixteen scalar,short */array,int */array,char *,usize *,isize *, or struct-pointer parameters, following the Microsoft x64 register and stack argument layout. Compound blocks andforinitializers now have lexical local scopes with shadowing and stack-slot reuse after scope exit. NUL-terminated string literals, signed 8-bitcharlocals/parameters, escaped character constants, byte-scaledchar *indexing, stores, dereference, and arithmetic, plus localchararrays with string or brace initializers, now support resident C string traversal and mutable text buffers. Struct pointer parameters can be passed through helper calls using the Microsoft x64 argument ABI; helpers can read and write fields through->.if/elseandwhilesupport signed relational/equality comparisons, and expressions support unary!plus short-circuit&&/||. Basicforloops now emit initialization, condition, body, and scalar assignment update code.breakandcontinuepatch to the nearest loop;forcontinue enters its update clause. Earlyreturnstatements work inside branches and loops; the subset still requires a final return at the end of each function. - Add a kernel-owned per-app window manager and port the desktop's file and editor windows to its surfaces. Syscall ABI v10 creates, focuses, destroys, validates and copies app surfaces, composites z-order, routes input with 64-bit handles and local/screen coordinates, and releases the session on app exit. Concurrent apps are still not displayed together.
- Run the native desktop under QEMU/OVMF at 1280x800. Both managed windows rendered; keyboard input created a third window, Escape returned to the shell, and Ctrl+S saved editor text through the file syscall. The full screen and B64S shell commands were also exercised in the same UEFI session.
- Port the remaining applications to the kernel window API. Syscall ABI v10
also provides bounded file reads/writes/list/delete, console I/O,
full-screen presentation, blocking PS/2 keyboard/mouse events, ABI query, and
process exit. The host
bob64ccbridge continues to package the broader C subset supported by GCC. - Port the remaining existing applications and full GUI/widget behavior; the current native text editor is an initial, limited desktop milestone.
- Run the kernel-generated ring-3
bob!app under x64 UEFI emulation. The resident compiler also compiled and launchedbob.c; B64Ssave/restorerecovered a deleted file, and the native GUI opened and saved an editor file. - Complete the full acceptance flow and add runtime tests for broader calling-convention/stack cases, hardware interrupts, mouse input, and persistent B64S storage. The ring-3 app image runs at a virtual address above 4 GiB; host tests also cover synthetic high addresses and broad ABI/memory cases.
- Verify bob64 boots in x64 UEFI emulation and performs its own kernel initialization without executing 32-bit kernel or app code. The broader migration remains active: the resident C subset, devices, persistence, and existing application ports still need work.
Build and test
build-tool --bob64
build-tool --bob64-test
build-tool --bob64-handoff
build-tool --bob64-handoff-test
tools/test_bob64_qemu.ps1 -QemuPath <qemu-system-x86_64.exe> -OvmfCodePath <edk2-x86_64-code.fd>
The safe output goes to build/bob64/EFI/BOOT/BOOTX64.EFI; the opt-in handoff
image goes to build/bob64-handoff/EFI/BOOT/BOOTX64.EFI. Host tests validate
the PE/COFF architecture and subsystem, 64-bit pointers, synthetic addresses
above 4 GiB, memory-map descriptor strides, page allocation/coalescing,
allocation-ledger capacity, invalid free handling and overflow checks. The
PowerShell QEMU test performs a real OVMF boot of the handoff image and checks
serial markers through the kernel shell; it needs the QEMU executable and
edk2-x86_64-code.fd paths supplied by the caller. The full script does not
replace testing on physical UEFI hardware.
Memory layout boundary
The current owned paging stage uses four-level tables. The image, bootstrap
stack, table pool, memory map, and supported GOP framebuffer are mapped at
identity addresses in the lower canonical half. Physical addresses below 1 MiB
remain reserved. The early heap occupies a 64 MiB window beginning at
0xffff900000000000. The B64E loader reserves a lower-half user image arena of
16 MiB beginning at 0x0000004000000000; its 64 KiB user stack begins at
0x0000004002001000, with an unmapped guard page at 0x0000004002000000.
The kernel smoke test clones the active root, privately copies the page-table
branch covering a 34 MiB image-and-stack arena, clears stale mappings in that
arena, and switches CR3 before loading and unloading a synthetic app. Other
kernel mappings remain shared. The kernel enters the synthetic app at CPL3
using an iretq frame. The app can query the syscall ABI, print characters
through int 0x80, and exit through a controlled return to the saved kernel
stack. The user-entry bridge enables interrupts for apps and restores the
caller's prior IF state when the app exits. The boot smoke app checks that the
keyboard IRQ path remains enabled after a user process returns. The versioned
syscall ABI also includes file handles and window services.
The fallback kernel privilege stack and each app's dedicated 16 KiB interrupt
stack are allocated from the supervisor-only heap. App runs also push a bounded
syscall context that preserves the parent's callbacks, user page table, scratch
buffers, and file handles; each app owns a separately allocated window-server
session. The user-entry return bridge still uses singleton state, so nested app
execution is not enabled yet.
The volatile
filesystem and B64S checkpoint live in the kernel heap; there is no full
physical direct map or filesystem mapping.