blaise/docs/rtl-unification-plan.adoc

= RTL Unification — Implementation Plan
:author: Graeme Geldenhuys
:revdate: 2026-06-25
:toc: left
:toclevels: 3
:source-highlighter: rouge
:icons: font
:sectanchors:
:sectnums:

== Purpose

Fold the always-linked runtime (`runtime/`) into the compiler so that a single
`pasbuild compile -m blaise-compiler` produces a self-contained toolchain — no
separate `runtime` module, no `cd runtime && make install`, no `ar`-built
`blaise_rtl.a`.  This is the first concrete step of the "one binary, embedded
RTL, no external archives" end-state recorded in
`docs/native-target-architecture.adoc`.

This document is the execution plan: the architecture facts that constrain it,
the chosen target shape, the full blast radius (every file/script touched), the
ordered migration with a self-hosting re-verify after each stage, and the
cold-bootstrap ordering that must never break mid-change.

== Background facts that constrain the design

These were established by investigation (2026-06-25) and dictate the approach.

. *A normal program `uses` nothing of the runtime explicitly.*  The compiler
  emits calls to `_SetArgs`, `_SysWriteInt`, `_BlaiseInit`, the ARC helpers,
  etc. as *undefined external symbols*.  An `--emit-asm` of a hello-world is
  ~100 lines with zero RTL bodies.  Those externals are satisfied at link time
  by `blaise_rtl.a`.  *The archive is the implicit runtime every binary needs.*

. *`blaise_rtl.a` is a Blaise-only artifact.*  It is assembled by
  `runtime/Makefile` with `ar` from per-unit `.o` files.  PasBuild's `library`
  packaging compiles units (`.o` + `.bif` for unit-path consumption) but does
  *not* emit an `ar` archive — so PasBuild has no native notion of this file.

. *The RTL is now pure Pascal.*  All four hand-written `*_x86_64.s` files were
  migrated to inline-asm `.pas` units, so every RTL unit builds through the
  compiler's own internal assembler.  Nothing external (gcc/as) is needed to
  build the runtime any longer.

. *The compiler already resolves RTL interfaces from the `.pas` on its
  unit-path* (for type/semantic resolution); only the *bodies* come from the
  archive.  The mechanism to compile a unit to an object in-process exists
  (the incremental worker path, `LowerToObject` / `AddOwnedObject`).

. *`stdlib/` stays separate.*  It is opt-in, not always-linked; it is not part
  of this unification.

== Unit relocation: dotted-flat names (recommended) vs subdirectory

The unit loader is *filename-based*: a unit named `X` is sought as
`<searchpath>/X.pas`, and a *dotted* unit name maps to a *dotted filename*
verbatim — `rtl.platform.posix` → `rtl.platform.posix.pas` — NOT to a
subdirectory (`uUnitLoader.pas`: `Path := Base + AName + '.pas'`).  Two
RTL units already use this convention (`rtl.platform*`).

This drives the relocation choice:

[cols="1,3,3",options="header"]
|===
| | Dotted-flat (recommended) | Subdirectory

| Layout
| `compiler/src/main/pascal/runtime.atomic.pas`
| `compiler/src/main/pascal/runtime/blaise_atomic.pas`

| Unit-path change
| *None* — files sit directly in the compiler's existing source dir, already on
  its path.  Best satisfies the "reduce unit paths" goal.
| Needs `compiler/src/main/pascal/runtime` added as a search path — re-introduces
  a unit-path entry, since the loader does not traverse subdirs.

| Unit names
| *Renamed* (`blaise_atomic` → `runtime.atomic`): churns every `uses` clause
  and `.bif`/cache filenames.
| *Preserved*: no `uses` churn.

| Emitted symbols
| *Unchanged* either way — runtime symbols (`_BlaiseGetMem`, `_SetArgs`, …) are
  explicit `external name` strings, independent of the unit name.
|===

Because the loader keys on filename, the dotted-flat scheme reaches the goal
(no extra unit-path) with the trade of a one-time `uses`-clause rename sweep,
whereas the subdirectory keeps a path entry.  *Recommendation: dotted-flat.*
Proposed name map (keep `rtl.platform*` as-is; they already fit):

[cols="2,2",options="header"]
|===
| Today | Dotted-flat

| blaise_start  | runtime.start
| blaise_atomic | runtime.atomic
| blaise_setjmp | runtime.setjmp
| blaise_utf8   | runtime.utf8
| blaise_mem    | runtime.mem
| blaise_str    | runtime.str
| blaise_set    | runtime.set
| blaise_arc    | runtime.arc
| blaise_weak   | runtime.weak
| blaise_float  | runtime.float
| blaise_thread | runtime.thread
| blaise_exc    | runtime.exc
| system        | system (unchanged — implicit System unit)
|===

(Whichever scheme is chosen, the `.o`-cache and link-model changes below are
identical.)

== Target shape (dotted-flat)

[source]
----
compiler/
  src/main/pascal/
    Blaise.pas, uParser.pas, ...           (compiler sources, as today)
    runtime.start.pas  runtime.atomic.pas  runtime.setjmp.pas  runtime.utf8.pas
    runtime.mem.pas    runtime.str.pas     runtime.set.pas     runtime.arc.pas
    runtime.weak.pas   runtime.float.pas   runtime.thread.pas  runtime.exc.pas
    rtl.platform.pas   rtl.platform.layout.linux.pas  rtl.platform.posix.pas
    system.pas
  target/
    blaise                                 (the compiler binary)
    rtl/                                    (RTL .o CACHE, replaces blaise_rtl.a)
      runtime.start.o ... rtl.platform.posix.o
----

* `pasbuild compile -m blaise-compiler` builds `blaise` *and* compiles the RTL
  units to `compiler/target/rtl/*.o` (a cache dir, not an `ar` archive).  No
  extra `<unitPaths>` entry is needed — the units are in the compiler's own
  source dir.
* At link time the native internal linker adds every `compiler/target/rtl/*.o`
  (via the existing `AddOwnedObject` path) instead of `AddArchive(blaise_rtl.a)`.
* The `runtime` PasBuild module is removed from the reactor.  Its tests
  (`runtime/src/test`, the punit programs + the `punit` framework unit) move to
  a test location under the compiler module (or a retained `runtime-tests`
  module — see open questions).

Why a `.o` *cache dir* and not in-process recompile-per-program (yet): it
matches current link performance (no per-program RTL recompile), removes
`ar`/`make install`/the separate module today, and keeps the change to "where
the RTL objects come from" rather than "when they are built".  The
recompile-per-target / embedded-source end-state is a *measured* follow-up
(<<followups>>).

== Blast radius

Every reference to the old `runtime/` location or to `blaise_rtl.a`.  Grouped by
kind; all must move in lockstep so the bootstrap never breaks.

=== Build / module wiring

* `project.xml` — remove the `runtime` module from `<modules>` (keep `stdlib`).
* `compiler/project.xml` — drop `../runtime/src/main/pascal` from `<unitPaths>`
  (both `<build>` and `<test>`); the units now live under the compiler's own
  source tree (already on its path).  Add the RTL `.o`-cache build step.
* `runtime/Makefile` — removed (its per-unit compile + `ar` is replaced by the
  compiler-driven `.o` cache).  Retain only if punit tests still need it.

=== Compiler code

* `uToolchain.pas` — `FindRTLArchive` / `ResolveToolchain.RTLPath`: replace
  "find `blaise_rtl.a`" with "find the `compiler/target/rtl/` object dir beside
  the compiler" (or, transitional, accept either).
* `blaise.codegen.native.driver.pas` — `LinkViaInternalLinker`: replace
  `Lk.AddArchive(TC.RTLPath)` with a loop adding each cached RTL `.o` via
  `AddOwnedObject`.  Update the "RTL not found" guard message.
* Anything embedding the RTL source path for messages/docs.

=== Self-hosting scripts (must stay green throughout)

* `scripts/fixpoint.sh` — builds the runtime (`cd runtime && make`) as stage-0;
  resolves `blaise_rtl.a` for the link steps; uses
  `--unit-path runtime/src/main/pascal`.  All three change to the new RTL
  source location + the `.o`-cache link inputs.
* `scripts/fixpoint-native.sh`, `scripts/fixpoint-native-internal.sh`,
  `scripts/fixpoint-warmcache.sh` — each resolves `blaise_rtl.a` and passes
  `--unit-path runtime/src/main/pascal`; update both.
* `scripts/rolling-bootstrap.sh` — replays `cd runtime && make BLAISE=… install`
  per commit and links `compiler/target/blaise_rtl.a`; update to the new build.
* `.github/workflows/bootstrap.yml` — caches `blaise_rtl.a`, copies it into
  release dirs and stage-1 tarballs, and runs
  `( cd runtime && make … && make … install )`.  Update cache key, copy paths,
  and the build step.

=== Release / bootstrap artifacts

* `releases/v*/` currently carry `blaise` + `blaise_rtl.a`.  Decide the new
  artifact set: `blaise` + a `rtl/` object dir (or a tarball of it).  The
  release/cut process (`cut-blaise-release` skill, CLAUDE.md "Release binary")
  must copy the new artifact.  *The current release binary still expects
  `blaise_rtl.a`* — so the migrating commit must keep cold-bootstrap working
  from the existing release (see <<coldstart>>).

=== Tests (≈12 files) and docs

* Test files passing `--unit-path runtime/src/main/pascal` (cp.test.cli,
  cp.test.linker, cp.test.assembler, cp.test.e2e.base, cp.test.strutils,
  cp.test.math, cp.test.numerics.*, cp.test.proctypes_ofobject,
  cp.test.publishedrtti, cp.test.attributes, cp.test.config): repoint to the
  new RTL source path (ideally via a single shared `ProjectRoot`/RTL-path
  helper so future moves touch one place).
* `runtime/src/test/pascal/*` (punit programs) — relocate or keep behind a
  retained tests-only module.
* Docs: `CLAUDE.md` (layout table, cold-bootstrap, fixpoint steps),
  `README.adoc`, `docs/design.adoc`, `docs/testing-strategy.adoc`,
  `docs/internal-linker-design.adoc`, `docs/self-contained-start-design.adoc`,
  `scripts/BOOTSTRAP.adoc`, and the three target/arch design docs.

== Migration order

Each stage ends with a *full re-verify* — all four fixpoints
(`fixpoint.sh`, `fixpoint-native.sh`, `fixpoint-native-internal.sh`,
`fixpoint-warmcache.sh`) plus the native-built test runner suite — and a
*cold-bootstrap-from-release* check.  Do not proceed past a red stage.

. *Move + rename the source, keep the archive.*  `git mv` each
  `runtime/src/main/pascal/blaise_X.pas` →
  `compiler/src/main/pascal/runtime.X.pas` (dotted-flat; `rtl.platform*` and
  `system.pas` keep their names), rename the `unit blaise_X;` declaration to
  `unit runtime.X;`, and sweep every `uses blaise_X` across the RTL, stdlib,
  compiler and tests to `uses runtime.X`.  Drop the `../runtime/src/main/pascal`
  `<unitPaths>` entry (units now sit in the compiler's own dir).  For the still-
  present archive build, point the Makefile's `PAS_SRC_DIR` at the new location
  (or invoke per-unit there).  `blaise_rtl.a` is still produced and linked via
  `AddArchive`; emitted symbols are unchanged, so *behaviour is identical*.
  This isolates the move+rename from the link-model change.
  Verify: all fixpoints + suite green; cold-bootstrap from the existing release.

. *Compiler builds the RTL `.o` cache.*  Teach `pasbuild compile -m
  blaise-compiler` (or a thin post-build) to compile the RTL units to
  `compiler/target/rtl/*.o`.  Still archive at link for now; just prove the
  cache is produced and correct (diff its `.o` against the Makefile's).
  Verify: cache present, byte-equivalent objects.

. *Link from the `.o` cache; drop the archive.*  `LinkViaInternalLinker` adds
  `compiler/target/rtl/*.o` instead of `AddArchive`.  `FindRTLArchive` →
  find-the-cache-dir.  Remove `ar`/`make install`/the `runtime` module and the
  Makefile.  Update all scripts + CI + release artifacts to the cache dir.
  Verify: all fixpoints + suite + cold-bootstrap green; *measure build time*
  vs the archive baseline.

. *Docs + release process.*  Update CLAUDE.md, README, design docs, and the
  release/cut process to the new layout and artifact set.

[#coldstart]
== Cold-bootstrap ordering (the thing that must not break)

The stage-1 binary is the *previous* release (`releases/v*/blaise`), which today
links `blaise_rtl.a`.  Two hazards:

. *The release binary predates the move.*  When stage-1 (the old release)
  builds the new compiler, it still needs a `blaise_rtl.a` to link the new
  compiler binary itself.  Stage 1 of the migration keeps the Makefile +
  archive precisely so the old release can still cold-bootstrap.  The archive
  is only *dropped* (stage 3) once a stage-2 binary built from moved source
  exists and can produce/consume the `.o` cache.

. *Layout-change rule.*  This does not change record/vtable layout, so a
  stage-2 rebuild is not strictly required for correctness — but the link-model
  change (stage 3) means the stage-1-from-old-release path and the
  stage-2-onward path briefly use *different* RTL sources (archive vs cache).
  Sequence so that at every commit, the binary that builds the next step has a
  working RTL: keep the archive available (built from the new source location)
  until the cache-link path is proven, then remove it in the same commit that
  switches `AddArchive` → cache and refreshes the `-pre` release artifact to one
  that ships the `.o` cache.

Concrete safe sequence: stages 1–2 are archive-compatible (old release works);
stage 3 is a single commit that (a) switches the link path, (b) updates all
scripts/CI, (c) refreshes `releases/v*-pre/` to carry the `rtl/` cache, and
(d) is validated by `rolling-bootstrap.sh` from a known-good release before any
push.

== Verification matrix

[cols="2,4",options="header"]
|===
| Check | Gate

| `fixpoint.sh` | `FIXPOINT_OK` after each stage
| `fixpoint-native.sh` | `NATIVE_FIXPOINT_OK`
| `fixpoint-native-internal.sh` | `NATIVE_INTERNAL_OK`
| `fixpoint-warmcache.sh` | `WARMCACHE_FIXPOINT_OK`
| native-built TestRunner | full suite `OK` (with `BLAISE_QBE_COMPILER` fresh)
| cold-bootstrap | new compiler builds from the *existing* release binary
| rolling-bootstrap | `./scripts/rolling-bootstrap.sh` green before any push
| build-time | stage-3 link time measured vs archive baseline (perf gate)
|===

[#followups]
== Follow-ups (not in this plan)

* *Compile-per-target RTL / embedded source.*  Replace the host-only `.o`
  cache with compiling the embedded RTL source per `--target`, so one binary
  cross-compiles to any target with nothing staged (the FreeBSD work needs a
  FreeBSD-built RTL).  Adopt only after measuring; add an in-memory/unit-cache
  if recompile cost is visible.
* *Drop the `.o` cache for true in-process linking.*  Compile RTL units in the
  same process and hand in-memory objects to the linker — removes the cache dir
  entirely.

== Open questions

* *punit tests' home.*  `runtime/src/test/pascal/*` are Blaise programs using
  `punit`.  Options: move under `compiler/src/test/`, or keep a minimal
  `runtime-tests` module.  Leaning: a retained tests-only module so the compiler
  module stays focused.
* *Release artifact form.*  Ship the `rtl/` dir as-is, or a single tarball/`.a`
  re-introduced *only* as a release-packaging convenience (built by the
  compiler, not `ar`)?  Decide at stage 4.
* *`system.pas` placement.*  It is special (implicit `System`); confirm it moves
  with the rest and that `bootstrapExclude` handling still applies.