docs: RTL-unification implementation plan
Execution plan for folding the runtime into the compiler: single pasbuild compile -m blaise-compiler, no separate runtime module, no make install, no ar-built blaise_rtl.a. Records the constraining facts (programs emit RTL calls as undefined externals satisfied by the archive; the archive is a Blaise-only ar artifact pasbuild does not produce; RTL is now pure Pascal), the chosen target shape (RTL units relocated into the compiler source tree, compiler produces a cached rtl/*.o dir the internal linker consumes), and the full blast radius — project.xml, uToolchain/FindRTLArchive, native driver AddArchive, all four fixpoint scripts, rolling-bootstrap, CI bootstrap.yml, ~12 tests, release artifacts, and docs. Recommends DOTTED-FLAT unit names (runtime.atomic, runtime.str, …) over a subdirectory: the unit loader is filename-based (dotted name -> dotted file, no subdir traversal), so dotted-flat keeps the units on the compiler's existing unit-path with zero new path entry, at the cost of a one-time uses-clause rename sweep; emitted runtime symbols are unchanged either way. Sequences the migration archive-compatible-first (old release keeps cold- bootstrapping) and only drops the archive once a stage-2 binary produces/ consumes the .o cache, with a full four-fixpoint + cold-bootstrap re-verify after each stage and a build-time measurement gate before keeping the compile-per-target follow-up.
This commit is contained in:
Graeme Geldenhuys
parent
743ca4f5e6
commit
55b308985e
317
docs/rtl-unification-plan.adoc
Normal file
317
docs/rtl-unification-plan.adoc
Normal file
|
|
@ -0,0 +1,317 @@
|
|||
= RTL Unification — Implementation Plan
|
||||
:author: Graeme Geldenhuys
|
||||
:revdate: 2026-06-25
|
||||
:toc: left
|
||||
:toclevels: 3
|
||||
:source-highlighter: rouge
|
||||
:icons: font
|
||||
:sectanchors:
|
||||
:sectnums:
|
||||
|
||||
== Purpose
|
||||
|
||||
Fold the always-linked runtime (`runtime/`) into the compiler so that a single
|
||||
`pasbuild compile -m blaise-compiler` produces a self-contained toolchain — no
|
||||
separate `runtime` module, no `cd runtime && make install`, no `ar`-built
|
||||
`blaise_rtl.a`. This is the first concrete step of the "one binary, embedded
|
||||
RTL, no external archives" end-state recorded in
|
||||
`docs/native-target-architecture.adoc`.
|
||||
|
||||
This document is the execution plan: the architecture facts that constrain it,
|
||||
the chosen target shape, the full blast radius (every file/script touched), the
|
||||
ordered migration with a self-hosting re-verify after each stage, and the
|
||||
cold-bootstrap ordering that must never break mid-change.
|
||||
|
||||
== Background facts that constrain the design
|
||||
|
||||
These were established by investigation (2026-06-25) and dictate the approach.
|
||||
|
||||
. *A normal program `uses` nothing of the runtime explicitly.* The compiler
|
||||
emits calls to `_SetArgs`, `_SysWriteInt`, `_BlaiseInit`, the ARC helpers,
|
||||
etc. as *undefined external symbols*. An `--emit-asm` of a hello-world is
|
||||
~100 lines with zero RTL bodies. Those externals are satisfied at link time
|
||||
by `blaise_rtl.a`. *The archive is the implicit runtime every binary needs.*
|
||||
|
||||
. *`blaise_rtl.a` is a Blaise-only artifact.* It is assembled by
|
||||
`runtime/Makefile` with `ar` from per-unit `.o` files. PasBuild's `library`
|
||||
packaging compiles units (`.o` + `.bif` for unit-path consumption) but does
|
||||
*not* emit an `ar` archive — so PasBuild has no native notion of this file.
|
||||
|
||||
. *The RTL is now pure Pascal.* All four hand-written `*_x86_64.s` files were
|
||||
migrated to inline-asm `.pas` units, so every RTL unit builds through the
|
||||
compiler's own internal assembler. Nothing external (gcc/as) is needed to
|
||||
build the runtime any longer.
|
||||
|
||||
. *The compiler already resolves RTL interfaces from the `.pas` on its
|
||||
unit-path* (for type/semantic resolution); only the *bodies* come from the
|
||||
archive. The mechanism to compile a unit to an object in-process exists
|
||||
(the incremental worker path, `LowerToObject` / `AddOwnedObject`).
|
||||
|
||||
. *`stdlib/` stays separate.* It is opt-in, not always-linked; it is not part
|
||||
of this unification.
|
||||
|
||||
== Unit relocation: dotted-flat names (recommended) vs subdirectory
|
||||
|
||||
The unit loader is *filename-based*: a unit named `X` is sought as
|
||||
`<searchpath>/X.pas`, and a *dotted* unit name maps to a *dotted filename*
|
||||
verbatim — `rtl.platform.posix` → `rtl.platform.posix.pas` — NOT to a
|
||||
subdirectory (`uUnitLoader.pas`: `Path := Base + AName + '.pas'`). Two
|
||||
RTL units already use this convention (`rtl.platform*`).
|
||||
|
||||
This drives the relocation choice:
|
||||
|
||||
[cols="1,3,3",options="header"]
|
||||
|===
|
||||
| | Dotted-flat (recommended) | Subdirectory
|
||||
|
||||
| Layout
|
||||
| `compiler/src/main/pascal/runtime.atomic.pas`
|
||||
| `compiler/src/main/pascal/runtime/blaise_atomic.pas`
|
||||
|
||||
| Unit-path change
|
||||
| *None* — files sit directly in the compiler's existing source dir, already on
|
||||
its path. Best satisfies the "reduce unit paths" goal.
|
||||
| Needs `compiler/src/main/pascal/runtime` added as a search path — re-introduces
|
||||
a unit-path entry, since the loader does not traverse subdirs.
|
||||
|
||||
| Unit names
|
||||
| *Renamed* (`blaise_atomic` → `runtime.atomic`): churns every `uses` clause
|
||||
and `.bif`/cache filenames.
|
||||
| *Preserved*: no `uses` churn.
|
||||
|
||||
| Emitted symbols
|
||||
| *Unchanged* either way — runtime symbols (`_BlaiseGetMem`, `_SetArgs`, …) are
|
||||
explicit `external name` strings, independent of the unit name.
|
||||
|===
|
||||
|
||||
Because the loader keys on filename, the dotted-flat scheme reaches the goal
|
||||
(no extra unit-path) with the trade of a one-time `uses`-clause rename sweep,
|
||||
whereas the subdirectory keeps a path entry. *Recommendation: dotted-flat.*
|
||||
Proposed name map (keep `rtl.platform*` as-is; they already fit):
|
||||
|
||||
[cols="2,2",options="header"]
|
||||
|===
|
||||
| Today | Dotted-flat
|
||||
|
||||
| blaise_start | runtime.start
|
||||
| blaise_atomic | runtime.atomic
|
||||
| blaise_setjmp | runtime.setjmp
|
||||
| blaise_utf8 | runtime.utf8
|
||||
| blaise_mem | runtime.mem
|
||||
| blaise_str | runtime.str
|
||||
| blaise_set | runtime.set
|
||||
| blaise_arc | runtime.arc
|
||||
| blaise_weak | runtime.weak
|
||||
| blaise_float | runtime.float
|
||||
| blaise_thread | runtime.thread
|
||||
| blaise_exc | runtime.exc
|
||||
| system | system (unchanged — implicit System unit)
|
||||
|===
|
||||
|
||||
(Whichever scheme is chosen, the `.o`-cache and link-model changes below are
|
||||
identical.)
|
||||
|
||||
== Target shape (dotted-flat)
|
||||
|
||||
[source]
|
||||
----
|
||||
compiler/
|
||||
src/main/pascal/
|
||||
Blaise.pas, uParser.pas, ... (compiler sources, as today)
|
||||
runtime.start.pas runtime.atomic.pas runtime.setjmp.pas runtime.utf8.pas
|
||||
runtime.mem.pas runtime.str.pas runtime.set.pas runtime.arc.pas
|
||||
runtime.weak.pas runtime.float.pas runtime.thread.pas runtime.exc.pas
|
||||
rtl.platform.pas rtl.platform.layout.linux.pas rtl.platform.posix.pas
|
||||
system.pas
|
||||
target/
|
||||
blaise (the compiler binary)
|
||||
rtl/ (RTL .o CACHE, replaces blaise_rtl.a)
|
||||
runtime.start.o ... rtl.platform.posix.o
|
||||
----
|
||||
|
||||
* `pasbuild compile -m blaise-compiler` builds `blaise` *and* compiles the RTL
|
||||
units to `compiler/target/rtl/*.o` (a cache dir, not an `ar` archive). No
|
||||
extra `<unitPaths>` entry is needed — the units are in the compiler's own
|
||||
source dir.
|
||||
* At link time the native internal linker adds every `compiler/target/rtl/*.o`
|
||||
(via the existing `AddOwnedObject` path) instead of `AddArchive(blaise_rtl.a)`.
|
||||
* The `runtime` PasBuild module is removed from the reactor. Its tests
|
||||
(`runtime/src/test`, the punit programs + the `punit` framework unit) move to
|
||||
a test location under the compiler module (or a retained `runtime-tests`
|
||||
module — see open questions).
|
||||
|
||||
Why a `.o` *cache dir* and not in-process recompile-per-program (yet): it
|
||||
matches current link performance (no per-program RTL recompile), removes
|
||||
`ar`/`make install`/the separate module today, and keeps the change to "where
|
||||
the RTL objects come from" rather than "when they are built". The
|
||||
recompile-per-target / embedded-source end-state is a *measured* follow-up
|
||||
(<<followups>>).
|
||||
|
||||
== Blast radius
|
||||
|
||||
Every reference to the old `runtime/` location or to `blaise_rtl.a`. Grouped by
|
||||
kind; all must move in lockstep so the bootstrap never breaks.
|
||||
|
||||
=== Build / module wiring
|
||||
|
||||
* `project.xml` — remove the `runtime` module from `<modules>` (keep `stdlib`).
|
||||
* `compiler/project.xml` — drop `../runtime/src/main/pascal` from `<unitPaths>`
|
||||
(both `<build>` and `<test>`); the units now live under the compiler's own
|
||||
source tree (already on its path). Add the RTL `.o`-cache build step.
|
||||
* `runtime/Makefile` — removed (its per-unit compile + `ar` is replaced by the
|
||||
compiler-driven `.o` cache). Retain only if punit tests still need it.
|
||||
|
||||
=== Compiler code
|
||||
|
||||
* `uToolchain.pas` — `FindRTLArchive` / `ResolveToolchain.RTLPath`: replace
|
||||
"find `blaise_rtl.a`" with "find the `compiler/target/rtl/` object dir beside
|
||||
the compiler" (or, transitional, accept either).
|
||||
* `blaise.codegen.native.driver.pas` — `LinkViaInternalLinker`: replace
|
||||
`Lk.AddArchive(TC.RTLPath)` with a loop adding each cached RTL `.o` via
|
||||
`AddOwnedObject`. Update the "RTL not found" guard message.
|
||||
* Anything embedding the RTL source path for messages/docs.
|
||||
|
||||
=== Self-hosting scripts (must stay green throughout)
|
||||
|
||||
* `scripts/fixpoint.sh` — builds the runtime (`cd runtime && make`) as stage-0;
|
||||
resolves `blaise_rtl.a` for the link steps; uses
|
||||
`--unit-path runtime/src/main/pascal`. All three change to the new RTL
|
||||
source location + the `.o`-cache link inputs.
|
||||
* `scripts/fixpoint-native.sh`, `scripts/fixpoint-native-internal.sh`,
|
||||
`scripts/fixpoint-warmcache.sh` — each resolves `blaise_rtl.a` and passes
|
||||
`--unit-path runtime/src/main/pascal`; update both.
|
||||
* `scripts/rolling-bootstrap.sh` — replays `cd runtime && make BLAISE=… install`
|
||||
per commit and links `compiler/target/blaise_rtl.a`; update to the new build.
|
||||
* `.github/workflows/bootstrap.yml` — caches `blaise_rtl.a`, copies it into
|
||||
release dirs and stage-1 tarballs, and runs
|
||||
`( cd runtime && make … && make … install )`. Update cache key, copy paths,
|
||||
and the build step.
|
||||
|
||||
=== Release / bootstrap artifacts
|
||||
|
||||
* `releases/v*/` currently carry `blaise` + `blaise_rtl.a`. Decide the new
|
||||
artifact set: `blaise` + a `rtl/` object dir (or a tarball of it). The
|
||||
release/cut process (`cut-blaise-release` skill, CLAUDE.md "Release binary")
|
||||
must copy the new artifact. *The current release binary still expects
|
||||
`blaise_rtl.a`* — so the migrating commit must keep cold-bootstrap working
|
||||
from the existing release (see <<coldstart>>).
|
||||
|
||||
=== Tests (≈12 files) and docs
|
||||
|
||||
* Test files passing `--unit-path runtime/src/main/pascal` (cp.test.cli,
|
||||
cp.test.linker, cp.test.assembler, cp.test.e2e.base, cp.test.strutils,
|
||||
cp.test.math, cp.test.numerics.*, cp.test.proctypes_ofobject,
|
||||
cp.test.publishedrtti, cp.test.attributes, cp.test.config): repoint to the
|
||||
new RTL source path (ideally via a single shared `ProjectRoot`/RTL-path
|
||||
helper so future moves touch one place).
|
||||
* `runtime/src/test/pascal/*` (punit programs) — relocate or keep behind a
|
||||
retained tests-only module.
|
||||
* Docs: `CLAUDE.md` (layout table, cold-bootstrap, fixpoint steps),
|
||||
`README.adoc`, `docs/design.adoc`, `docs/testing-strategy.adoc`,
|
||||
`docs/internal-linker-design.adoc`, `docs/self-contained-start-design.adoc`,
|
||||
`scripts/BOOTSTRAP.adoc`, and the three target/arch design docs.
|
||||
|
||||
== Migration order
|
||||
|
||||
Each stage ends with a *full re-verify* — all four fixpoints
|
||||
(`fixpoint.sh`, `fixpoint-native.sh`, `fixpoint-native-internal.sh`,
|
||||
`fixpoint-warmcache.sh`) plus the native-built test runner suite — and a
|
||||
*cold-bootstrap-from-release* check. Do not proceed past a red stage.
|
||||
|
||||
. *Move + rename the source, keep the archive.* `git mv` each
|
||||
`runtime/src/main/pascal/blaise_X.pas` →
|
||||
`compiler/src/main/pascal/runtime.X.pas` (dotted-flat; `rtl.platform*` and
|
||||
`system.pas` keep their names), rename the `unit blaise_X;` declaration to
|
||||
`unit runtime.X;`, and sweep every `uses blaise_X` across the RTL, stdlib,
|
||||
compiler and tests to `uses runtime.X`. Drop the `../runtime/src/main/pascal`
|
||||
`<unitPaths>` entry (units now sit in the compiler's own dir). For the still-
|
||||
present archive build, point the Makefile's `PAS_SRC_DIR` at the new location
|
||||
(or invoke per-unit there). `blaise_rtl.a` is still produced and linked via
|
||||
`AddArchive`; emitted symbols are unchanged, so *behaviour is identical*.
|
||||
This isolates the move+rename from the link-model change.
|
||||
Verify: all fixpoints + suite green; cold-bootstrap from the existing release.
|
||||
|
||||
. *Compiler builds the RTL `.o` cache.* Teach `pasbuild compile -m
|
||||
blaise-compiler` (or a thin post-build) to compile the RTL units to
|
||||
`compiler/target/rtl/*.o`. Still archive at link for now; just prove the
|
||||
cache is produced and correct (diff its `.o` against the Makefile's).
|
||||
Verify: cache present, byte-equivalent objects.
|
||||
|
||||
. *Link from the `.o` cache; drop the archive.* `LinkViaInternalLinker` adds
|
||||
`compiler/target/rtl/*.o` instead of `AddArchive`. `FindRTLArchive` →
|
||||
find-the-cache-dir. Remove `ar`/`make install`/the `runtime` module and the
|
||||
Makefile. Update all scripts + CI + release artifacts to the cache dir.
|
||||
Verify: all fixpoints + suite + cold-bootstrap green; *measure build time*
|
||||
vs the archive baseline.
|
||||
|
||||
. *Docs + release process.* Update CLAUDE.md, README, design docs, and the
|
||||
release/cut process to the new layout and artifact set.
|
||||
|
||||
[#coldstart]
|
||||
== Cold-bootstrap ordering (the thing that must not break)
|
||||
|
||||
The stage-1 binary is the *previous* release (`releases/v*/blaise`), which today
|
||||
links `blaise_rtl.a`. Two hazards:
|
||||
|
||||
. *The release binary predates the move.* When stage-1 (the old release)
|
||||
builds the new compiler, it still needs a `blaise_rtl.a` to link the new
|
||||
compiler binary itself. Stage 1 of the migration keeps the Makefile +
|
||||
archive precisely so the old release can still cold-bootstrap. The archive
|
||||
is only *dropped* (stage 3) once a stage-2 binary built from moved source
|
||||
exists and can produce/consume the `.o` cache.
|
||||
|
||||
. *Layout-change rule.* This does not change record/vtable layout, so a
|
||||
stage-2 rebuild is not strictly required for correctness — but the link-model
|
||||
change (stage 3) means the stage-1-from-old-release path and the
|
||||
stage-2-onward path briefly use *different* RTL sources (archive vs cache).
|
||||
Sequence so that at every commit, the binary that builds the next step has a
|
||||
working RTL: keep the archive available (built from the new source location)
|
||||
until the cache-link path is proven, then remove it in the same commit that
|
||||
switches `AddArchive` → cache and refreshes the `-pre` release artifact to one
|
||||
that ships the `.o` cache.
|
||||
|
||||
Concrete safe sequence: stages 1–2 are archive-compatible (old release works);
|
||||
stage 3 is a single commit that (a) switches the link path, (b) updates all
|
||||
scripts/CI, (c) refreshes `releases/v*-pre/` to carry the `rtl/` cache, and
|
||||
(d) is validated by `rolling-bootstrap.sh` from a known-good release before any
|
||||
push.
|
||||
|
||||
== Verification matrix
|
||||
|
||||
[cols="2,4",options="header"]
|
||||
|===
|
||||
| Check | Gate
|
||||
|
||||
| `fixpoint.sh` | `FIXPOINT_OK` after each stage
|
||||
| `fixpoint-native.sh` | `NATIVE_FIXPOINT_OK`
|
||||
| `fixpoint-native-internal.sh` | `NATIVE_INTERNAL_OK`
|
||||
| `fixpoint-warmcache.sh` | `WARMCACHE_FIXPOINT_OK`
|
||||
| native-built TestRunner | full suite `OK` (with `BLAISE_QBE_COMPILER` fresh)
|
||||
| cold-bootstrap | new compiler builds from the *existing* release binary
|
||||
| rolling-bootstrap | `./scripts/rolling-bootstrap.sh` green before any push
|
||||
| build-time | stage-3 link time measured vs archive baseline (perf gate)
|
||||
|===
|
||||
|
||||
[#followups]
|
||||
== Follow-ups (not in this plan)
|
||||
|
||||
* *Compile-per-target RTL / embedded source.* Replace the host-only `.o`
|
||||
cache with compiling the embedded RTL source per `--target`, so one binary
|
||||
cross-compiles to any target with nothing staged (the FreeBSD work needs a
|
||||
FreeBSD-built RTL). Adopt only after measuring; add an in-memory/unit-cache
|
||||
if recompile cost is visible.
|
||||
* *Drop the `.o` cache for true in-process linking.* Compile RTL units in the
|
||||
same process and hand in-memory objects to the linker — removes the cache dir
|
||||
entirely.
|
||||
|
||||
== Open questions
|
||||
|
||||
* *punit tests' home.* `runtime/src/test/pascal/*` are Blaise programs using
|
||||
`punit`. Options: move under `compiler/src/test/`, or keep a minimal
|
||||
`runtime-tests` module. Leaning: a retained tests-only module so the compiler
|
||||
module stays focused.
|
||||
* *Release artifact form.* Ship the `rtl/` dir as-is, or a single tarball/`.a`
|
||||
re-introduced *only* as a release-packaging convenience (built by the
|
||||
compiler, not `ar`)? Decide at stage 4.
|
||||
* *`system.pas` placement.* It is special (implicit `System`); confirm it moves
|
||||
with the rest and that `bootstrapExclude` handling still applies.
|
||||
Loading…
Reference in a new issue