tooling: rolling-bootstrap script for development-checkout builds

Add scripts/rolling-bootstrap.sh to rebuild the self-hosting chain from
the last release binary up to the checked-out revision. A released binary
can only compile source up to its own feature set, so between releases it
can no longer cold-bootstrap master directly — the moment a feature is
taught to the parser in one commit and used in the runtime/compiler in a
later commit, the gap opens. The script replays the chain commit-by-commit
(each step built by the previous step's binary), carrying the binary
forward, and installs the result as releases/v<version>-pre/.

It runs in a throwaway git worktree, so the live tree (including
uncommitted changes) is untouched; it passes QBE= and BLAISE= into the
runtime Makefile and uses the main repo's built QBE; and it smoke-tests
each step. A failed step names the exact commit (BOOTSTRAP_BROKEN),
making it usable as a CI check that the two-step discipline holds.

Document the prerequisite (place the last release binary under
releases/vX.Y.Z/, which is gitignored) and usage in scripts/BOOTSTRAP.adoc,
with a pointer from README.adoc.
This commit is contained in:
Graeme Geldenhuys 2026-06-03 08:05:16 +01:00
parent 6711a6cfdd
commit 2d509cf764
3 changed files with 455 additions and 0 deletions

View file

@ -207,6 +207,18 @@ Once `compiler/target/blaise` exists, subsequent RTL rebuilds (`make && make ins
work without the override. The QBE build in step 0 is a one-off and does not work without the override. The QBE build in step 0 is a one-off and does not
need to be repeated when rebuilding the compiler. need to be repeated when rebuilding the compiler.
=== Bootstrap a development checkout
The procedure above works while the latest release binary is new enough to
compile the current source. Between releases that ceases to hold: once a commit
teaches the parser a new feature and a later commit uses it in the
runtime/compiler, the release binary can no longer build `master` directly.
`scripts/rolling-bootstrap.sh` rebuilds the chain commit-by-commit from the last
release binary up to the checked-out revision, producing a working `-pre`
bootstrap binary. See link:scripts/BOOTSTRAP.adoc[scripts/BOOTSTRAP.adoc] for the
prerequisite (placing the release binary under `releases/`) and usage.
=== Build via PasBuild === Build via PasBuild
PasBuild can drive the full compile and test cycle using a Blaise binary: PasBuild can drive the full compile and test cycle using a Blaise binary:

138
scripts/BOOTSTRAP.adoc Normal file
View file

@ -0,0 +1,138 @@
= Bootstrapping a development checkout
:icons: font
:source-highlighter: rouge
:toc: macro
This document describes `scripts/rolling-bootstrap.sh`: when it is required, the
prerequisite it depends on, how to run it, and how to interpret its output.
toc::[]
== Background
Blaise is self-hosting: each compiler binary is built by the previous compiler
binary. A released binary can compile source only up to its own feature set.
Within a single release this is never a problem, because every commit needs only
the *previous* commit's binary. New language features are introduced in two
steps: one commit teaches the parser and code generator to understand the
feature, and a later commit uses it in the runtime or compiler source. The
binary built at the first commit can compile the second.
The consequence is that no single old binary spans an entire development cycle.
Shortly after a release, the first feature commit plus its first use commit make
the release binary unable to cold-bootstrap `master` directly. A contributor who
checks out `master` between releases therefore has no single binary new enough to
build the current source in one step.
`scripts/rolling-bootstrap.sh` resolves this by replaying the chain: starting
from the last release binary, it checks out each intervening commit in order and
rebuilds the compiler and runtime using the binary produced at the previous step,
carrying that binary forward until it reaches the target revision.
== Prerequisite: the release binary
The script needs a known-good starting binary. Release binaries are *not*
committed to the repository — `releases/` is listed in `.gitignore` — so a fresh
clone does not contain one. Before running the script, place the most recent
release binary (and its runtime archive) at:
[source]
----
releases/vX.Y.Z/blaise
releases/vX.Y.Z/blaise_rtl.a
----
where `vX.Y.Z` is the release tag (for example `releases/v0.9.0/blaise`). The tag
must exist in the repository history so the script can resolve it to a commit.
The script auto-selects the newest `releases/v*/blaise` whose tag is an ancestor
of the target revision. A specific starting point can be forced with `--from`.
The vendored QBE backend must also be built once:
[source,shell]
----
cd vendor/qbe && make && cd ../..
----
== Usage
Run from the project root.
[source,shell]
----
# Replay from the auto-selected release up to HEAD, then install the
# resulting binary under releases/v<version>-pre/:
./scripts/rolling-bootstrap.sh
# Replay an explicit range without installing the result:
./scripts/rolling-bootstrap.sh --from v0.9.0 --to HEAD --no-install
# Keep the temporary worktree for inspection after a run:
./scripts/rolling-bootstrap.sh --keep
----
=== Options
[cols="1,3", options="header"]
|===
| Option | Effect
| `--from REF`
| Commit or tag to start from. A binary must exist at `releases/<REF>/blaise`.
Defaults to the newest release tag that is an ancestor of `--to`.
| `--to REF`
| Target revision to bootstrap up to. Defaults to `HEAD`.
| `--keep`
| Retain the temporary worktree on success rather than removing it.
| `--no-install`
| Report the final binary path without copying it into `releases/`.
|===
== Behaviour
The script operates entirely inside a throwaway git worktree. The live working
tree — including any uncommitted changes — is never modified.
At each commit it rebuilds the runtime and the compiler with the binary carried
from the previous step, then runs a small, feature-agnostic smoke test (integer
arithmetic and `WriteLn`) to confirm the freshly built binary is functional. On
reaching the target it installs the final binary to `releases/v<version>-pre/`,
where `<version>` is read from the `Version` constant in `Blaise.pas`. The
`-pre` binary is the same artefact otherwise produced by a manual rolling build,
suitable for use as the next stage-1.
== Exit status and failure modes
[cols="1,4", options="header"]
|===
| Status | Meaning
| `0`
| The target revision was reached and the final binary produced.
| `BOOTSTRAP_BROKEN at <commit>`
| The previous step's binary could not build the named commit. This indicates a
broken self-hosting two-step: a feature was used before, or without, a prior
commit teaching the parser to understand it. The fix is to split the change so
the feature is introduced in one commit and used in the next.
| `SMOKE_FAILED at <commit>`
| The binary built but produced incorrect output for the smoke probe, indicating
a regression in basic code generation at that commit.
|===
A non-zero exit names the exact offending commit, which makes the script useful
as a continuous-integration check that the self-hosting chain remains unbroken.
== Relationship to other scripts
`scripts/fixpoint.sh` verifies that a *single* binary reproduces itself
(stage-2 IR equals stage-3 IR). `scripts/rolling-bootstrap.sh` produces that
binary in the first place when the latest release is too old to do so directly.
The two are complementary: run the rolling bootstrap to obtain a current `-pre`
binary, then run the fixpoint check to confirm it is self-consistent.

305
scripts/rolling-bootstrap.sh Executable file
View file

@ -0,0 +1,305 @@
#!/bin/bash
# Rolling bootstrap for the Blaise self-hosting chain.
#
# Problem this solves:
# A release binary (e.g. v0.9.0) can only compile source up to its own
# feature set. The moment a post-release commit adds a new language feature
# AND a later commit uses it in the runtime/compiler, the release binary can
# no longer cold-bootstrap HEAD. End-users following `master` between releases
# are then stuck — they have no binary new enough to build current source.
#
# This is fine *during* development because each commit only needs the
# PREVIOUS commit's binary: a feature is taught to the parser in one commit
# and used in the next (the self-hosting two-step). The chain is therefore
# continuous even though no single old binary spans the whole range.
#
# What this script does:
# Replays that chain. Starting from a known-good release binary, it checks
# out each commit in order and rebuilds the compiler + RTL using the PREVIOUS
# step's freshly-built binary, carrying the result forward, until it reaches
# the target ref (HEAD by default). The final binary is a working `-pre`
# bootstrap binary for the current source — the same way the committed
# v0.10.0-pre binary was produced by hand.
#
# All work happens in a throwaway git worktree, so your live working tree
# (including uncommitted changes) is never touched.
#
# Usage:
# ./scripts/rolling-bootstrap.sh [--from <tag-or-commit>] [--to <ref>]
# [--keep] [--no-install]
#
# --from REF Commit/tag to start from. Its release binary must exist at
# releases/<REF>/blaise. Default: latest releases/v* with a
# binary that is an ancestor of --to.
# --to REF Target ref to bootstrap up to. Default: HEAD.
# --keep Keep the temporary worktree on success (for inspection).
# --no-install Do not copy the final binary into releases/. Just report.
#
# Exit codes:
# 0 reached target; final binary produced (and installed unless --no-install)
# 1 usage / environment error
# 2 a step failed to build — the offending commit is reported (likely a
# broken two-step: a feature was used before/without teaching the parser)
set -u
# ---------------------------------------------------------------------------
# Argument parsing
# ---------------------------------------------------------------------------
FROM_REF=""
TO_REF="HEAD"
KEEP_WORKTREE=0
DO_INSTALL=1
while [ $# -gt 0 ]; do
case "$1" in
--from) FROM_REF="$2"; shift 2 ;;
--to) TO_REF="$2"; shift 2 ;;
--keep) KEEP_WORKTREE=1; shift ;;
--no-install) DO_INSTALL=0; shift ;;
-h|--help) grep '^#' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;;
*) echo "Unknown argument: $1" >&2; exit 1 ;;
esac
done
if [ ! -f "compiler/src/main/pascal/Blaise.pas" ]; then
echo "Run this script from the project root." >&2
exit 1
fi
ROOT="$(pwd)"
TO_SHA="$(git rev-parse --verify "$TO_REF^{commit}" 2>/dev/null)" || {
echo "Bad --to ref: $TO_REF" >&2; exit 1; }
# ---------------------------------------------------------------------------
# Pick the start binary: explicit --from, else the newest release tag whose
# binary exists and is an ancestor of the target.
# ---------------------------------------------------------------------------
pick_start() {
if [ -n "$FROM_REF" ]; then
echo "$FROM_REF"
return
fi
# Newest-first over release dirs that have a binary and lie on the path to TO.
local d ref
for d in $(ls -d releases/v* 2>/dev/null | sort -rV); do
ref="${d#releases/}"
[ -x "$d/blaise" ] || continue
git rev-parse --verify "$ref^{commit}" >/dev/null 2>&1 || continue
if git merge-base --is-ancestor "$ref" "$TO_SHA" 2>/dev/null; then
echo "$ref"
return
fi
done
}
START_REF="$(pick_start)"
if [ -z "$START_REF" ]; then
echo "No usable start binary found under releases/ that is an ancestor of $TO_REF." >&2
echo "Pass --from <tag> where releases/<tag>/blaise exists." >&2
exit 1
fi
START_BIN="$ROOT/releases/$START_REF/blaise"
START_RTL="$ROOT/releases/$START_REF/blaise_rtl.a"
if [ ! -x "$START_BIN" ]; then
echo "Start binary not found: $START_BIN" >&2
exit 1
fi
START_SHA="$(git rev-parse --verify "$START_REF^{commit}")"
if ! git merge-base --is-ancestor "$START_SHA" "$TO_SHA"; then
echo "Start ref $START_REF is not an ancestor of $TO_REF — cannot replay linearly." >&2
exit 1
fi
# Ordered list of commits to replay: every commit strictly after START up to TO.
mapfile -t COMMITS < <(git rev-list --reverse --first-parent "${START_SHA}..${TO_SHA}")
echo "Rolling bootstrap"
echo " from : $START_REF ($START_SHA)"
echo " to : $TO_REF ($TO_SHA)"
echo " steps: ${#COMMITS[@]} commit(s) to replay"
echo
if [ "${#COMMITS[@]}" -eq 0 ]; then
echo "Target is the start commit — nothing to replay. Start binary is already current."
exit 0
fi
# ---------------------------------------------------------------------------
# Throwaway worktree so the live tree is never disturbed.
# ---------------------------------------------------------------------------
WT="$(mktemp -d /tmp/blaise-rollboot.XXXXXX)"
cleanup() {
if [ "$KEEP_WORKTREE" -eq 1 ]; then
echo "Worktree kept at: $WT"
else
git -C "$ROOT" worktree remove --force "$WT" 2>/dev/null
rm -rf "$WT"
fi
}
trap cleanup EXIT
git worktree add --detach "$WT" "$START_SHA" >/dev/null 2>&1 || {
echo "Failed to create worktree." >&2; exit 1; }
# The worktree checks out vendored QBE *source* but not a built binary. The
# runtime Makefile invokes $(QBE) (default ../vendor/qbe/qbe) and we call QBE
# directly for the compiler link. Use the main repo's already-built QBE for
# both. The QBE binary is stable across the replayed range; if a step bumps the
# vendored QBE version, rebuild vendor/qbe in the main repo before replaying.
QBE_BIN="$ROOT/vendor/qbe/qbe"
if [ ! -x "$QBE_BIN" ]; then
echo "QBE binary not built: $QBE_BIN (run 'make' in vendor/qbe)" >&2
exit 1
fi
# CUR_BIN / CUR_RTL hold the most recently built (or starting) artifacts.
CUR_BIN="$START_BIN"
CUR_RTL="$START_RTL"
# If the start release shipped no RTL, we will build one at the first step.
# ---------------------------------------------------------------------------
# build_step <worktree-dir> <compiler-binary> <rtl-archive-or-empty>
# Builds RTL (with the given compiler) then the compiler itself, in WT.
# Writes WT/_boot/blaise and WT/_boot/blaise_rtl.a on success.
# ---------------------------------------------------------------------------
build_step() {
local wt="$1" cc="$2"
local out="$wt/_boot"
mkdir -p "$out"
# 1. Build + install the RTL using the previous-step compiler.
# runtime/Makefile honours BLAISE=<compiler> and COMPILER_BIN for install.
if ! ( cd "$wt/runtime" && make clean >/dev/null 2>&1 && \
make BLAISE="$cc" QBE="$QBE_BIN" >"$out/rtl.log" 2>&1 && \
make BLAISE="$cc" QBE="$QBE_BIN" install >>"$out/rtl.log" 2>&1 ); then
echo " RTL build failed:"; tail -8 "$out/rtl.log" | sed 's/^/ /'
return 1
fi
# FindRTL looks beside the compiler binary; the Makefile install target
# places blaise_rtl.a under compiler/target/. Use that as the link archive.
local rtl="$wt/compiler/target/blaise_rtl.a"
if [ ! -s "$rtl" ]; then
echo " RTL archive missing after build"; return 1
fi
# 2. Compile the compiler to IR with the previous-step binary, then
# assemble + link. Mirrors fixpoint.sh's direct-invocation approach so we
# do not depend on pasbuild internals varying across history.
if ! "$cc" --source "$wt/compiler/src/main/pascal/Blaise.pas" \
--unit-path "$wt/compiler/src/main/pascal" \
--unit-path "$wt/runtime/src/main/pascal" \
--unit-path "$wt/stdlib/src/main/pascal" \
--emit-ir > "$out/blaise.ssa" 2>"$out/compile.err"; then
echo " compiler IR generation failed:"; head -5 "$out/compile.err" | sed 's/^/ /'
return 1
fi
if [ ! -s "$out/blaise.ssa" ] || \
head -1 "$out/blaise.ssa" | grep -qiE 'error|exception'; then
echo " compiler IR invalid:"; head -3 "$out/blaise.ssa" | sed 's/^/ /'
head -3 "$out/compile.err" | sed 's/^/ /'
return 1
fi
if ! "$QBE_BIN" -o "$out/blaise.s" "$out/blaise.ssa" 2>"$out/qbe.err"; then
echo " QBE failed:"; head -5 "$out/qbe.err" | sed 's/^/ /'; return 1
fi
if ! gcc -o "$out/blaise" "$out/blaise.s" "$rtl" 2>"$out/gcc.err"; then
echo " link failed:"; head -5 "$out/gcc.err" | sed 's/^/ /'; return 1
fi
cp "$rtl" "$out/blaise_rtl.a"
return 0
}
# ---------------------------------------------------------------------------
# smoke_test <compiler-binary> <rtl-archive>
# Compiles + runs a trivial program to prove the freshly-built binary is
# functional (compiles → valid QBE → links → runs → correct stdout). Kept
# deliberately feature-agnostic: it must pass at EVERY commit in history, so
# it probes only arithmetic + WriteLn, not any specific later feature.
# ---------------------------------------------------------------------------
smoke_test() {
local cc="$1" rtl="$2"
local d; d="$(mktemp -d)"
cat > "$d/s.pas" <<'PAS'
program P;
var X: Integer;
begin
X := 6 * 7;
WriteLn(X)
end.
PAS
local ok=1
if "$cc" --source "$d/s.pas" --emit-ir > "$d/s.ssa" 2>"$d/s.err" \
&& "$QBE_BIN" -o "$d/s.s" "$d/s.ssa" 2>>"$d/s.err" \
&& gcc -o "$d/s" "$d/s.s" "$rtl" 2>>"$d/s.err"; then
[ "$("$d/s" 2>/dev/null)" = "42" ] && ok=0
fi
rm -rf "$d"
return $ok
}
# ---------------------------------------------------------------------------
# Replay loop.
# ---------------------------------------------------------------------------
STEP=0
for sha in "${COMMITS[@]}"; do
STEP=$((STEP + 1))
short="$(git rev-parse --short "$sha")"
subj="$(git log -1 --format=%s "$sha")"
printf '[%d/%d] %s %s\n' "$STEP" "${#COMMITS[@]}" "$short" "$subj"
git -C "$WT" checkout --quiet --detach "$sha" 2>/dev/null || {
echo " checkout failed"; exit 2; }
if ! build_step "$WT" "$CUR_BIN"; then
echo
echo "BOOTSTRAP_BROKEN at $short ($subj)"
echo " The previous step's binary could not build this commit. This usually"
echo " means a language feature was USED here before (or without) a prior"
echo " commit teaching the parser/codegen to understand it — a broken"
echo " self-hosting two-step. Fix by splitting: introduce the feature in one"
echo " commit, use it in the next."
exit 2
fi
if ! smoke_test "$WT/_boot/blaise" "$WT/_boot/blaise_rtl.a"; then
echo
echo "SMOKE_FAILED at $short ($subj)"
echo " The binary built but produced wrong output for the local-const-array"
echo " probe. Treating as a broken step."
exit 2
fi
# Carry this step's artifacts forward as the compiler for the next commit.
cp "$WT/_boot/blaise" "$WT/_carry_blaise"
cp "$WT/_boot/blaise_rtl.a" "$WT/_carry_rtl.a"
CUR_BIN="$WT/_carry_blaise"
CUR_RTL="$WT/_carry_rtl.a"
done
echo
echo "REACHED $TO_REF — rolling bootstrap succeeded through ${#COMMITS[@]} commit(s)."
# ---------------------------------------------------------------------------
# Install the final binary as the current -pre release.
# ---------------------------------------------------------------------------
VER="$(grep -m1 -oE "Version = '[^']+'" "$WT/compiler/src/main/pascal/Blaise.pas" \
| sed "s/Version = '//; s/'//")"
# 0.10.0-dev -> v0.10.0-pre ; 0.10.0 -> v0.10.0-pre
PRE="v${VER%-dev}"
case "$PRE" in *-pre) ;; *) PRE="${PRE}-pre" ;; esac
if [ "$DO_INSTALL" -eq 1 ]; then
DEST="$ROOT/releases/$PRE"
mkdir -p "$DEST"
cp "$CUR_BIN" "$DEST/blaise"
cp "$CUR_RTL" "$DEST/blaise_rtl.a"
chmod +x "$DEST/blaise"
echo "Installed: $DEST/blaise (+ blaise_rtl.a) [version $VER]"
else
echo "Final binary: $CUR_BIN [version $VER, would install to releases/$PRE]"
fi