Host-side static ELF checks (EI_OSABI/ET_EXEC/no-PT_INTERP, already asserted by
TInternalLinkerE2ETests.TestCompile_FreeBSDTarget_EmitsStaticFreeBSDExe) cannot
catch a wrong FreeBSD struct-stat offset or syscall number — only running the
binary on a real FreeBSD kernel does.
Add a freebsd-crosscheck CI job (needs: bootstrap) that:
* downloads the -pre binary the bootstrap job built,
* cross-compiles the fixtures in compiler/src/it/freebsd (hello, fileio) with
--target freebsd-x86_64 on the Linux runner via
scripts/freebsd-crosscheck-build.sh (which also sanity-checks each emitted
ELF is a FreeBSD ET_EXEC), and
* boots a FreeBSD VM (vmactions/freebsd-vm, release 14.2 to match the
struct-stat offsets in rtl.platform.layout.freebsd) and runs the binaries
there, asserting hello prints 'Hello' and fileio round-trips a file
(write/read/stat/delete) with 'file-io-ok'/'exists-ok'/'done'.
The FreeBSD output is a static, freestanding raw-syscall ET_EXEC, so the VM
needs no extra packages to run it. fileio is the meaningful check: it exercises
open/write/read/stat/unlink through the FreeBSD leaf and layout, which the
static checks are blind to.
The rolling-bootstrap smoke probe ran `$cc --source s.pas --output s` with no
unit-path, relying on the binary finding the RTL beside itself. That worked
while the RTL was an archive (FindRTL looks beside the binary), but once a
commit makes the compiler source-build the RTL, the boot/carried binary is not
beside its compiler/ source tree, so RTLSourceDir (binary-relative, then
CWD-relative) misses and the probe fails with "RTL source directory not found".
Point the probe at the worktree's RTL source via $BLAISE_RTL_SRC and the unit
path. Backward-compatible: pre-source-build commits ignore $BLAISE_RTL_SRC and
still link the archive copied beside the binary; the extra unit paths are
harmless for a program that uses no units.
The E2E test helpers linked Blaise programs against the shipped blaise_rtl.a.
With the archive on its way out (RTL-unification Stage 3), migrate them to the
source-built RTL:
- cp.test.e2e.base gains LinkWithRTL: assembles the program object, builds the
RTL from source via scripts/build-rtl-objects.sh (--exclude-defined-by drops
the RTL units the whole-program assembly already inlined), and links the loose
objects. Its five cc-link sites and the ToolchainAvailable gate now use it /
the RTL source instead of the archive.
- New cp.test.rtllink unit factors the same link helper for the standalone
helpers (publishedrtti, proctypes_ofobject, attributes) that lower programs
themselves rather than via the base class.
- cp.test.cli and cp.test.assembler drive an external blaise binary that
source-builds its own RTL, so their gates now require the binary + RTL source,
not the archive.
- cp.test.linker's TestArchive_ParsesRTL kept its archive-parser coverage
(including the GNU long-name member) by building a throwaway .a from the
source-built objects with ar — a test-time-only ar use, not a product one.
build-rtl-objects.sh resolves its RTL source relative to its own location (so it
works from any CWD — the fixpoint scripts run from the project root, the test
runner from compiler/), caches a per-object .syms list to keep repeated
--exclude-defined-by calls cheap, and skips rebuilding fresh objects.
Full suite green (3829 tests) on both QBE- and native-built runners; the
previously-skipped TInternalAsmE2ETests / TCLIContractTests and the archive
parser test now run.
Builds on the source-built-RTL link path. Two correctness fixes plus the
fixpoint-script migration that takes the self-host pipeline off blaise_rtl.a.
1. Symbol-based dedup. A program that uses an RTL unit (e.g. `uses classes`
pulls runtime.arc) compiles that unit into its own object cache, so
EnsureRTLObjects must not supply a second copy or the link double-defines it.
The dedup now compares the DEFINED symbols of the caller's objects against
each RTL object (via the ELF reader) and skips one only when fully redundant
— matching the old archive's member selection. An earlier filename-only
check wrongly suppressed the real bare-symbol provider when a same-named but
differently-mangled stale object was present.
2. Race-free build. EnsureRTLObjects now builds into a per-process private
directory (rtl/build-<pid>/) and atomically renames each finished object
into the shared cache. Parallel builders (e.g. e2e suites warming a cold
cache) no longer corrupt each other: a reader never sees a half-written
object, and two builders just publish identical results. The private dir is
cleaned up afterwards. getpid/rmdir are declared directly rather than via
GRtlPlatform to avoid a unit-init-order crash in the driver.
3. Fixpoint scripts off the archive. New scripts/build-rtl-objects.sh builds
the RTL from source (the script-level equivalent of EnsureRTLObjects), with
--exclude-defined-by so a whole-program --emit-ir/--emit-asm dump (which
inlines the RTL units the compiler uses) is not handed a duplicate copy.
fixpoint.sh and fixpoint-native.sh link via this helper instead of
blaise_rtl.a; fixpoint-native-internal.sh and fixpoint-warmcache.sh no longer
copy the archive beside the compiler (the driver source-builds the RTL).
All four fixpoints green (FIXPOINT_OK, NATIVE_FIXPOINT_OK, NATIVE_INTERNAL_OK,
WARMCACHE_FIXPOINT_OK); full suite green (3829 tests) under QBE- and
native-built runners; four cold-cache parallel suite runs pass.
The runtime Makefile / ar / make install are still present; removing them is the
final Stage-3 step.
When a unit's source content changed, it was correctly recompiled from a warm
--unit-cache, but a cached unit that USES it kept its cached interface and was
imported in the cached-iface phase — which runs BEFORE source-recompiled units
are analysed. Importing the cached dependent then failed to resolve a type
defined only by the not-yet-analysed dependency, raising EImportError (empty
message), e.g. editing blaise.codegen.driver and rebuilding the whole compiler
warm broke on blaise.frontend.opts (TBackendKind unresolved). The same
corrupted warm cache also produced an intermittent SIGSEGV in the e2e runner.
uUnitLoader now tracks units taken via the source path (FSourceLoadedNames).
In the cached path, after recursing into a unit's interface-use dependencies,
if any dependency was source-loaded the cached iface is discarded and the unit
is recompiled from source too, so it is analysed after its dependency.
Post-order DFS makes this transitive.
fixpoint-warmcache.sh now edits a dependency unit's content between its two
builds (previously two clean fills that never exercised staleness propagation);
it FAILS on the pre-fix compiler and PASSES on the fixed one.
The internal linker silently emitted a runtime-less binary when blaise_rtl.a
could not be found beside the compiler (RTLPath empty): every RTL symbol became
an undefined dynamic import, so the binary linked but died at runtime with
`undefined symbol: _SetArgs`.
scripts/rolling-bootstrap.sh hit exactly this: it carried each step's binary
forward as _carry_blaise with its RTL as _carry_rtl.a, but FindRTLArchive looks
for `blaise_rtl.a` beside the binary (CompilerBinDir) — a name it never used —
so the next step linked the compiler with no runtime archive and SMOKE_FAILED.
- Carry the binary + RTL into a dedicated _carry/ dir as blaise + blaise_rtl.a,
so FindRTLArchive resolves it. Rolling bootstrap reaches HEAD again.
- Make an empty RTLPath a hard error on both link paths (internal and external
cc) instead of producing a broken binary.
- Regression test TInternalLinkerE2ETests.TestLink_MissingRTL_FailsLoudly:
runs a symlinked compiler from an RTL-free dir and asserts a non-zero exit, a
message naming blaise_rtl.a, and no output binary.
The four fixpoint scripts missed this because fixpoint-native.sh links the .s
via gcc, never exercising the internal-linker + FindRTL path on a carried
binary layout.
v0.12.0 is the clean rolling-bootstrap anchor — the broken v0.11.0->v0.12.0
commit range sits before it, so the replay range (v0.12.0..HEAD) is entirely
native-default with the internal assembler + linker and needs no external tools.
scripts/rolling-bootstrap.sh:
- build_step / smoke_test now compile straight to an executable with the native
default backend (`--source ... --output ...`): no --emit-ir, no QBE, no gcc.
- Dropped the QBE_BIN requirement and the QBE= make var from the RTL build.
.github/workflows/bootstrap.yml:
- STAGE1_TAG v0.10.0 -> v0.12.0 (the new cold-start anchor).
- Removed the "Build QBE" and "Expose QBE to the compiler" steps and the QBE=
make var from the -pre RTL rebuild — the bootstrap chain no longer needs them.
- Branch-scoped validate-only triggers (ci/**, feature/**, bootstrap/**) and
master-only cache writes were already in place (Step 3d-bis).
These reference the v0.12.0 tag/release asset, which the cut creates next.
None of the existing fixpoint scripts exercise the incremental warm-cache
rebuild path: fixpoint.sh uses --emit-ir (no cache), fixpoint-native.sh diffs
.s (no cache), fixpoint-native-internal.sh compiles a tiny program. So a
regression in the cached-interface import/round-trip (dropped field, wrong
vtable slot, duplicate global, missing init) passes all of them and only
surfaces later as a corrupt binary — exactly the class of bug just fixed.
scripts/fixpoint-warmcache.sh builds the whole compiler twice into the same
--unit-cache (clean then warm), asserts the warm rebuild is non-empty and not
drastically smaller than the clean build, and compiles+runs hello-world with
both stage-2 binaries to prove the warm-cache-rebuilt compiler is correct, not
just present. Prints WARMCACHE_FIXPOINT_OK on success.
Phase 3 of the native-default drive — the locally-verifiable items (CI and
rolling-bootstrap changes are deferred to Phase 4, where they become correct
once v0.12.0 is the rolling-bootstrap anchor):
- Help text now marks the backends explicitly: "qbe (deprecated) | native
(default)". QBE is retained as the e2e parity oracle and QBE-fixpoint guard;
removal is a later release.
- fixpoint-native-internal.sh now drives the FULL internal pipeline (internal
assembler + internal linker) against an all-external (gcc assemble + gcc link)
reference, instead of only differing on the assembler. Both internal tools are
now the default toolchain, so the conformance guard must exercise both. Header
comments updated to reflect that it guards the internal linker too and why it
stays a differential probe (the internal assembler buffers the whole object in
memory and OOMs on the compiler's own ~631k-line .s).
- language-rationale.adoc records the decision: native backend + internal
assembler + internal linker as the default toolchain (no external tools beyond
the C runtime startup objects), why QBE is kept-but-deprecated, and the
per-invocation escape hatches (--assembler external, --linker external,
--backend qbe, --incremental).
Verified: FIXPOINT_OK, NATIVE_FIXPOINT_OK, NATIVE_INTERNAL_OK; full suite
3541 tests / 0 failures on both the QBE-built and native-built test runners.
The native x86-64 backend is now the default (bkNative). QBE remains
available via --backend qbe. Several codegen bugs that were invisible
while QBE was the default are fixed in this commit:
Native backend fixes:
- EmitLeaqGlobal helper: threadvar globals now emit the correct
fs:0 + @tpoff TLS sequence instead of bare %rip-relative leaq.
Fixes static array write (FreeLists), string subscript assignment,
EmitExprAddr, record memcpy, method-ptr cast, and open-array push
paths that all used raw leaq %s(%rip).
- EmitIncDec: unified local/global paths through VarOperand so
threadvar scalar Inc/Dec (e.g. LargeFreeCount) gets TLS addressing.
- EmitIncDec FAE.Base: Inc(P^.Field) where P is a pointer now
evaluates the base expression instead of leaq on the stack slot.
- FinalizeEmit: data section (.data/.bss/.rodata) now emits once at
the end of unit compilation via FinalizeEmit, not inside each
EmitUnit call. Fixes duplicate/missing string literals in sep-compile.
- EmitFunctionDef AExported: implementation-only helpers in monolithic
mode stay file-local (.globl suppressed) to avoid symbol collisions
with the RTL archive.
QBE backend fixes:
- AppendUnit typeinfo: export prefix added to data $typeinfo_TObject,
$typeinfo_TCustomAttribute, $vtable_TObject, $vtable_TCustomAttribute
so native-built RTL can resolve them.
- FSuppressSystemDefs: decoupled from FExportAll so CreateUnitCodeGen
can export all symbols without suppressing system defs.
Infrastructure:
- runtime/Makefile: BLAISE_FLAGS variable for passing extra compiler
flags (e.g. --backend qbe) from build scripts.
- fixpoint.sh: make clean before RTL rebuild to pick up backend change.
Neither fixpoint exercises the in-process internal assembler (--assembler
internal): fixpoint.sh assembles via qbe+gcc, fixpoint-native.sh assembles
the native .s with gcc. A miscompilation that only corrupts the internal
assembler's object output therefore passes both fixpoints cleanly — which is
exactly how the sret-Result field-read bug (fixed in the previous commit)
escaped the fixpoint gate.
scripts/fixpoint-native-internal.sh closes that gap. It compiles a small
representative program (record-returning function with sret-Result field
reads, plus immutable string literals) with BOTH --assembler internal and
--assembler external, then asserts the two binaries behave identically
(stdout + exit code). Behavioural equivalence is the sound invariant: the
two assemblers may emit different-but-valid encodings, so a byte-level
section compare would false-positive.
A true self-hosting internal-assembler fixpoint (compile the compiler with
itself via --assembler internal) is deferred: the internal assembler buffers
the whole object in memory and needs ~53 GB for the compiler's own 631k-line
.s, OOM-killing. That scalability limit is tracked in bugs.txt; until the
assembler streams its output, this differential conformance check plus
TInternalAsmE2ETests are the internal-assembler guards.
Verified: prints NATIVE_INTERNAL_OK on a fixed compiler, exits non-zero with
EXIT_MISMATCH on a compiler carrying the sret-Result bug.
scripts/fixpoint-native.sh verifies that the native backend is fully
self-hosting: stage-0 (QBE-compiled compiler) emits native assembly,
the linked stage-1 binary recompiles itself via --backend native to
produce stage-2, and stage-2 recompiles itself to produce stage-3.
A clean fixpoint means stage-2 and stage-3 assembly are identical.
Both fixpoints (QBE and native) are now required before every commit.
With mandatory () on all zero-argument calls (51ebf23), the
IsNoArgFuncCall/NoArgFuncDecl fields on TIdentExpr and the
synthesised TFuncCallExpr codegen path are dead code. Remove them.
This exposed ~300 bare function calls missed by the original
migration script across compiler, runtime, stdlib, tests, and
embedded Blaise source strings. Fix all of them.
Add IsProcFieldCall support to TMethodCallExpr so that
H.Handler() works for procedure-type fields — the parser creates
TMethodCallExpr for Obj.Name(), and the semantic pass now detects
when 'Name' is a procedural-typed field rather than a method,
routing through the indirect-call codegen path.
Update fixpoint.sh to fall back to an existing blaise_rtl.a when
the release binary is too old to build the runtime.
2627 tests pass, FIXPOINT_OK.
fixpoint.sh's runtime build (step 1) relied on the runtime Makefile's default
BLAISE=../compiler/target/blaise. After scripts/rolling-bootstrap.sh — which
installs only to releases/ and leaves the live compiler/target/ empty — that
path does not exist, so `cd runtime && make` failed with
"compiler/target/blaise: No such file or directory" (RUNTIME_FAIL) even though
the release binary was perfectly usable.
Pass BLAISE=<abs stage-1 path> into both the runtime `make` and `make install`
so the build never depends on a pre-existing compiler/target/blaise. Resolve
the RTL archive into $RTL_ARCHIVE once (installed compiler/target copy, falling
back to runtime/target/), and use it at both link sites.
Verified by running the patched script from a fresh clone with no
compiler/target/ present (the exact post-rolling-bootstrap state): reaches
FIXPOINT_OK where it previously aborted at step 1.
Add scripts/rolling-bootstrap.sh to rebuild the self-hosting chain from
the last release binary up to the checked-out revision. A released binary
can only compile source up to its own feature set, so between releases it
can no longer cold-bootstrap master directly — the moment a feature is
taught to the parser in one commit and used in the runtime/compiler in a
later commit, the gap opens. The script replays the chain commit-by-commit
(each step built by the previous step's binary), carrying the binary
forward, and installs the result as releases/v<version>-pre/.
It runs in a throwaway git worktree, so the live tree (including
uncommitted changes) is untouched; it passes QBE= and BLAISE= into the
runtime Makefile and uses the main repo's built QBE; and it smoke-tests
each step. A failed step names the exact commit (BOOTSTRAP_BROKEN),
making it usable as a CI check that the two-step discipline holds.
Document the prerequisite (place the last release binary under
releases/vX.Y.Z/, which is gitignored) and usage in scripts/BOOTSTRAP.adoc,
with a pointer from README.adoc.
Separate always-linked runtime code (system.pas, ARC, strings, platform
abstraction) from opt-in standard library units (sysutils, classes, math,
dateutils, generics, etc.). This mirrors the Go runtime/ vs stdlib, Rust
core vs std, and Swift SwiftRuntime vs SwiftCore patterns.
- runtime/ contains code linked into every binary + C shims
- stdlib/ contains units imported explicitly via uses clause
- Single platform abstraction layer (rtl.platform.*) in runtime/,
shared by both runtime/ and stdlib/ units
- bcl.testing renamed to blaise.testing across all 95+ test files
- Updated PasBuild project.xml, fixpoint.sh, README, unit search paths
- All 1931 tests pass, fixpoint OK (stage-3/stage-4)
When stage-1 is an older release with different codegen, stage-2 and
stage-3 IR will differ. The script now automatically extends to
stage-3 -> stage-4 and checks that fixpoint instead of failing.
Bump version to 0.7.0. Update fixpoint.sh to use the latest release
binary as stage-1 instead of FPC — FPC is no longer needed anywhere
in the toolchain.
Two codegen improvements that eliminate all non-@start `alloc`
instructions from the emitted QBE IR. Both are prerequisite work for
the self-hosting fixpoint (currently being debugged).
(1) Short-circuit boolean operators now use QBE phi instead of alloc4.
Previously `A and B` / `A or B` emitted:
%slot =l alloc4 1 <- dynamic alloc, grows stack per execution
storew %L, %slot
jnz %L, @rhs, @end
@rhs storew %R, %slot jmp @end
@end %T =w loadw %slot
The slot was allocated in whichever basic block the expression appeared
in. QBE emits a runtime `sub $16, %rsp` for every alloc outside @start
— inside a while-condition this ran on every loop iteration, growing the
stack by 16 bytes per iteration indefinitely.
New form uses a phi node at the join block:
jnz %L, @rhs, @end
@rhs (evaluate R) jmp @end
@end %T =w phi @lhs %L, @rhs %R
No alloc needed; %T is always well-defined at @end by SSA.
FCurrentBlockLabel tracks the current basic block name so the phi can
correctly name its predecessors.
(2) Exception frames pre-allocated at @start via EmitExcFrameAllocs.
Previously EmitTryFinallyStmt / EmitTryExceptStmt emitted `alloc16 512`
inline at the point of the try statement. Inside a loop (e.g. `for I
:= 0 to N-1 do begin try ... finally ... end end`), each iteration
allocated 512 bytes of stack that was never released until the function
returned. Over hundreds of iterations this drifted the stack pointer
into parent exception frame territory, corrupting the exc-frame chain.
New approach: CountTryStmts recursively counts all try blocks in the
function body, EmitExcFrameAllocs pre-allocates %_exc_frame_0 …
%_exc_frame_N at @start (QBE hoists these to the static prologue sub),
and EmitTryFinallyStmt/Except index into the pool via FExcFrameNext.
Result: zero alloc instructions outside @start blocks in the emitted IR.
All 1242 tests pass. Stage-1 IR and stage-2 IR are now byte-for-byte
identical (md5 verified) — the fixpoint is correct at the IR level.
The stage-2 BINARY still crashes during stage-3 generation; this is
under active investigation and may be a QBE vs FPC code-generation
quality issue for the same IR.
mzhoot
