Project provenance
A study that builds on DendriNet keeps its own code (configurations, runners,
analyses, figure scripts) in its own repository and runs it against an exact
library commit. Four standard-library utilities in
dendritic_modeling/utils record and check that relationship:
Utility |
What it does |
|---|---|
|
Verifies a project’s source contract against a library checkout and runs the project’s declared entry points with the verified package, writing a receipt (Source contracts). |
|
Exports an allow-listed set of committed project files to a deterministic archive with an export record (Exports). |
|
Checks that the files a project declares as evidence for its results still have their recorded SHA-256 digests (Evidence checks). |
|
Python functions that freeze a list of input files with their digests and rehash them, shared across data-parallel ranks, before a run uses them (Exact-file inventories). |
The first three are programs that run by file path, so that they can check a checkout before any copy of DendriNet is imported; they never edit a checkout or change a Git reference. None of them assesses scientific equivalence: a verified source or a matching digest says which bytes ran, not that two versions compute the same results.
Source contracts
A project declares the library it runs on in dendrinet.project.json at the
root of its repository:
{
"schema_version": 1,
"project": "example-study",
"library": {
"line": "core",
"commit": "0123456789abcdef0123456789abcdef01234567",
"ref": "refs/tags/example-study/v1",
"maintained_ref": "refs/remotes/origin/develop"
},
"source_paths": ["scripts", "configs"],
"entry_points": {
"train": "scripts/train.py",
"figures": "scripts/figures.py"
}
}
Field |
Meaning |
|---|---|
|
|
|
The project’s name. |
|
|
|
The exact 40-character commit the project runs on. |
|
A tag ( |
|
Required for |
|
Optional, for a release built by squashing (Releases built by squashing). |
|
The project’s code and configuration paths, relative to the repository root. |
|
Names mapped to Python scripts inside |
Check a contract, or run an entry point, with the utility of the library checkout the contract pins:
python <library>/src/dendritic_modeling/utils/project_source.py check \
--contract <project>/dendrinet.project.json --library-root <library>
python <library>/src/dendritic_modeling/utils/project_source.py run \
--contract <project>/dendrinet.project.json --library-root <library> \
--entry figures --receipt <runs>/figures_source.json -- --out <runs>/figures
--library-root defaults to $DENDRINET_LIBRARY_ROOT; there is no
fallback to an installed copy. A tool from a newer checkout can verify an
older library commit that predates the utility.
To keep the contract in a subdirectory, pass --paper-root <project> (or
paper_root=Path(...) to verify_contract). Source paths and entry points
still resolve from that explicit Git root, and the nested contract is checked
for uncommitted changes. Without this option the contract must remain at the
root, preserving existing calls and receipts. The contract cannot be outside
the declared repository, including through a symlink. Within that repository,
the contract and each component of its requested path must be free of symlinks;
name the actual tracked file instead. External directory aliases into the
repository are allowed, but every component after first entering the repository
is checked, including components reached through an alias target. Ownership and
cleanliness checks treat filenames
literally, including names containing Git wildcard characters. Inherited Git
pathspec settings cannot change those checks; the caller’s environment remains
unchanged. Root inference resolves the parent separately from the original
request. The original traversal is checked before normalizing safe ..
components, so ../dendrinet.project.json works from a paper subdirectory
without hiding internal symlinks. Once inside the repository, traversal cannot
leave it, even temporarily.
The original contract path must also name a regular file: missing/../contract.json
and file/../contract.json are rejected even if normalization would find a valid
contract. This check applies to both check and run before reading the contract
or creating a run receipt. Export manifests retain their separate committed-file rules.
check verifies, and fails with SourceContractError at the first
problem:
the contract’s fields as described above;
the project directory (the contract’s directory) and the library root are each the root of a Git repository;
the library’s
HEADand the commit oflibrary.refare bothlibrary.commit;for a maintained line,
library.maintained_refresolves to a commit andlibrary.commitis that commit or one of its ancestors (or the release bridge holds);the library has a package at
src/dendritic_modeling/__init__.py;src,configs,pyproject.toml,setup.pyandrequirements.txtof the library, and thesource_pathsand contract of the project, have no uncommitted or untracked changes, no ignored files with a source suffix (.py,.so,.pyd,.yaml,.yml,.json,.jsonl,.toml,.sh,.sbatch,.slurm,.ipynb) and no tracked symbolic links;every declared source path exists, and every entry point is a tracked Python file inside a declared source path.
It then prints the receipt:
Receipt field |
Content |
|---|---|
|
|
|
Path and SHA-256 digest of the contract. |
|
|
|
The verified library commit, its tag and its line. |
|
The maintained branch and the commit it pointed at ( |
|
Only when the contract declares a bridge: its |
|
The library checkout, its |
|
The interpreter. |
|
|
run verifies the contract the same way and then runs the named entry point
as __main__, with the arguments after --, from the project directory.
It imports dendritic_modeling from <library>/src and fails when another
copy was already imported; it sets DENDRINET_LIBRARY_ROOT and
DENDRINET_CONFIG_ROOT (the library’s configs) and puts <library>/src
first on PYTHONPATH for child processes. The receipt must be a new file
outside the project and library repositories: an existing receipt is never
overwritten. It is written when the run starts and again when it ends, with
entry_point, script, arguments, imported_package_file,
status (started, then completed or failed) and exit_code,
or error_type and error for an exception. run exits with the entry
point’s exit status.
Releases built by squashing
A public release can be built by squashing the maintained line into a single
commit whose tree equals a commit of that line. The release then has none of
the line’s commits as ancestors, so a contract that pins a commit of the line
and names the release branch as maintained_ref fails the ancestry check
even though the released files are identical. The contract states that
relationship explicitly:
{
"library": {
"line": "core",
"commit": "0123456789abcdef0123456789abcdef01234567",
"ref": "refs/tags/example-study/v1",
"maintained_ref": "refs/remotes/origin/develop",
"release_bridge": {
"source_commit": "89abcdef0123456789abcdef0123456789abcdef",
"release_commit": "fedcba9876543210fedcba9876543210fedcba98"
}
}
}
With a bridge, the ancestry check is replaced by these checks, all of which must hold:
source_commitandrelease_commitare exact 40-character commits present in the library checkout (fetch both before verification);library.commitissource_commitor one of its ancestors, so the release contains the pinned commit;release_commitis themaintained_refcommit or one of its ancestors;source_commitandrelease_commithave the same tree (git rev-parse <commit>^{tree}).
The receipt records the bridge and the tree. Without a bridge the ancestry check is unchanged, and a historical contract cannot declare one. The bridge establishes identical released files only; it does not approve a release or assess its results.
Exports
project_export.py writes the files that a manifest at the root of the
project repository lists, as committed at a given commit, to a new ZIP
archive:
python <library>/src/dendritic_modeling/utils/project_export.py \
--manifest <project>/manuscript_export.json \
--commit <project commit> --output <exports>/manuscript.zip
The manifest is read from that commit, not from the working tree:
An optional --paper-root <project> (paper_root=Path(...) in the Python
API) allows the manifest itself to live in a subdirectory. Its listed file
paths remain relative to the explicit repository root. Containment, committed
input checks and exclusive creation are unchanged. Existing root-level calls
retain their default behavior. At the same exporter revision, equivalent root
and nested manifests produce byte-identical archives; changing exporter
revisions changes the recorded tool hashes, not the exported file contents.
The receipt hashes both project_export.py and its sibling
project_source.py, which supplies the shared path validation. Both utilities
use only the standard library and must come from the same tooling checkout.
The requested repository-relative manifest path selects a regular file in the
specified commit. Working-tree symlinks cannot redirect that selection: a
contained link still selects the original committed path, while a link outside
the repository is rejected, even if a later link points back inside. Each hop
after first entering the canonical repository root is checked, and symlink loops
are rejected. The manifest need not still exist in the working tree. As with
source contracts, external directory aliases into the repository are allowed.
Root inference never replaces the original request with a resolved path.
Parent-directory components are normalized only after traversal is checked, so
../manuscript_export.json works from a paper subdirectory, with or without
--paper-root. A path cannot leave the repository after first entering it,
even if it later returns.
{
"schema_version": 1,
"kind": "manuscript",
"files": ["main.tex", "references.bib", "figures/figure1.pdf"]
}
kind is manuscript or paper-code. Manuscript files have the
suffixes .tex, .bib, .bst, .cls, .sty, .pdf, .eps,
.png or .jpg. Paper-code files have the suffixes .py, .sh,
.sbatch, .slurm, .yaml, .yml, .json, .md, .txt,
.tsv, .csv, .toml, .ini, .cpp or .h; explicitly
listed files named LICENSE or Makefile are also allowed. Paper-code paths
cannot contain src, dendritic_modeling, runtime, historical_runtimes,
source_snapshots, frozen_sources, .git, outputs, results
or logs (library source is never exported with a project). A
paper-code manifest also declares publication_status: internal_candidate,
a library_dependency object and optionally reproduction_status. Every
listed file must be a regular file committed at that commit; absolute paths,
.. and backslashes are refused, and so is a list with duplicates.
The archive must be a new file outside the project repository and the library
checkout. It holds each file with its Git mode and a fixed timestamp, so the
same commit and manifest give a byte-identical archive, and
EXPORT_SOURCE.json with schema_version, kind, paper_commit,
manifest_sha256, exporter_sha256, source_checker_sha256 and files
(path, SHA-256 digest and size of each); a paper-code export adds publication_status,
reproduction_status (default not validated), library_dependency
and library_source_included: false. An export does not make a candidate
ready for publication.
Evidence checks
project_evidence.py checks a manifest that binds a project’s results to
the files that support them:
python <library>/src/dendritic_modeling/utils/project_evidence.py \
--manifest <project>/evidence.json \
--root paper=<project> --root runs=<results root> \
--report <reports>/evidence_report.json --require-qualified
{
"schema_version": 1,
"project": "example-study",
"baseline_paper_commit": "0123456789abcdef0123456789abcdef01234567",
"files": [
{"id": "fig1_data", "root": "runs", "path": "figure1/summary.csv",
"sha256": "<64 hexadecimal digits>"}
],
"results": [
{"id": "Figure 1", "artifact": "fig1_data", "evidence": [],
"workflow_kind": "native", "qualification": "pending"}
],
"missing_evidence": []
}
Each file names one of the --root NAME=PATH directories and a relative path
inside it, with its expected SHA-256 digest. Each result names its artifact
and evidence files by id and classifies its computation in
workflow_kind; it counts as qualified only with qualification:
numerically_validated and a non-empty validation_evidence list of file
id values. Unknown roots, paths that leave their root, duplicate
identifiers and references to undeclared files raise.
The report, a new file outside every root, holds integrity (passed
when every file matches), one check per file (match, changed or
missing), unqualified_results, unresolved_dependencies (the
manifest’s missing_evidence), the manifest digest and the baseline commit.
The command exits with status 1 when a file does not match, and with
--require-qualified also when a result is unqualified or a dependency is
unresolved. Matching digests show that the declared bytes are preserved, not
that the workflow that produced them is valid.
Exact-file inventories
dendritic_modeling.utils.exact_file_inventory freezes the input files of a
run after the application has verified them, and rehashes every file before
each execution; no size-or-timestamp shortcut is accepted. An inventory lists
each file’s absolute resolved path, size, SHA-256 digest and roles, sorted by
path with task identifiers file_00000, file_00001, …, and carries its
own digest:
build_inventorycanonicalizes the application’s list of files (merging the roles of a repeated path) with the application’s generation binding;validate_inventorychecks an inventory’s schema, order and digest without reading the files;balanced_assignmentassigns the files to ranks, largest first, each to the rank with the fewest bytes so far;verify_shardrehashes one rank’s files, comparing a symlink-awarefile_snapshotbefore and after each read, and returns a shard receipt;merge_verified_shardsaccepts exactly one shard per rank with exactly the assigned files, checks that no file changed since it was hashed, and returns the verification receipt (with its own digest) and the snapshots.
The functions are listed in Python API.