Skip to main content
ARPI
Michael Palmer on GitHub (opens in a new tab)Michael Palmer on LinkedIn (opens in a new tab)

Technical

A layered batch pipeline, with every layer answerable

Deterministic generation from a seeded profile, in-memory validation, CSV with a content-digest manifest, then PostgreSQL through raw, staging, warehouse and reporting. Every run records its outcome in an audit schema.

Build scripts
178
Lexical order is execution order
Reporting views
28
The only schema the model reads
Quality checks
114
In memory, before anything is written
Reconciliations
58
Each proves a number rather than asserting it

Deterministic synthetic data.Granite Auto Group is fictional.Real-engine validation pending.How this is governed

Pipeline complete through reportingSemantic model: real-engine validation pendingARCHITECTURE.mdbinding architectureView ARCHITECTURE.md (binding architecture) on GitHub (opens in a new tab)docs/diagrams/source-controlled diagramsView docs/diagrams/ (source-controlled diagrams) on GitHub (opens in a new tab)
  1. Generated CSVWith a content-digest manifest
  2. rawLanded as untyped text, digest retained
  3. stagingTyped, deduplicated, rejections kept
  4. warehouseConformed dimensions and declared grain
  5. reportingThe published surface
  6. Semantic modelTMDL, import mode, reporting schema onlyNever loaded by an engine
Each layer is answerable on its own: the audit schema records every run's outcome, and the reporting schema is the only one the semantic model may read.
  • Implemented and tested
  • Built, validation pending
  • Not built

Select a component to highlight its dependencies. Tab into the diagram and use the arrow keys, or read the full component list below.

No component selected

Choose a component in the diagram, or from the list below, to see what owns it, what state it is in, which files implement it, and which database role can reach it.

Every component, in order

  1. CONFIGURATION

    Configuration profiles

    Complete

    Typed, validated configuration for the development, test and portfolio profiles.

    pydantic-settings resolves a profile from YAML, then allows any key to be overridden from the environment with an ARPI_ prefix. The database password is never read from a configuration file - only from ARPI_DATABASE__PASSWORD or a PGPASSWORD fallback - so a committed profile cannot carry a credential even by accident.

    Ownership
    Python, pydantic-settings
    Privacy boundary
    No data. Configuration only, and no secret value is ever stored here.
    Role access
    Not applicable - not a database object.
    Depends on
    Nothing - this is an entry point.
    Feeds
    Python generators, Validation framework, CSV and generation manifest, raw schema, staging schema, warehouse schema, reporting schema, TMDL semantic model, Power BI Desktop validation, Microsoft Fabric validation, Report pages, Public case study, audit schema
    Source and documentation
  2. GENERATION

    Python generators

    Complete

    Fourteen generators that produce every source record from documented rules and one fixed seed.

    Deterministic by construction: the same profile and seed reproduce byte-identical output. The generators model plausible dealership behaviour - seasonality, selling days, lead-source mix, inventory ageing - without making any relationship perfectly predictable, because a dataset where every correlation is exact teaches a reader nothing about analysis.

    Ownership
    Python 3.11
    Privacy boundary
    Synthetic only. The generators are prohibited from producing names, street addresses, email addresses, phone numbers, full birth dates, government identifiers or bank information.
    Role access
    Not applicable - runs before any database connection.
    Depends on
    Configuration profiles
    Feeds
    Validation framework, CSV and generation manifest, raw schema, staging schema, warehouse schema, reporting schema, TMDL semantic model, Power BI Desktop validation, Microsoft Fabric validation, Report pages, Public case study, audit schema
    Source and documentation
  3. VALIDATION

    Validation framework

    Complete

    In-memory data-quality checks that run before a single row is offered to the database.

    Every check declares a severity, and a critical failure exits non-zero so the pipeline composes in a script and in CI. Running validation in memory rather than post-load is deliberate: a bad row that never reaches the warehouse cannot be reported from it.

    Ownership
    Python
    Privacy boundary
    A dedicated privacy check asserts that no prohibited attribute appears in any generated dataset.
    Role access
    Not applicable.
    Depends on
    Python generators, Configuration profiles
    Feeds
    CSV and generation manifest, raw schema, staging schema, warehouse schema, reporting schema, TMDL semantic model, Power BI Desktop validation, Microsoft Fabric validation, Report pages, Public case study, audit schema
    Source and documentation
  4. GENERATION

    CSV and generation manifest

    Complete

    Deterministic CSV output plus a manifest carrying a SHA-256 digest of each file.

    The digest is what makes reproducibility checkable rather than claimed: a reviewer can confirm that the CSV they hold is the CSV the tests ran against. A small extract is committed under data/sample/ so the repository is inspectable without running anything.

    Ownership
    Python, pandas
    Privacy boundary
    The committed sample is synthetic and contains no prohibited attribute. Generated output under data/raw/ is gitignored.
    Role access
    Not applicable - filesystem.
    Depends on
    Validation framework, Python generators, Configuration profiles
    Feeds
    raw schema, staging schema, warehouse schema, reporting schema, TMDL semantic model, Power BI Desktop validation, Microsoft Fabric validation, Report pages, Public case study, audit schema
    Source and documentation
  5. POSTGRESQL

    raw schema

    Complete

    Source records exactly as imported, every column as text, with load lineage.

    Nothing is cast, cleaned or rejected here. Keeping the raw layer literal is what makes a later disagreement about a value answerable by looking rather than arguable.

    Ownership
    PostgreSQL 16
    Privacy boundary
    Synthetic only. Never receives data from a real source system.
    Role access
    arpi_loader writes. arpi_reporter provably cannot read it.
    Depends on
    CSV and generation manifest, Validation framework, Python generators, Configuration profiles
    Feeds
    staging schema, warehouse schema, reporting schema, TMDL semantic model, Power BI Desktop validation, Microsoft Fabric validation, Report pages, Public case study, audit schema
    Source and documentation
  6. POSTGRESQL

    staging schema

    Complete

    Typed, deduplicated views over raw exposing the most recent load batch.

    A reconciliation proves the arithmetic per entity: raw rows equal accepted plus rejected plus deduplicated, stated as an addition so that a lost row and an extra duplicate cannot cancel each other out.

    Ownership
    PostgreSQL 16
    Privacy boundary
    Synthetic only. Standardises what the raw schema received and introduces no attribute of its own.
    Role access
    arpi_loader reads and writes. arpi_reporter provably cannot read it.
    Depends on
    raw schema, CSV and generation manifest, Validation framework, Python generators, Configuration profiles
    Feeds
    warehouse schema, reporting schema, TMDL semantic model, Power BI Desktop validation, Microsoft Fabric validation, Report pages, Public case study, audit schema
    Source and documentation
  7. POSTGRESQL

    warehouse schema

    Complete

    Conformed dimensions and facts, each fact at one explicitly declared grain.

    The grain is enforced by a UNIQUE constraint in DDL and asserted by the integration suite, so it is a property of the database rather than a promise in a document. Dimension history policy is fixed by ADR-0006 per dimension rather than applied uniformly.

    Ownership
    PostgreSQL 16
    Privacy boundary
    Synthetic only. No customer-level attribute beyond an age band and a market area exists in the model at all.
    Role access
    arpi_loader writes. arpi_reporter provably cannot read it.
    Depends on
    staging schema, raw schema, CSV and generation manifest, Validation framework, Python generators, Configuration profiles
    Feeds
    reporting schema, TMDL semantic model, Power BI Desktop validation, Microsoft Fabric validation, Report pages, Public case study, audit schema
    Source and documentation
  8. POSTGRESQL

    reporting schema

    Complete

    Documented, stable views - the only surface a semantic model or a workbook may read.

    Eight dimension views, five grain-preserving fact views, and the governed analytical views that own the SQL side of every KPI. Every KPI computed here is tested against an independent derivation from the warehouse, and every ratio is asserted to return NULL rather than zero or infinity on an empty denominator.

    Ownership
    PostgreSQL 16
    Privacy boundary
    Synthetic only. The reporting layer exposes no column the privacy policy prohibits, and the model checker fails the build if a PII-bearing column appears.
    Role access
    arpi_reporter reads. This is the only schema it can reach.
    Depends on
    warehouse schema, staging schema, raw schema, CSV and generation manifest, Validation framework, Python generators, Configuration profiles, audit schema
    Feeds
    TMDL semantic model, Power BI Desktop validation, Microsoft Fabric validation, Report pages, Public case study
    Source and documentation
  9. POSTGRESQL

    audit schema

    Complete

    Pipeline runs, row counts, validation results, reconciliations and rejected records.

    Data quality is reportable data in this project, not a log file. Every run records its outcome here, which is what makes the data-quality domain answerable from the same governed layer as sales and gross.

    Ownership
    PostgreSQL 16
    Privacy boundary
    Operational metadata about synthetic runs. No record-level personal data.
    Role access
    arpi_loader writes. arpi_reporter reads it only through reporting views.
    Depends on
    warehouse schema, staging schema, raw schema, CSV and generation manifest, Validation framework, Python generators, Configuration profiles
    Feeds
    reporting schema, TMDL semantic model, Power BI Desktop validation, Microsoft Fabric validation, Report pages, Public case study
    Source and documentation
  10. SEMANTIC MODEL

    TMDL semantic model

    Built, real-engine validation pending

    A Power BI Project stored as TMDL: text, diffable, reviewable without a licence.

    Import mode over the reporting schema and no other schema, with vw_calendar marked as the date table. Validated statically on every push - the TMDL is parsed and asserted against the model documentation - which proves shape and cannot prove arithmetic. No Microsoft semantic-model engine has loaded it.

    Ownership
    Power BI, TMDL and DAX
    Privacy boundary
    Reads the reporting schema only. The model checker fails the build if the model references any other schema or any PII-bearing column.
    Role access
    Connects as arpi_reporter, which can reach reporting and nothing else.
    Depends on
    reporting schema, warehouse schema, staging schema, raw schema, CSV and generation manifest, Validation framework, Python generators, Configuration profiles, audit schema
    Feeds
    Power BI Desktop validation, Microsoft Fabric validation, Report pages, Public case study
    Source and documentation
  11. SEMANTIC MODEL

    Power BI Desktop validation

    Pending external validation

    ADR-0008 path one: a person on Windows opens the project, refreshes and evaluates the DAX.

    Requires Windows and Power BI Desktop, neither of which the project owner has. It is an accepted path of equal standing to the Fabric route, not a lesser one, and either path completed in full closes the gate.

    Ownership
    Power BI Desktop, on Windows
    Privacy boundary
    Reads a local PostgreSQL reporting schema holding synthetic data.
    Role access
    arpi_reporter.
    Depends on
    TMDL semantic model, reporting schema, warehouse schema, staging schema, raw schema, CSV and generation manifest, Validation framework, Python generators, Configuration profiles, audit schema
    Feeds
    Nothing - this is a terminal node.
    Source and documentation
  12. SEMANTIC MODEL

    Microsoft Fabric validation

    Pending external validation

    ADR-0008 path two: the committed TMDL is deployed to a Fabric workspace and queried through REST.

    Needs a Fabric tenant, a workspace and a cloud PostgreSQL database holding the reporting schema. The automation and the contract are written; the workspace and the database are not provisioned. Continuous integration must never attempt either path - it has no engine and could only assert something it cannot observe.

    Ownership
    Microsoft Fabric, Power BI REST APIs
    Privacy boundary
    Would read a cloud PostgreSQL reporting schema holding synthetic data. No credential is committed; the deployment tooling reads them from the environment.
    Role access
    arpi_reporter, against a managed cloud database.
    Depends on
    TMDL semantic model, reporting schema, warehouse schema, staging schema, raw schema, CSV and generation manifest, Validation framework, Python generators, Configuration profiles, audit schema
    Feeds
    Nothing - this is a terminal node.
    Source and documentation
  13. PRESENTATION

    Report pages

    Not built

    The seven unblocked MVP report pages. None exists.

    The PBIR project is a shell: a .platform file and a definition.pbir pointing at the semantic model, with no page, no visual and no bookmark. The static model check fails the build if report visual content appears before this increment formally starts.

    Ownership
    Power BI report layer
    Privacy boundary
    Would read the semantic model only, which reads reporting only.
    Role access
    Inherits the semantic model connection.
    Depends on
    TMDL semantic model, reporting schema, warehouse schema, staging schema, raw schema, CSV and generation manifest, Validation framework, Python generators, Configuration profiles, audit schema
    Feeds
    Public case study
    Source and documentation
  14. PRESENTATION

    Public case study

    Gated by Gate 2

    The public analytical write-up. Held closed by Gate 2, and this site ships a locked shell for it.

    Gate 2 requires complete report pages, reconciled SQL and Power BI totals, and drafted executive findings. None of the three is met. ADR-0009 records why the portfolio website foundation was permitted to ship while the analytical case study stayed gated.

    Ownership
    This website
    Privacy boundary
    Static content. No database connection, no query interface and no embedded live report.
    Role access
    None. It reads nothing at run time.
    Depends on
    Report pages, TMDL semantic model, reporting schema, warehouse schema, staging schema, raw schema, CSV and generation manifest, Validation framework, Python generators, Configuration profiles, audit schema
    Feeds
    Nothing - this is a terminal node.
    Source and documentation
178Ordered SQL scriptsNumbered so lexical order is execution order. Re-runnable end to end against an empty database.sql/View sql/ on GitHub (opens in a new tab)
28Reporting viewsEight dimension views, five grain-preserving fact views, and the governed analytical views behind every KPI.sql/05_reporting/View sql/05_reporting/ on GitHub (opens in a new tab)
58Reconciliations per runEach proves a number rather than asserting it, and every critical rule has been observed failing.sql/08_validation/View sql/08_validation/ on GitHub (opens in a new tab)
Why no page on this site has a loading state

The last layer is a build step, not a server. Records are read from the workbooks and the export at build time and ship as data, so a filter is a synchronous pass over rows that arrived with the page. Sorting by price puts an unpriced listing last in both directions rather than treating a missing price as zero.