Data Pipeline

Pipeline overview

Stackless pipelines move data through five practical stages:

1. Connect source systems.
2. Load source data into the customer warehouse.
3. Transform raw data into governed datasets.
4. Model business metrics and dimensions.
5. Serve dashboards, Agent artifacts, exports, and operational reporting.

[picture 1. Data pipeline status view with source sync, model refresh, and dashboard freshness]

Connect

Source connectors bring upstream data into Stackless. Depending on the source, setup may involve OAuth, an embedded connector setup card, secure credentials, a database connection, a file upload, or a custom API path.

The Sources page is the first place to check connector availability, setup state, sync health, and recent status.

Load

Connected sources land in the customer warehouse. Raw source tables keep the source system's shape as much as possible so the original records can be traced.

Users usually should not build final dashboards directly on raw source tables. Raw source data often needs:

• Time zone normalization.
• Stable IDs and joins.
• Refund, cancellation, and deletion handling.
• Type casting and null handling.
• Source-specific business rules.
• Deduplication.
• Attribution or channel normalization.

Transform

Transformation Models turn source tables into business-ready datasets. A transformation may create a table or view, depending on the materialization and refresh policy.

Common transformation patterns include:

• Staging raw source columns into clean names and types.
• Joining customer, order, payment, campaign, and location tables.
• Flattening nested JSON into rows and columns.
• Keeping the latest record per entity.
• Aggregating to daily, weekly, channel, location, customer, or campaign grain.
• Pivoting or unpivoting reporting data into the shape a dashboard needs.

Each model can be previewed before publishing. Published models are the stable inputs for Semantic Models and dashboards.

Model

Semantic Models define the business layer on top of transformed data. They describe measures, dimensions, joins, titles, descriptions, and source lineage. The semantic layer is what lets the Agent and dashboards use the same metric definitions.

Example:

• A Transformation Model may produce orders_enriched.
• A Semantic Model may expose orders.revenue, orders.count, orders.channel, and orders.created_at.
• A dashboard may use those members for KPIs, trends, and tables.

Serve

Served outputs include:

• Published dashboards for recurring reporting.
• Private dashboard drafts.
• Agent table artifacts and chart cards.
• Briefs and finding cards.
• Scheduled dashboard exports.
• Direct read-only Agent analysis.

Table artifacts are snapshots. Dashboards and published models are the right place for recurring, refreshed reporting.

Refresh and freshness

Freshness can come from multiple layers:

• Source sync freshness.
• Transformation Model materialization status.
• Semantic Model publish state.
• Dashboard refresh state.
• Scheduled export run history.

When a number looks stale, check the upstream layers in order: source, transformation, semantic model, dashboard.

Failure handling

Failures can occur at any layer. Examples:

• A source credential expired.
• A source API changed a schema.
• A Transformation Model test failed.
• A publish or materialization is still queued.
• A dashboard depends on an unpublished or stale model.
• A scheduled export failed.

Use Monitor for recurring jobs and run history. Use the Catalog to trace which assets depend on the failing layer.