The DataPulse AI user guide, published from the product documentation on 2026-10-13. Pictures use sample data. The admin guide and the Application Manual ship with the product.

Pipelines

Pipelines is one board for every Lakeflow job and every Lakeflow declarative pipeline across your workspaces. It shows what is failing, what is stuck, what it cost, and what breaks downstream. In the left navigation it is Pipelines.

Each row carries a Job or Pipeline badge next to its id, so you can always tell which you are looking at. DataPulse watches and repairs your pipelines; it never writes or changes their code.

How it works#

A job or pipeline is compared with its own history, not a fixed number. Stalled means it has run longer than twice its own p90 (the run time that 90% of its runs beat). A job has runs; a declarative pipeline has updates. The board treats them the same way.

Walkthrough 1: read the board#

1. Run Health Check. Checks your data SLAs now. Operators and admins only.
2. Failing card. Jobs whose latest run ended in error.
3. Wasted Cost card. Cost of failed and retried runs.
4. Sub-tabs. Job Board & Heatmap, Incidents & MTTR, Data SLAs.
5. Job Execution Health Matrix (Job × Day). One row per job, one square per day. Legend: OK, Slow / Stalled, Failed, No Run. Hover a square for details.

What each number means#

Status is always text, icon, and colour together: Failing, Stalled (>2× p90), Slow, Stale (SLA), and healthy. A pipeline update that was cancelled is not a failure: somebody stopped it, so there is nothing to diagnose or retry.

Walkthrough 2b: open a declarative pipeline#

Click Inspect on any row badged Pipeline. The drawer shows:

1. Update history. Every update in the window with its state, how long it took, what started it, what it cost, and any table it fully refreshed.
2. Failing flow. When an update failed, the flow (dataset) that failed and the error it reported, in its own words.
3. Expectations. How many records passed, were dropped, and failed your expectations in the window. If these are blank, the reason is shown — most often that only the pipeline's owner may read its event log in Lakeflow.
4. Cost and cost of failure. The pipeline's spend, and the part of it spent on updates that failed.
5. Settings. Failure alerts, edition, the Unity Catalog target, and the owner.
6. Hygiene. What DataPulse would change, and why.

What DataPulse checks on a pipeline#

A check that DataPulse could not run says so. It never reports "passed" for something it did not read.

Repairing a pipeline#

Two actions, both through the approval flow, both acting as you (your own permission on the pipeline decides):

• Retry update (low risk). Starts a fresh update after a failed one. It reprocesses the same data and truncates nothing. Only available when the newest update actually failed — otherwise the button is disabled and says why.
• Full refresh a table (high risk). Truncates one table and rebuilds it from the source. DataPulse never offers this for the whole pipeline, because that would empty every table it owns.

Neither action can be undone. DataPulse can stop a running update, but rows it has already written stay written, and a full refresh empties its table before the first new row lands. The confirm dialog says this before you approve. To get previous data back, restore the table from its Delta history (DESCRIBE HISTORY, then RESTORE TABLE).

If a failed update needs explaining#

Diagnose with AI reads the failed update's own error text and the failing flow, and answers with a cause, a category, and fix steps. Unlike the job version, it also opens an incident on the pipeline (or joins one already open), so the failure gets an owner, an MTTR and a permanent record. Your table rows are never sent to the model — only the error text.

Walkthrough 2: open a job#

1. Cross-Workspace Board. Sort, search, choose columns, or export CSV.
2. The status pill, for example Failing.
3. Inspect opens the job drawer. Clicking the job name does the same.
4. Repair starts a repair of the failed tasks.

The Hygiene column counts setup problems: no timeout, no retry policy, no owner, or scheduled work on all-purpose compute.

1. Duration p50 and p90.
2. Cost of Failure.
3. Task Execution Timeline. Each task is a bar. Red is the failed one. You can see which task broke and how long each took.
4. Root Cause Analysis (AI Diagnosis). The AI reads the error text and suggests a cause, fix steps, and a confidence. Treat it as a lead, not a verdict.
5. Repair Run. Re-runs only the failed tasks. Low risk.
6. Add Timeout. Stops a run that goes too long. High risk, so it needs an admin.

1. Root Cause Analysis.
2. Diagnose with AI. Choose a model and click it. It answers with a cause, fix steps and a confidence.

Below the timeline, Downstream Lineage Impact lists what Unity Catalog lineage shows downstream of the job's output table: the tables, and any dashboards, notebooks or jobs that read them (shown by id, because lineage carries no name). A table gets a Last failed run badge only when a job that is failing now writes it, and an SLA breach badge only when an SLA you defined on that table is breached; a table with neither has no badge. Open lineage in Catalog Explorer ↗ opens the full graph in Databricks.

The Repair Pipeline Run dialog shows the change, LOW RISK, and the rollback note. Confirm & Execute sends it through the approval path.

Walkthrough 3: work an incident#

1. Mean Time To Resolution (MTTR). The average hours to close an incident. Active Open Incidents and Total Tracked Incidents sit beside it.
2. Each incident shows a status pill: OPEN, ACK, or RESOLVED, its source (job, sla, or expectation), and how many failures were grouped into it.
3. Ack says you are looking at it. Resolve closes it. AI Explain writes the root cause card. These buttons need the operator role or higher.

Failures of the same job within a two-hour window are grouped into one incident.

Walkthrough 4: keep a data promise (Data SLAs)#

1. Each SLA shows its type and status, for example FRESHNESS and BREACHED, the target table, and the owner.
2. Evaluate SLAs Now checks them immediately.
3. New SLA creates one (operator or admin). Choose the table, the type, and the limit.

Readers and cleansing engineers see SLA Configuration Restricted (Viewer Role): creating or changing an SLA needs the operator or admin role.

Walkthrough 5: see what breaks downstream#

There is no separate lineage page. DataPulse shows the downstream impact where you decide something, and leaves the full graph (column lineage, upstream tables) to Databricks Catalog Explorer.

1. From a job. Open a job's drawer and read Downstream Lineage Impact. Click Open lineage in Catalog Explorer ↗ for the whole graph. Lineage ↗ in the telemetry strip at the top of the page opens the telemetry table the same way.
2. Before an action. In the Actions inbox, Approve on an action that targets a table shows a line such as Lineage shows N tables … downstream of <table> in the confirm dialog when the proposal found any, with the same link.
3. Before a cleansing run. In Data Cleansing, the Run and Productionize dialogs say Replacing <table> affects N tables … only when the target is an existing DataPulse table that something reads, and then show MEDIUM RISK. A new table shows no warning.

The counts come only from lineage: tables are always counted, and dashboards, notebooks or jobs only when lineage reported them. If lineage is not available you see a "lineage unavailable" message instead of a guess.

Known limitations#

If no pipelines appear#

If your account cannot read the Lakeflow pipeline tables, the board says so above the rows, with the exact GRANT to run. It never shows an empty list as if you had no pipelines. A pipeline is only shown to someone allowed to see it, so two people may see different rows.

Empty, denied, and failed states#

No jobs or pipelines found, No Data SLAs Configured, No open incidents, Pipeline Health Access Restricted, and Pipeline Telemetry Query Error are the messages you may see. See Troubleshooting.