Analytics etl metrics
1. Dataset Name
analytics_etl_metrics_dataset
2. Description
Pipeline-health dataset exposing per-table, per-day ETL load statistics for the analytics warehouse. Each row reports how many records were expected versus successfully loaded for a customer view on a given ingestion date, the resulting completion rate as a percentage, and the p90 data-freshness latency in minutes. Used internally to monitor data completeness and freshness SLAs behind the analytics product.
3. Source Tables
- customer_views.etl_latency_consistency_metric (single source table - no joins)
4. Grain
One row per source table per ingestion date.
5. Dashboards Supported
- Analytics Data Health / ETL Monitoring (Eightfold internal)
6. Metrics Supported
| Metric | Supported Breakdowns |
| Load Completion Rate | Table name, ingestion date |
| P90 Data Freshness Latency | Table name, ingestion date |
| Records Loaded vs. Expected | Table name, ingestion date |
| Tables Below SLA | Ingestion date (thresholded on completion_rate) |
7. Known Exclusions / Filters Applied
- No row-level filters are applied; the dataset is a projection with two cast/rescale operations.
- completion_rate is multiplied by 100 and cast to DECIMAL(10,5) - it is already a percentage and must not be re-multiplied in the visual.
- p90_data_freshness_latency_minutes is cast to DECIMAL(10,2) and is a p90, not a mean; do not average it across tables.
- Only tables covered by the ETL latency instrumentation appear - absence of a table is not evidence of a healthy load.
- This dataset is flagged isEfInternal = 1.
8. Schema
| Column Name | Data Type | Field Type | Notes |
| rds_table_name | varchar | Dimension | Name of the upstream source (RDS) table |
| table_name | varchar | Dimension | Name of the resulting customer_views table |
| ingestion_date | datetime | Dimension | Date of the ETL load |
| successfully_loaded_count | bigint | Measure | Records written successfully in the load |
| total_record_count | bigint | Measure | Records expected in the load |
| completion_rate | decimal | Measure | Percentage of expected records loaded (already scaled to 0-100) |
| p90_data_freshness_latency_minutes | decimal | Measure | 90th percentile lag in minutes between source write and warehouse availability |