Definition

The Data Freshness model provides visibility into the currency and timeliness of data across the iPCV platform. This model tracks the start and end times for data processing, along with when data was last updated. This enables users to understand the recency of information and make informed decisions about data reliability and usage.

Information

The data model captures freshness metrics for data across various datasets and tables. It provides timestamps and indicators that help users assess whether the data meets their timeliness requirements for analysis and reporting.

Data refreshes for individual iPCV data models are carried out in parallel. The data is made available to customers as soon as each model is completed so customers do not need to wait for all models to complete before they start their data extractions.

In the Data Freshness model, several indicators are available to assess the currency of data:

• start_datetime - Indicates when the data processing period begins, representing the start of the data range included in the refresh.
• end_datetime - Shows when the data processing period ends, representing the end of the data range included in the refresh.
• last_updated_datetime - Indicates when the data processing was completed and the data was last refreshed or updated in the system.

Grain and Scope

The model contains one current row for each iPCV data model, identified by table_name. It covers the data models available in the iPCV platform and provides the date range and completion time for each model’s latest refresh.

Data models are refreshed independently. When a model’s daily run completes, its row in the Data Freshness model is updated. Customers do not need to wait for every model to finish.

Overview

flowchart LR
  n1["iPCV daily run"] --> n2["Individual model completes"]
  n2 --> n3["Data Freshness row updated"]

Examples

Check freshness for a specific table

SELECT
  table_name,
  last_updated_datetime,
  start_datetime,
  end_datetime
FROM
  hive.explorer_ipcv_vanilla.data_freshness
WHERE
  table_name = 'user_in_role';

Check freshness for all tables

SELECT
  table_name,
  last_updated_datetime,
  start_datetime,
  end_datetime
FROM
  hive.explorer_ipcv_vanilla.data_freshness
ORDER BY
  last_updated_datetime DESC;