Cross-Domain Changes

This page describes changes introduced in iPCV V2 that affect multiple domains and data models across the platform. For model-specific changes, refer to the relevant domain change pages in the model documentation.

Optimised Data Ingestion Timing

The start time of the data ingestion process has moved from midnight to 11 PM UTC every day. This delivers EMIS Web data updates earlier, giving access to fresh data faster.

Action required: Review any schedules, pipelines, reports or downstream processes that depend on data refresh timings and update them to align with the new ingestion schedule. Inform relevant team members about the change to ensure smooth adaptation.

Standardised Column Names and New Datetime Fields

New columns and standardised column names have been introduced across iPCV V2 to improve consistency and support more precise time-based filtering.

Key changes:

A new transform_datetime column has been added to all views, showing the exact datetime the data was made available in the model.
All bulk views (except MKB-related models) that previously had _ingest_time now also include a formatted load_datetime column for faster query response.
Ingest time, deletion status and execution date are now consistently represented as _ingest_time, is_deleted and _execution_date across all iPCV V2 flavours.

Action required: Update any queries or applications that reference old column names to use the new standardised names. Use the new datetime columns in filters where applicable.

Simplified and Faster Delta Views

Delta processing has been simplified in V2 to improve performance and reduce ingestion complexity.

Changes:

Delta tables now include only one entry per record, containing the latest update within the last 28 days.
All changes — whether additions or updates — are presented as update records. There is no longer a distinction between new and updated records.
event_type now contains only 'update' or 'deletion'.
_ingest_time and is_deleted have been removed from delta tables.
Delta data is generated at the point of data insertion, eliminating additional post-processing steps and making it available sooner.

Action required:

Treat all 'update' records as upserts, even if the record is new or absent in the target system.
If previously ingesting deltas sequentially by execution_date, this can continue but is no longer required.
Remove any CTEs or deduplication logic previously used to handle multiple delta versions of a record.
For changes older than 28 days, use the main incremental tables directly.

Removal of FHIR-Derived Fields

Fields that originated from FHIR specifications have been removed from the relevant data models. This affects fields such as fhir_patient_active, fhir_registration_type and similar attributes.

These fields have been removed because:

The FHIR version implemented (3.0) is outdated compared to current standards (5.0), creating compliance gaps.
FHIR-derived fields often misrepresent EMIS Web data, leading to inaccurate clinical and analytical outputs.
EMIS-native fields already provide the required functionality without the added complexity.

Action required: Update ETL pipelines, queries and transformations to remove references to deprecated FHIR fields. Use the EMIS-native alternatives listed in the table below.

Data Model	Removed FHIR field	EMIS-native alternative
Patient	`fhir_patient_active`	Use `is_registered` and `is_active`
Patient	`fhir_registration_type`	Use `patient_type`
Allergy	`fhir_status`	No replacement needed (was hard-coded)
Allergy	`fhir_category`	Use `category`
Allergy	`fhir_episodicity`	Use `emis_episodicity`
Medication	`intent` / `fhir_medication_intent`	TBD
Medication	`agency` / `nhs_prescribing_agency`	No replacement needed (was hard-coded)
Medication	`type` / `nhs_prescription_type`	Use `emis_prescription_type`
Medication	`status` / `fhir_medication_status`	Use `emis_drug_status`; derive issue record status from cancellation flag
Observation	`interpretation_code` / `fhir_interpretation_code`	Use `abnormal_flag`
Problem	`fhir_problem_significance_description` / `fhir_significance`	Use `significance`
Problem	`relation` / `fhir_relation`	Use `relation`
Problem	`status` / `fhir_status`	Use `problem_status`
Referral	`nhs_priority_description`	Use `urgency`
Referral	`priority` / `nhs_priority`	Use `urgency`
Referral	`source` / `nhs_source`	Use `direction`

MKB Version Column Removed

The mkb_version column and associated version management columns have been removed from iPCV V2. The _record_version, _mkb_version and _ingest_time columns are also removed. Only the latest version of drug codes is now displayed; version management is handled internally.

Action required: Review any queries or applications that explicitly reference mkb_version or the removed MKB version columns and remove those references.

Streamlined ODS Code Management

ODS Code has been removed from all data models except Organisation. Customers who need ODS Code must now join their model to Organisation using the organisation key.

This change was made because previously ODS Code existed in multiple large models. When NHS documentation updated an ODS Code, it required re-processing millions of rows across all models. By centralising ODS Code in Organisation, only a single record needs updating, significantly reducing ETL cost and compute overhead.

Action required: Update any queries that previously referenced ODS Code directly from clinical or other models to join to Organisation instead.

SELECT p.patient_id, p.registration_organisation_id, o.ods_code
FROM hive.<flavour_schema>.patient_v2 AS p
JOIN hive.<flavour_schema>.organisation_v2 AS o
  ON p.registration_organisation_id = o.organisation_id
  AND p.organisation = o.organisation;

Review any ETL workflows and reports or dashboards that used ODS Code to confirm they now reference Organisation.

Consolidation of Patient Filter and Opt-Out Columns

All patient-level filter and opt-out flags have been consolidated into the Patient model. These columns have been removed from clinical event models in iPCV V2.

Affected models: Observation, Allergy, Immunisation, Consultation (Encounter), Medication, Recall (Diary).

Note: The Observation Opt-Out view, which was available in some flavours in iPCV V1, is no longer available in any flavour in V2. All patient filter and opt-out data is now exclusively in the Patient model.

The following columns have been removed from clinical models:

confidential_patient_flag
dummy_patient_flag
opt_out_93c1_flag
opt_out_9nd19nu09nu4_flag
opt_out_9nd19nu0_flag
opt_out_9nu0_flag
non_regular_and_current_active_flag
regular_and_current_active_flag
regular_current_active_and_inactive_flag
regular_patient_flag
sensitive_patient_flag

Record-level confidentiality and sensitivity flags (confidential_flag, sensitive_flag) are retained in clinical models as these indicate record-specific attributes, not patient-level attributes.

Action required: Update any queries that previously used patient filter columns from clinical models to join to patient_v2 instead.

-- V2: join to patient_v2 to access patient filter and opt-out flags
SELECT
  r.diary_id,
  r.emis_diary_guid,
  r.emis_patient_id,
  r.recorded_date,
  p.is_regular,
  p.is_opt_out_9nu0,
  p.is_dummy
FROM hive.explorer_ipcv_vanilla.recall_v2 AS r
LEFT JOIN hive.explorer_ipcv_vanilla.patient_v2 AS p
  ON r.emis_patient_id = p.emis_patient_id
  AND r.organisation = p.organisation
WHERE r.is_deleted = FALSE
  AND p.is_regular = TRUE
  AND COALESCE(p.is_opt_out_9nu0, FALSE) = FALSE
  AND COALESCE(p.is_dummy, FALSE) = FALSE;

If you were using the Observation Opt-Out model in V1, migrate to Patient model joins as shown above. For detailed information on available opt-out flags and their meanings, refer to the Patient model documentation.

The Patient model in V2 includes patients that are not in scope (with sensitive data nulled) to enable full understanding of patient status, so you can identify why a patient might be excluded from your dataset.