Glossary

This glossary explains terms that are frequently used throughout iPCV documentation.

iPCV & Healthcare Terms

• DSA: Data Sharing Agreement.
• EMIS Now: Knowledge base and customer support platform. Learn more.
• EMIS Web: Clinical software used by healthcare organisations to record and manage patient information. The underlying EMIS Web database is one of the source systems used by Explorer and iPCV. Learn more.
• MKB: Medical Knowledge Base.
• Index: A central service used within the Explorer platform to track and manage information about data held within the EMIS Web estate.
• EXA: EMIS-X Analytics, scalable access to data for analytic applications. Learn more.
• NDOP (National Data Opt-Out): A service that allows patients to opt out of their confidential patient information being used for purposes beyond their individual care, such as research and planning. Where applicable, NDOP controls are applied in accordance with national policy and an organisation’s agreed data-sharing arrangements. Learn more.
• Incremental Primary Care Views (iPCV): A collection of analytics-focused data models that transform and pre-compute EMIS Web data into healthcare concepts that are easier to query and analyse.
• iPCV V1: The original implementation of Incremental Primary Care Views.
• iPCV V2: The latest implementation of Incremental Primary Care Views, including revised data models, processing logic and migration improvements.
• Transform Date Time (transform_datetime): Identifier representing a completed iPCV processing execution. Commonly used when consuming delta data and identifying newly available records.
• Replicator: The EMIS written software to pull data out of our source systems.
• Ingestor: The EMIS written process that takes the replicated data and merges it into the lake.
• Delta: A dataset containing records that have changed since a previous processing cycle. Delta datasets are commonly used to keep local reporting and analytics environments up to date without performing full data reloads.
• Data Freshness: A measure of how recently data available in iPCV reflects changes made in source clinical systems.
• Event Type: A field within delta datasets that indicates how a record should be processed. Supported values are update and deletion.
• Run Status / Data Freshness: Metadata that provides information about iPCV processing executions, including data availability, execution history and successful processing runs.

Technical & Data Engineering Terms

• ERD: Entity Relationship Diagram, a visual representation of the relationships between data entities in a database.
• SQL: Structured Query Language, there are many flavours of SQL due to modern technologies but there is also a foundation referred to as ANSI.
• Flavours: Different configurations of the same model.
• Backfilling: The act of populating either all or a section of data historically. Usually requires a monumental amount of resources.
• Grain: The lowest level of detail represented in a dataset, often defined by the unique identifier for a record. See the Kimball documentation.
• Soft delete: A technique for excluding data without removing it from a data system.
• Hard delete: As above, but actually removes the data rather than excluding it.
• Lookup table: A table that exists to exchange one value for another. Typically narrow in design.
• Bridge table: A common technique to help resolve many-to-many relationships. An example.
• Normalised data: A method of structuring a database to reduce redundancy and improve data integrity by organising data into tables and defining relationships between them. Learn more.
• Denormalised data: The opposite of normalisation. Compute is expensive (joins) and storage is cheap in the world of analytics, so some duplication and redundancy is added to improve performance.
• Natural duplicates: Data that is an exact copy for the whole row and can be removed by using DISTINCT or GROUP BY SQL syntax.
• Unnatural duplicates: Where the primary key remains the same but there are multiple rows and data differs between them. There is no simple fix for this; the issue should be pushed up to the source system.
• Upsert: An operation that inserts a record if it does not already exist, or updates it if it does.

Advanced Technical Terms

• Table formats: Originally using Hive, but other available formats include Iceberg, Delta Lake and Hudi.
• File formats: Typically referring to one of Parquet, ORC or Avro.
• Compression: There are different algorithms for compressing files; popular ones include ZSTD, Snappy, LZ4 and gzip.
• Partitioning: When talking about object storage (S3, GCS etc.) the partitioning scheme physically splits the data into different prefixes. This can help improve performance but there is a balance to be had.
• DAG: Directed Acyclic Graph, a workflow where tasks are executed in a defined order without circular dependencies.
• Orchestration: Synonym for scheduling — running tasks at a specified time in a specified order.
• Time travel: The ability to see what data was in a table at a point in time. Depends on whether the table format implements it and how much history is kept.