Skip to content

MeaurementPhenotype

Bases: CodelistPhenotype

What is MeasurementPhenotype for?

The MeasurementPhenotype is for manipulating numerical data found in RWD data sources e.g. laboratory or observation results. These tables often contain numerical values (height, weight, blood pressure, lab results). As an event-based table, each row records a single measurement value for a single patient with a date. All numerical values are in a 'value' column. A medical code indicates the type of numerical measurement and the units of measurement are in an additional column.

MeasurementPhenotype is a subclass of CodelistPhenotype, inheriting all of its functionality to identify patients by single or sets of medical codes (e.g. 'test type') within a specified time period. It can also :

  • identify patients with a measurement value within a value range and
  • return a measurement value, either all measurements values within filter criteria or perform simple aggregations (mean, median, max, min).

Example data:

PersonID MedicalCode EventDate Value Unit
1 HbA1c 2010-01-01 4.2 %
1 HT 2010-01-02 121 cm
2 WT 2010-01-01 130 kg

Note on data cleaning

In general, data cleaning operations should be performed upstream of Phenex. This includes:

  • unit harmonization: ideally all values should be in the same unit for a given measurement type test. There are workarounds to deal with multiple units for a single measurement, but it is not recommended.

  • removing nonsensical values: e.g. negative blood pressures, or values outside of a physiological range. While MeasurementPhenotype provides a clean_nonphysiologicals_value_filter parameter to remove such values, is recommended to perform this operation upstream of apex.

  • dealing with duplicate entries: e.g. multiple entries for the same patient on the same day. The meaning of multiple entries on the same day may vary between data sources, so it is recommended to handle duplicate entries upstream of apex. In some EHR datasources, duplicate entries may suggest that this value is 'more accurate', as it is passed and recorded through multiple providers and systems, while in other datasources multiple entries may suggest faulty data entry.

Parameters:

Name Type Description Default
clean_nonphysiologicals_value_filter str

A value filter to be applied prior to any filtering or aggregation. This should be used to remove nonsensical values e.g. negative blood pressures, or values outside of a physiological range that are certain to be due to measurement error. Ideally, such cleaing steps should performed upstream of apex, but have been provided due to realization of practical necessity.

 None
value_aggregation str

A string representing the aggregation operation (mean, median, min, max) to be performed on the measurement values occurring on the same day. This operation occurs after the cleaning value filter but prior to the primary value_filter. This is also considered a cleaning step to deal with duplicate entries on the same day. In general, if duplicate entries on the same day are a consideration, handling should be done upstream of apex. If not specified or set to None, no daily aggregation is performed prior to the primary value filter.

 None
clean_null_values str

A boolean indicating whether to remove null values from the measurement table. If set to True, null values are removed prior to value filtering. If set to False, null values are not removed. If not specified, null values are removed (default is true)

 True
value_filter str

A value filter to be applied to the measurement values. This filter is applied after the clean_nonphysiologicals_value_filter and the value_aggregation. This filter is used to identify patients with a measurement value within a value range. If not specified, no value filter is applied. For example, to identify patients with a 1. systolic blood pressure above 120 mmHg, the value_filter would be set to ValueFilter(operator='>', value=120). 2. systolic blood pressure above 120 mmHg but below 140 mmHg, the value_filter would be set to ValueFilter(operator='>', value=120) & ValueFilter(operator='<', value=140)

 None
return_value str

A string representing if a value should be returned, and if so, what, if any, aggregation should be performed. Any aggregation operations occurs after the value_filter, and thus do not influence the filtering of patients. Possible options are "daily_mean", "daily_median", "daily_min", "daily_max", "daily_sum", "mean", "median", "min", "max", and "all". If not specified, no values are returned. If a "daily" aggregation is specified, return_date must also be specified in order to specify which on which date the aggregation should be performed.

 None
further_value_filter_phenotype str

If the input to the current MeasurementPhenotype is the output of a previous MeasurementPhenotype, set this parameter to the previous MeasurementPhenotype.

 None
Source code in phenex/phenotypes/measurement_phenotype.py 
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
class MeasurementPhenotype(CodelistPhenotype):
    """
    # What is MeasurementPhenotype for?
    The MeasurementPhenotype is for manipulating numerical data found in RWD data sources e.g. laboratory or observation results. These tables often contain numerical values (height, weight, blood pressure, lab results). As an event-based table, each row records a single measurement value for a single patient with a date. All numerical values are in a 'value' column. A medical code indicates the type of numerical measurement and the units of measurement are in an additional column.

    MeasurementPhenotype is a subclass of CodelistPhenotype, inheriting all of its functionality to identify patients by single or sets of medical codes (e.g. 'test type') within a specified time period. It can also :

    - identify patients with a measurement value within a value range and
    - return a measurement value, either all measurements values within filter
      criteria or perform simple aggregations (mean, median, max, min).

    # Example data:

    | PersonID    |   MedicalCode   |   EventDate   |   Value    | Unit|
    |-------------|-----------------|---------------|------------|-----|
    | 1           |   HbA1c         |   2010-01-01  |   4.2      | %   |
    | 1           |   HT            |   2010-01-02  |   121      | cm  |
    | 2           |   WT            |   2010-01-01  |   130      | kg  |

    # Note on data cleaning
    In general, data cleaning operations should be performed upstream of Phenex.
    This includes:

    - unit harmonization: ideally all values should be in the same unit for a given measurement type test. There are workarounds to deal with multiple units for a single measurement, but it is not recommended.

    - removing nonsensical values: e.g. negative blood pressures, or values outside of a physiological range. While MeasurementPhenotype provides a clean_nonphysiologicals_value_filter parameter to remove such values,  is recommended to perform this operation upstream of apex.

    - dealing with duplicate entries: e.g. multiple entries for the same patient on the same day. The meaning of multiple entries on the same day may vary between data sources, so it is recommended to handle duplicate entries upstream of apex. In some EHR datasources, duplicate entries may suggest that this value is 'more accurate', as it is passed and recorded through multiple providers and systems, while in other datasources multiple entries may suggest faulty data entry.


    Parameters:
        clean_nonphysiologicals_value_filter (str): A value filter to be applied **prior** to any filtering or aggregation. This should be used to remove nonsensical values e.g. negative blood pressures, or values outside of a physiological range that are certain to be due to measurement error. Ideally, such cleaing steps should performed upstream of apex, but have been provided due to realization of practical necessity.
        value_aggregation (str): A string representing the aggregation operation (mean, median, min, max) to be performed on the measurement values occurring on the same day. This operation occurs **after** the cleaning value filter but **prior** to the primary value_filter. This is also considered a cleaning step to deal with duplicate entries on the same day. In general, if duplicate entries on the same day are a consideration, handling should be done upstream of apex. If not specified or set to None, no daily aggregation is performed prior to the primary value filter.
        clean_null_values (str): A boolean indicating whether to remove null values from the measurement table. If set to True, null values are removed prior to value filtering. If set to False, null values are not removed. If not specified, null values are removed (default is true)
        value_filter (str): A value filter to be applied to the measurement values. This filter is applied **after** the clean_nonphysiologicals_value_filter and the value_aggregation. This filter is used to identify patients with a measurement value within a value range. If not specified, no value filter is applied. For example, to identify patients with a 1. systolic blood pressure above 120 mmHg, the value_filter would be set to ValueFilter(operator='>', value=120). 2. systolic blood pressure above 120 mmHg but below 140 mmHg, the value_filter would be set to ValueFilter(operator='>', value=120) & ValueFilter(operator='<', value=140)
        return_value (str): A string representing if a value should be returned, and if so, what, if any, aggregation should be performed. Any aggregation operations occurs **after** the value_filter, and thus do not influence the filtering of patients. Possible options are "daily_mean", "daily_median", "daily_min", "daily_max", "daily_sum", "mean", "median", "min", "max", and "all". If not specified, no values are returned. If a "daily" aggregation is specified, return_date must also be specified in order to specify which on which date the aggregation should be performed.
        further_value_filter_phenotype (str): If the input to the current MeasurementPhenotype is the output of a previous MeasurementPhenotype, set this parameter to the previous MeasurementPhenotype.
    """

    def __init__(
        self,
        value_filter: Optional["ValueFilter"] = None,
        clean_nonphysiologicals_value_filter: Optional["ValueFilter"] = None,
        clean_null_values: Optional[bool] = True,
        value_aggregation: Optional[str] = None,
        return_value: Optional[str] = None,
        further_value_filter_phenotype: Optional["MeasurementPhenotype"] = None,
        **kwargs,
    ):
        # Default value of return_date in codelist_phenotype is 'first'. This is not helpful behavior for measurementphenotype as we will perform further operations that require all values. For example, if we want the mean of all values in the post index period, setting return_date = 'first' will return only the values on the first day
        if "return_date" not in kwargs:
            kwargs["return_date"] = "all"
        super(MeasurementPhenotype, self).__init__(
            **kwargs,
        )
        self.clean_nonphysiologicals_value_filter = clean_nonphysiologicals_value_filter
        self.clean_null_values = clean_null_values
        self.value_filter = value_filter
        self.value_aggregation = value_aggregation
        self.return_value = return_value
        self.further_value_filter_phenotype = further_value_filter_phenotype

        if self.further_value_filter_phenotype is not None:
            self.children.append(self.further_value_filter_phenotype)

    def _execute(self, tables) -> PhenotypeTable:
        # perform codelist filtering
        # perform nonphysiological value filtering
        # perform value aggreation
        # perform value filter
        # perform value and dateaggregation
        code_table = tables[self.domain]
        code_table = self._perform_codelist_filtering(code_table)
        code_table = self._perform_time_filtering(code_table)
        code_table = self._perform_date_selection(code_table, reduce=False)
        code_table = self._perform_nonphysiological_value_filtering(code_table)
        code_table = self._perform_value_aggregation(code_table)
        code_table = self._perform_value_filtering(code_table)
        return select_phenotype_columns(code_table)

    def _perform_nonphysiological_value_filtering(self, code_table):
        if self.clean_nonphysiologicals_value_filter is not None:
            code_table = self.clean_nonphysiologicals_value_filter.execute(code_table)
        return code_table

    def _perform_value_aggregation(self, code_table):
        if self.value_aggregation is not None:
            code_table = self.value_aggregation.aggregate(code_table)
        return code_table

    def _perform_value_filtering(self, code_table):
        if self.value_filter is not None:
            code_table = self.value_filter.filter(code_table)
        return code_table

namespaced_table property

A PhenotypeTable has generic column names 'person_id', 'boolean', 'event_date', and 'value'. The namespaced_table appends the phenotype name to all of these columns. This is useful when joining multiple phenotype tables together.

Returns:

Name Type Description
table Table

The namespaced table for the current phenotype.

execute(tables)

Executes the phenotype computation for the current object and its children. This method recursively iterates over the children of the current object and calls their execute method if their table attribute is None.

Parameters:

Name Type Description Default
tables Dict[str, PhenexTable]

A dictionary mapping table names to PhenexTable objects. See phenex.mappers.DomainsDictionary.get_mapped_tables().

 required

Returns:

Name Type Description
table PhenotypeTable

The resulting phenotype table containing the required columns. The PhenotypeTable will contain the columns: PERSON_ID, EVENT_DATE, VALUE. DATE is determined by the return_date parameter. VALUE is different for each phenotype. For example, AgePhenotype will return the age in the VALUE column. A MeasurementPhenotype will return the observed value for the measurement. See the specific phenotype of interest to understand more.
Source code in phenex/phenotypes/phenotype.py 
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
def execute(self, tables: Dict[str, Table]) -> PhenotypeTable:
    """
    Executes the phenotype computation for the current object and its children. This method recursively iterates over the children of the current object and calls their execute method if their table attribute is None.

    Args:
        tables (Dict[str, PhenexTable]): A dictionary mapping table names to PhenexTable objects. See phenex.mappers.DomainsDictionary.get_mapped_tables().

    Returns:
        table (PhenotypeTable): The resulting phenotype table containing the required columns. The PhenotypeTable will contain the columns: PERSON_ID, EVENT_DATE, VALUE. DATE is determined by the return_date parameter. VALUE is different for each phenotype. For example, AgePhenotype will return the age in the VALUE column. A MeasurementPhenotype will return the observed value for the measurement. See the specific phenotype of interest to understand more.
    """
    logger.info(f"Phenotype '{self.name}': executing...")
    for child in self.children:
        if child.table is None:
            logger.debug(
                f"Phenotype {self.name}: executing child phenotype '{child.name}'..."
            )
            child.execute(tables)
        else:
            logger.debug(
                f"Phenotype {self.name}: skipping already computed child phenotype '{child.name}'."
            )

    table = self._execute(tables).mutate(BOOLEAN=True)

    if not set(PHENOTYPE_TABLE_COLUMNS) <= set(table.columns):
        raise ValueError(
            f"Phenotype {self.name} must return columns {PHENOTYPE_TABLE_COLUMNS}. Found {table.columns}."
        )

    self.table = table.select(PHENOTYPE_TABLE_COLUMNS)
    # for some reason, having NULL datatype screws up writing the table to disk; here we make explicit cast
    if type(self.table.schema()["VALUE"]) == ibis.expr.datatypes.core.Null:
        self.table = self.table.cast({"VALUE": "float64"})

    assert is_phenex_phenotype_table(self.table)
    logger.info(f"Phenotype '{self.name}': execution completed.")
    return self.table

get_codelists()

Get all codelists used in the phenotype definition, including all children / dependent phenotypes.

Returns:

Name Type Description
codeslist List[Codelist]

A list of codelists used in the cohort definition.
Source code in phenex/phenotypes/codelist_phenotype.py 
167
168
169
170
171
172
173
174
175
176
177
def get_codelists(self) -> List[Codelist]:
    """
    Get all codelists used in the phenotype definition, including all children / dependent phenotypes.

    Returns:
        codeslist: A list of codelists used in the cohort definition.
    """
    codelists = [self.codelist]
    for p in self.children:
        codelists.extend(p.get_codelists())
    return codelists