flexmeasures.data.schemas.sensors

Module Attributes

Functions

flexmeasures.data.schemas.sensors.floor_bdf_event_starts(bdf: BeliefsDataFrame, event_resolution: timedelta) → BeliefsDataFrame

Classes

One inflexible device: a sensor reference with optional source filters.

Used both in the flex-context (as a list, for site-level inflexible load), and in a flex-model entry (as a single reference, when an inflexible device is modelled as its own asset). Deserializes to a plain Sensor when no source filters are given (a backward-compatible shape downstream), and to a SensorReference otherwise.

class Meta

fields: dict[str, Field]: Dictionary mapping field_names -> Field objects

refuse_bounds(data: dict, **kwargs): Refuse the bounds this schema inherits, since scheduling does not apply them yet.

Sensor reference for recording generated data.

fields: dict[str, Field]: Dictionary mapping field_names -> Field objects

class flexmeasures.data.schemas.sensors.PriceField(to_unit, *args, default_src_unit: str | None = None, return_magnitude: bool = False, timezone: str | None = None, event_resolution: timedelta | None = None, value_validator: Validator | None = None, additional_sensor_units: list[str] | None = None, **kwargs)

VariableQuantityField for monetary values.

Price fields participate in currency validation: all price fields in the flex-context must share one currency (recorded as the flex-context’s shared_currency_unit), and price fields in a flex-model must use a currency that is convertible to the flex-context’s shared currency.

class flexmeasures.data.schemas.sensors.QuantityOrSensor(*args, **kwargs)

__init__(*args, **kwargs): Deprecated class. Use VariableQuantityField instead.

Represents a quantity string like ‘1 EUR/MWh’.

fields: dict[str, Field]: Dictionary mapping field_names -> Field objects

class flexmeasures.data.schemas.sensors.RepurposeValidatorToIgnoreSensorsAndLists(original_validator, *, error: str | None = None)

Validator that executes another validator (the one you initialize it with) only on non-Sensor and non-list values.

__init__(original_validator, *, error: str | None = None)

Schema for uploading a file with sensor data. This one describes only the file upload part, not the sensor itself. See SensorDataFileSchema for the full schema.

fields: dict[str, Field]: Dictionary mapping field_names -> Field objects

class flexmeasures.data.schemas.sensors.SensorDataFileRequestSchema(*args, source_user: User | None = None, **kwargs)

Validate a sensor data upload without parsing or resampling its files.

fields: dict[str, Field]: Dictionary mapping field_names -> Field objects

post_load(fields, **kwargs): Process the deserialized and validated fields. Remove the ‘sensor’ and ‘files’ fields, and add the ‘data’ field containing a list of BeliefsDataFrames.

class flexmeasures.data.schemas.sensors.SensorDataFileSchema(*args, source_user: User | None = None, **kwargs)

__init__(*args, source_user: User | None = None, **kwargs)

fields: dict[str, Field]: Dictionary mapping field_names -> Field objects

post_load(fields, **kwargs): Process the deserialized and validated fields. Remove the ‘sensor’ and ‘files’ fields, and add the ‘data’ field containing a list of BeliefsDataFrames.

validate_unit(data, **kwargs): Validate unit compatibility with the sensor’s unit.

validate_uploaded_files(files: list[FileStorage], **kwargs): Validate the deserialized fields.

class flexmeasures.data.schemas.sensors.SensorIdField(asset: GenericAsset | None = None, unit: str | Quantity | None = None, *args, **kwargs)

Field that deserializes to a Sensor and serializes back to an integer.

__init__(asset: GenericAsset | None = None, unit: str | Quantity | None = None, *args, **kwargs)

_deserialize(value: Any, attr, data, **kwargs) → Sensor: Turn a sensor id into a Sensor.

_serialize(value: Sensor, attr, obj, **kwargs) → int: Turn a Sensor into a sensor id.

class flexmeasures.data.schemas.sensors.SensorIdOrReferenceField(*args, **kwargs)

Field accepting either a sensor ID or a sensor reference, which may filter by source and carry cleaning bounds.

__init__(*args, **kwargs)

_deserialize(value: Any, attr, data, **kwargs) → Sensor | SensorReference

Deserialize value. Concrete Field classes should implement this method.

Parameters:

value – The value to be deserialized.
attr – The attribute/key in data to be deserialized.
data – The raw input data passed to the Schema.load <marshmallow.Schema.load>.
kwargs – Field-specific keyword arguments.

Raises:

ValidationError – In case of formatting or validation failure.

Returns:

The deserialized value.

Changed in version 3.0.0: Added **kwargs to signature.

_serialize(value: Sensor | SensorReference, attr, obj, **kwargs) → int | dict[str, Any]

Serializes value to a basic Python datatype. Noop by default. Concrete Field classes should implement this method.

Example:

class TitleCase(Field):
    def _serialize(self, value, attr, obj, **kwargs):
        if not value:
            return ""
        return str(value).title()

Parameters:

value – The value to be serialized.
attr – The attribute or key on the object to be serialized.
obj – The object the value was pulled from.
kwargs – Field-specific keyword arguments.

Returns:

The serialized value

class flexmeasures.data.schemas.sensors.SensorReference(sensor: Sensor, source_types: list[str] | None = None, exclude_source_types: list[str] | None = None, sources: list[DataSource] | None = None, source_account: list[Account] | None = None, default: ur.Quantity | None = None, lower: Any = None, upper: Any = None, snap: dict = <factory>)

A sensor reference that wraps a Sensor with optional query settings.

Exposes the same unit, id, name, event_resolution and timezone properties as a plain Sensor, so code that reads those properties works without modification. The source filters and optional default value are passed through to get_series_from_quantity_or_sensor().

__init__(sensor: Sensor, source_types: list[str] | None = None, exclude_source_types: list[str] | None = None, sources: list[DataSource] | None = None, source_account: list[Account] | None = None, default: ur.Quantity | None = None, lower: Any = None, upper: Any = None, snap: dict = <factory>) → None

property _parsed_bounds: tuple[float | None, float | None, list[tuple[float, float, float]]]: The bounds as magnitudes in the sensor’s unit, parsed once per reference rather than once per read.

apply_bounds(values: ndarray) → ndarray

Snap and clip readings taken from the sensor, in the sensor’s own unit.

Parameters:: values – Readings in the unit of the referenced sensor.
Returns:: The readings, cleaned by this reference’s bounds, or unchanged if it has none.
Raises:: ValueError – If a bound cannot be read in the sensor’s unit (normally caught when the reference is loaded).

property event_resolution: timedelta: Event resolution of the underlying sensor.

property has_bounds: bool: Whether this reference asks for its readings to be cleaned at all.

property id: int: ID of the underlying sensor.

property name: str: Name of the underlying sensor.

property timezone: str: Timezone of the underlying sensor.

property unit: str: Unit of the underlying sensor.

Sensor reference with optional source filters, fallback value and cleaning bounds.

class Meta

fields: dict[str, Field]: Dictionary mapping field_names -> Field objects

remove_unset_default_and_bounds(data: dict, **kwargs) → dict

Leave out default and the bounds entirely when the reference does not define them.

Without this, references that set no fallback and no bounds would serialize default: None and empty bound keys, which are not valid input on the way back in. A zero bound is meaningful, so only None and an empty snap mapping are dropped.

validate_bounds(data: dict, **kwargs)

Fail fast on a bound that cannot be applied to the referenced sensor.

The sensor is already loaded at this point, so besides whether each bound can be read as a quantity, this also checks that it is compatible with the sensor’s unit, that each snap target lies within its interval, and that the lower bound does not exceed the upper bound.

class flexmeasures.data.schemas.sensors.SensorSchema(*args, **kwargs)

Sensor schema with validations.

class Meta

model: alias of Sensor

opts: LoadInstanceMixin.Opts = <flask_marshmallow.sqla.SQLAlchemySchemaOpts object>

Base sensor schema.

Here we include all fields which are implemented by timely_beliefs.SensorDBMixin All classes inheriting from timely beliefs sensor don’t need to repeat these. In a while, this schema can represent our unified Sensor class.

When subclassing, also subclass from ma.SQLAlchemySchema and add your own DB model class, e.g.:

class Meta:
model = Asset

fields: dict[str, Field]: Dictionary mapping field_names -> Field objects

set_default_timezone(data, **kwargs): Set the default timezone to the server timezone only for a full load (POST, not PATCH).

timezone_validator(): Validate timezone, suggesting the closest match if possible or the server default otherwise.

fields: dict[str, Field]: Dictionary mapping field_names -> Field objects

class flexmeasures.data.schemas.sensors.TimeSeriesOrSensor(*args, **kwargs)

__init__(*args, **kwargs): Deprecated class. Use VariableQuantityField instead.

List of time series segments.

fields: dict[str, Field]: Dictionary mapping field_names -> Field objects

class flexmeasures.data.schemas.sensors.TimedEventSchema(timezone: str | None = None, event_resolution: timedelta | None = None, value_validator: Validator | None = None, to_unit: str | None = None, default_src_unit: str | None = None, return_magnitude: bool = True, *args, **kwargs)

A time period (or single point) with a value.

Parameters:: timezone – Optionally, set a timezone to be able to interpret nominal durations.

check_time_window(data, **kwargs)

Checks whether a complete time interval can be derived from the timing fields.

The data is updated in-place, guaranteeing that the ‘start’ and ‘end’ fields are filled out.

fields: dict[str, Field]: Dictionary mapping field_names -> Field objects

class flexmeasures.data.schemas.sensors.VariableQuantityField(to_unit, *args, default_src_unit: str | None = None, return_magnitude: bool = False, timezone: str | None = None, event_resolution: timedelta | None = None, value_validator: Validator | None = None, additional_sensor_units: list[str] | None = None, **kwargs)

__init__(to_unit, *args, default_src_unit: str | None = None, return_magnitude: bool = False, timezone: str | None = None, event_resolution: timedelta | None = None, value_validator: Validator | None = None, additional_sensor_units: list[str] | None = None, **kwargs)

Field for validating, serializing and deserializing a variable quantity.

A variable quantity can be represented by a sensor, time series or fixed quantity.

# todo: Sensor should perhaps deserialize already to sensor data

NB any value validators passed are only applied to Quantities. For example, value_validator=validate.Range(min=0) will raise a ValidationError in case of negative quantities, but will let pass any sensor that has recorded negative values.

Parameters:

to_unit –
Unit to which the sensor, time series or quantity should be convertible. - Sensors are checked for convertibility, but the original sensor is returned,

so its values are not yet converted.
- Time series and quantities are already converted to the given unit.
- Units starting with ‘/’ (e.g. ‘/MWh’) lead to accepting any value, which will be converted to the given unit. For example, a quantity of 1 EUR/kWh with to_unit=’/MWh’ is deserialized to 1000 EUR/MWh.
default_src_unit – What unit to use in case of getting a numeric value. Does not apply to time series or sensors. In case to_unit is dimensionless, default_src_unit defaults to dimensionless; as a result, numeric values are accepted.
return_magnitude – In case of getting a time series, whether the result should include the magnitude of each quantity, or each Quantity object itself.
timezone – Only used in case a time series is specified and one of the timed events in the time series uses a nominal duration, such as “P1D”.
additional_sensor_units – Additional sensor units (besides those convertible to to_unit) that are accepted for sensor references. Use this only for units that are dimensionally incompatible with to_unit but contextually meaningful — for example, ["%"] allows sensors with a percentage unit for fields where the conversion requires an external capacity factor (such as soc-max). The actual unit conversion must be handled downstream by the caller. Do not use this as a general-purpose unit allowlist.

Deserialize value. Concrete Field classes should implement this method.

Parameters:

value – The value to be deserialized.
attr – The attribute/key in data to be deserialized.
data – The raw input data passed to the Schema.load <marshmallow.Schema.load>.
kwargs – Field-specific keyword arguments.

Raises:

ValidationError – In case of formatting or validation failure.

Returns:

The deserialized value.

Changed in version 3.0.0: Added **kwargs to signature.

_deserialize_default(value, attr, data, **kwargs) → Quantity: Deserialize a sensor reference fallback value.

_deserialize_dict(value: dict[str, Any], attr, data, **kwargs) → Sensor | SensorReference

Deserialize a sensor reference to a Sensor or SensorReference.

Returns a plain Sensor when no source filter or default keys are present (backward compatible), and a SensorReference when any of source-types, exclude-source-types, sources, source-account or default are provided.

_deserialize_list(value: list[dict]) → list[dict]: Deserialize a time series to a list of timed events.

_deserialize_numeric(value: Real, attr, data, **kwargs) → Quantity: Try to deserialize a numeric value to a Quantity, using the default_src_unit.

_deserialize_source_filters(value: dict[str, Any]) → tuple[list[str] | None, list[str] | None, list[DataSource] | None, list[Account] | None]

Deserialize and validate source filter fields from a sensor-reference dict.

Returns (source_types, exclude_source_types, sources, source_account).

_deserialize_str(value: str) → Quantity: Deserialize a string to a Quantity.

_get_original_unit(serialized_variable_quantity: str | list[dict] | dict, deserialized_variable_quantity: Quantity | list[dict] | Sensor | SensorReference) → str: Obtain the original unit from the still serialized variable quantity.

_get_unit(variable_quantity: Quantity | list[dict] | Sensor | SensorReference) → str

Obtain the unit from the (deserialized) variable quantity.

>>> VariableQuantityField("MW")._get_unit(ur.Quantity("3 kWh"))
'kWh'
>>> VariableQuantityField("/MW")._get_unit([{'value': ur.Quantity("3 kEUR/MWh")}, {'value': ur.Quantity("0 EUR/kWh")}])
'kEUR/MWh'

_serialize(value: Sensor | SensorReference | Series | Quantity, attr, obj, **kwargs) → str | dict[str, Any]

Serializes value to a basic Python datatype. Noop by default. Concrete Field classes should implement this method.

Example:

class TitleCase(Field):
    def _serialize(self, value, attr, obj, **kwargs):
        if not value:
            return ""
        return str(value).title()

Parameters:

value – The value to be serialized.
attr – The attribute or key on the object to be serialized.
obj – The object the value was pulled from.
kwargs – Field-specific keyword arguments.

Returns:

The serialized value

get_obj_type(obj): Returns name of the schema during dump() calls, given the object being dumped.