flagit package

Submodules

flagit.flagit module

exception flagit.flagit.FormatError

Bases: Exception

class flagit.flagit.Interface(data, sat_point=None, depth_from=None)

Bases: object

class provides interface to apply ISMN quality control procedures to in situ soil moisture data.

upon initialization it checks if the provided DataFrame has the required format. Quality control procedures can then be applied using the Interface.run function. The flags are provided as additional tags in column “qflag” which is one of three main categories: C (exceeding plausible geophysical range), D (questionable/dubious) or G (good). For a detailed description of the algorithms please see: Dorigo, W. A., Xaver, A., Vreugdenhil, M., Gruber, A., Hegyiova, A., Sanchis-Dufau, A. D., … & Drusch, M. (2013). Global automated quality control of in situ soil moisture data from the International Soil Moisture Network. Vadose Zone Journal, 12(3), doi:10.2136/vzj2012.0097.

If variable is not soil moisture but one of the following list: [soil temperature, air temperature, precipitation, soil suction, snow water equivalent, snow depth, soil surface temperature] then for the ISMN quality control only threshold flags (c01 and c02) are applied.

It is required that the first column of the dataframe is equal to the name of the variable.

Parameters:

• data (pandas.DataFrame) – Input for Interface Object containing in situ soil moisture measurements

• sat_point (float) – Saturation Point in % vol for soil at the respective location. At ISMN the saturation point is calculated from Harmonized World Soil Database (HWSD) sand, clay and organic content for each station using Equations [2,3,5] from Saxton & Rawls (2006). (Saxton, K. E., & Rawls, W. J. (2006). Soil water characteristic estimates by texture and organic matter for hydrologic solutions. Soil science society of America Journal, 70(5), 1569-1578. doi:10.2136/sssaj2005.0117)

Raises:

FormatError – if provided Input is no DataFrame or does not meet the required format

data

DataFrame containing in situ soil moisture measurement

Type:

pandas.DataFrame

run(name, sat_point)

Apply ISMN quality control to in situ soil moisture measurements

apply_savgol() → None

Calculates and adds derivations 1 and 2 using Savitzky-Golay filter

flag_C01(tag)

Soil moisture below threshold: Flags when measurement is below threshold

Parameters:

• tag (string or int, optional) –

• met (code added to qflag-column when flag-criteria are) –

flag_C02(tag)

Soil moisture above threshold: Flags when measurement is above threshold

Parameters:

• tag (string or int, optional) –

• met (code added to qflag-column when flag-criteria are) –

flag_C03(tag)

Soil moisture above saturation point: Flags when soil moisture is above saturation point. At ISMN the saturation point is calculated from Harmonized World Soil Database (HWSD) sand, clay and organic content for each station using Equations [2,3,5] from Saxton & Rawls (2006). doi:10.2136/vzj2012.0097.

Parameters:

• tag (string) –

• met (code added to qflag-column when flag-criteria are) –

flag_D01(tag)

In situ soil temperature below threshold: Flags when ancillary in situ soil temperature is below threshold

Parameters:

• tag (string or int, optional) –

• met (code added to qflag-column when flag-criteria are) –

flag_D02(tag)

In situ air temperature below threshold: Flags when ancillary in situ air temperature is below threshold

Parameters:

• tag (string or int, optional) –

• met (code added to qflag-column when flag-criteria are) –

flag_D03(tag)

GLDAS soil temperature below threshold: Flags when ancillary GLDAS NOAA soil temperature is below threshold

Parameters:

• tag (string or int, optional) –

• met (code added to qflag-column when flag-criteria are) –

flag_D04(tag)

Soil moisture rise without precipitation event (in situ): Flags when soil moisture increased both during the last hour and during the preceding 24h (increase is larger than 2x std-dev during this period), yet ancillary in situ data shows there was no precipitation event greater or equal to the minimum precipitation (depending on sensor depth).

At ISMN this flag is only applied to surface soil moisture sensors (<= 10cm sensor depth)

Parameters:

• tag (string or int, optional) –

• met (code added to qflag-column when flag-criteria are) –

flag_D05(tag)

Soil moisture rise without precipitation event (Gldas precipitation): Flags when soil moisture increased both during the last hour and during the preceding 24h (increase is larger than 2x std-dev during this period), yet ancillary GLDAS data shows there was no precipitation event greater or equal to the minimum precipitation (depending on sensor depth).

At ISMN this flag is only applied to surface soil moisture sensors (<= 10cm sensor depth)

Parameters:

• tag (string or int, optional) –

• met (code added to qflag-column when flag-criteria are) –

flag_D06(tag)

Soil moisture spike: Flags when soil moisture shows a positive or negative spike. See Eq [4,5,6] in Dorigo et al. (2013), Global Automated Quality Control of In Situ Soil Moisture Data from the International Soil Moisture Network,VZJ.

Parameters:

• tag (string or int, optional) –

• met (code added to qflag-column when flag-criteria are) –

flag_D07(tag)

!Includes jumps (D08)! Soil moisture drop (D07) or jump (D08): Flags when time-series shows a break based on relative (and absolute) change in soil moisture, a comparison of the first derivatives to the average of first derivatives centered at t and a large negative (positive) second derivative at t followed by a large postive (negative) value at t+1. The resective observations are then flagged as drop “D07” (or jump “D08”) when the 1. derivative at t is negative (positive).

See Eq [7,8,9] in Dorigo et al. (2013), Global Automated Quality Control of In Situ Soil Moisture Data from the International Soil Moisture Network,VZJ.

Includes an alternative drop type, which was not included in VJZ paper: drop from above 0.05m³/m³ to zero

Parameters:

• tag (string or int, optional) –

• met (code added to qflag-column when flag-criteria are) –

flag_D08()

!Included in flag_D07!

flag_D09(tag)

Low constant values: Flags where a previous soil moisture break (D07) and a period of low relative variance (variance/mean < 0.001 m³m⁻³) coincide, soil moisture observations are flagged as “D09” as long as the relative variance remains below the treshold. The defined minimum duration of a low plateau is 13h.

See Eq [14] in Dorigo et al. (2013), Global Automated Quality Control of In Situ Soil Moisture Data from the International Soil Moisture Network,VZJ.

Parameters:

• tag (string or int, optional) –

• met (code added to qflag-column when flag-criteria are) –

flag_D10(tag)

Invariant high soil moisture values: Flags where the variance of soil moisture values within 12h is below 0.05 -> period of low variance (plv) with a min_len of 12h. The plv requires a rise in the first derivative of at least 0.25 in beginning of the plv +/- 12h and a drop in the first derivative lower or equal to 0 at the end of the plv +/- 12h a mean of the soil moisture values between the rise and drop (or if they occur beyond plv scope, beginning and/or respective end of plv) of above 0.95% of the previous highest soil moisture value ever detected (highest_sm).

See Eq [10,11,12,13] in Dorigo et al. (2013), Global Automated Quality Control of In Situ Soil Moisture Data from the International Soil Moisture Network,VZJ.

Parameters:

• tag (string or int, optional) –

• met (code added to qflag-column when flag-criteria are) –

flag_G(tag)

Applies tag for all unflagged observations

Parameters:

• tag (string or int, optional) –

• met (code added to qflag-column when flag-criteria are) –

get_flag_description() → None

Prints out table with flag codes and a short description.

get_variable_from_data() → str

Gets first occuring and known Variable from the pandas dataframe Returns v:string ——-

run(name=None, sat_point=None, depth_from=None, flag_numbers=False) → DataFrame

Applies all quality control algorithms when keyword name is not set. However for flag C03 a threshold value (saturation point: highest soil moisture value depending on soil properties) is needed as input.

Parameters:

• name (list) – provide list of flags to only apply these flags

• sat_point (float) – Saturation Point in % vol for soil at the respective location. At ISMN the saturation point is calculated from Harmonized World Soil Database (HWSD) sand, clay and organic content for each station using Equations [2,3,5] from Saxton & Rawls (2006). (Saxton, K. E., & Rawls, W. J. (2006). Soil water characteristic estimates by texture and organic matter for hydrologic solutions. Soil science society of America Journal, 70(5), 1569-1578.)

• depth_from (Decimal) – Used to calculate minimum precipitation necessary to consitute a rain event for flags D04 and D05. Also used to skip sensor depths >=10cm for D04 and D05 (applied to surface soil moisture sensors only).

• flag_numbers (bool) – if true flag numbers are used as tags in the qflag column instead of flag ids (e.g.: ‘1’ instead of ‘C01’, ‘14’ instead of ‘G’)

Returns:

DataFrame including ISMN quality flags in column “qflag”.

Return type:

pandas.DataFrame

exception flagit.flagit.VariableNotKnown

Bases: Exception

flagit.settings module

class flagit.settings.Variables

Bases: object

Class for flagging thresholds of soil moisture, accompanying ancillary observations and

ancillary_p_min = 0.2

ancillary_ta_lower = 0

ancillary_ts_lower = 0

hi_boundary(var)

Upper threshold for quality control units: soil moisture : m³/m³ soil temperature, air temperture, surface temperature : degree Celsius precipitation, snow water equivalent, snow depth : mm soil suction : kPa

Parameters:

var : string variable name (some examples are: soil_moisture, soil_temperature, snow_water_equivalent)

low_boundary(var)

Lower threshold for quality control units: soil moisture : m³/m³ soil temperature, air temperture, surface temperature : degree Celsius precipitation, snow water equivalent, snow depth : mm soil suction : kPa

Parameters:

var : string variable name (some examples are: soil_moisture, soil_temperature, snow_water_equivalent)

plateau_count = 0

variable_list = ['soil_moisture', 'soil_temperature', 'air_temperature', 'precipitation', 'surface_temperature', 'soil_suction', 'snow_water_equivalent', 'snow_depth']

Module contents