One of the problems facing a real-time oceanographic observatory is the ability to provide a fast and accurate assessment of the data quality. The VENUS Coastal Network is in the process of implementing measures of real-time quality control on incoming scalar data that meet the guidelines of the Quality Assurance of Real Time Oceanographic Data (QARTOD) group. QARTOD is a US organization tasked with identifying issues involved with incoming real-time data from the U.S Integrated Ocean Observing System (IOOS). A large portion of their agenda is to create guidelines for how the quality of real-time data is to be determined and reported to the scientific community. VENUS is striving to adhere to the QARTOD’s ‘Seven Laws of Data Management’ to provide trusted data to the scientific community.
QARTOD’s Seven Laws of Data Management:
- Every real-time observation distributed to the ocean community must be accompanied by a quality descriptor.
- All observations should be subject to some level of automated real-time quality test.
- Quality flags and quality test descriptions must be sufficiently described in the accompanying metadata.
- Observers should independently verify or calibrate a sensor before deployment.
- Observers should describe their method / calibration in the real-time metadata.
- Observers should quantify the level of calibration accuracy and the associated expected error bounds.
- Manual checks on the automated procedures, the real-time data collected and the status of the observing system must be provided by the observer on a timescale appropriate to ensure the integrity of the observing system.
Real-time data quality testing at the VENUS Coastal Network includes tests designed to catch instrument failures and major spikes or data dropouts before the data is made available to the user. Real-time quality tests include meeting instrument manufacturer’s standards and overall observatory/site ranges determined from previous year’s data. Due to the positioning of some VENUS instrument platforms in highly productive areas, we have also designed dual-sensor tests to catch conductivity cell plugs which cause a sudden drop in conductivity.
The quality control testing at VENUS is split into 3 separate categories. The first category is in real-time and tests the data before the data are parsed into the DMAS database. The second category is delayed-mode testing where archived data are subject to testing after a certain period of time. The third category is manual quality control by a VENUS data expert.
Quality Control Flags:
VENUS has adopted the ARGO quality control flags. These flag and descriptions are as follows:
| ARGO Data Quality Flag | Description |
|---|---|
| 0 | No quality control on data |
| 1 | Data passed all tests |
| 2 | Data probably good. |
| 3 | Data probably bad. Failed minor tests. |
| 4 | Data bad. Failed major tests. |
| 7 | Averaged Value |
| 8 | Interpolated Value |
| 9 | Missing data |
VENUS Terminology
Q. How does VENUS determine the final quality control flag?
All VENUS data are passed through each level of testing to create a quality control vector containing the output for each test. The overall output quality control flag is determined from the set of QC flags for each datum as follows:
- If passed all tests, the final output flag assigned is 1 (Data passed all tests).
- If passed major tests but failed minor tests, the final output flag assigned is 3 (Data probably bad. Failed minor tests.)
- If failed major tests, the the final flag is 4 (Data bad. Failed major tests.)
Q. How do you determine which tests have been applied to the data you downloaded?
In the accompanying metadata, there is a section called Data Quality Information that contains all the information regarding quality control for the requested data. Quality control test information is based on device and is listed, if available, along with the valid time period of the test as well as the values used in the formula. Also listed in this section are time gaps greater than 15 minutes in duration.
