Using The Indico Validation Module Brochure

Volume 5 Airviro User’s Reference Using the Indico Validation Module How to validate measured data Using the Indico Validation Module How to validate measured data Amendments Version Date changed Cause of change Signature 3.11 January 2009 New Module GS 3.12/3.13 January 2009 Upgrade GS 3.20 June 2010 Upgrade GS Contents 5.1 INTRODUCTION.............................................................................................. 5 5.2 GETTING STARTED ....................................................................................... 7 5.3 Validating TIME SERIES DATA...................................................................... 9 5.3.1 Domain .....................................................................................................................9 5.3.2 Selecting time series.................................................................................................9 5.3.3 Period .....................................................................................................................11 5.3.4 Macro Load.............................................................................................................12 5.3.5 Validate...................................................................................................................13 5.3.6 Ratify ......................................................................................................................20 5.3.7 Scaling....................................................................................................................21 5.4 OUTPUT ........................................................................................................ 23 5.5 DELETE ......................................................................................................... 24 Using the Indico Validationt Swedish Meteorological and Hydrological Institute, SE-601 76 Norrköping, Sweden Page 5 (24) Phone: +46-11 495 8000, Fax: +46-11 495 8001 5.1 INTRODUCTION Indico Validation is a JAVA-based user interface to the time series database at the Airviro server. Indico Validation client can be executed from a PC or other device running Internet Explorer 6, Netscape Navigator 7 (or higher), or Mozilla based web browser (Firefox). Please note that the Indico Validation client does not require any installation on the PC side. Anyone who knows the URL, knows user id and password and has sufficient privileges to the Airviro server can access the databases. Indico Validation is - together with Indico Admin, Indico Report and Indico Presentation - part of the integrated Indico package intended for acquiring, storing, editing, presenting, analyzing, validating, reporting and exporting time series data Indico Validation is used to validate the data. Data is usually collected through a modem. The Airviro computer first runs an protocol (data collection program), which contacts a remote station. The station answers and waits for instructions. The protocol then sends instructions to the remote station requesting for the latest data. The station responds to the instructions, the computer receives the new data and ends the communication. The new data is then loaded into the time series database so that other programs (such as the Indico Presentation module and Indico Validation) can access it. Automatic data collection (Indico Administration module) is carried out via protocol programs. Different loggers require different protocols. Therefore, it is not within the scope of this manual to explain the configuration of data collection. See the Indico Admin manual. With Indico Validation you can view or edit the contents of the time series database. If raw (not scaled) data has been stored in the time series database, you can configure scaling of them and then have both raw and scaled data in the database. First you must select the data you want to edit either by clicking on the Time Series menu or by loading an existing Indico Presentation macro by clicking on the Macro Load menu. What is Indico Validation? Indico Validation is a powerful tool for presenting, editing and checking data in the time series database. With Indico Validation, you can • View data and its status in a table and in a graph at the same time. • Indicates statuses with colour coding. • Edit and mark as invalid individual data as well as blocks of data while keeping the original raw value. Airviro 3.20 Page 6(24) June 2010 • Scale data with zero and span point from calibrations. • Scale data with data from multi point calibrations. • Save validated data in a new time series. • Delete block of data from the time series database. Using the Indico Validation Swedish Meteorological and Hydrological Institute, SE-601 76 Norrköping, Sweden Page 7 (24) Phone: +46-11-495 8000, Fax: +46-11 495 8001 5.2 GETTING STARTED Indico Validation is used to access and validate data stored in the Time Series Database. Once Airviro has been installed on the Server, you can begin using it by typing the correct URL in your web browser over Intranet/Internet. After logging in the application by entering a username and password (Figure 5.2.1), the user is presented with a list of all available Domains (if none has being previously selected and stored), By selecting a Domain the top menu list will be refreshed to show all available Airviro Modules. Figure 5.2.1. Getting started. By clicking on the Indico Validation menu, the main Indico Validation page will be shown. The main page in Indico Validation contains the following: Airviro 3.20 DOMAIN Lets you select the domain and time resolution, for example: REF and 1 hour data. TIME SERIES Provides an interface to the Time Series Database, where you can select time series (the stations, parameters and instances). PERIOD Select time period for the validation session. MACRO LOAD Load the time series from an existing Indico Presentation macro. These are created using the Indico Presentation module. VALIDATE Starts a validation session. RATIFY Starts a ratification session. Data can be stored as a new time series. OUTPUT Displays the selected time series in a new window as text. DELETE Allows the user to delete a selected time series. Page 8(24) June 2010 Using the Indico Validation Swedish Meteorological and Hydrological Institute, SE-601 76 Norrköping, Sweden Page 9 (24) Phone: +46-11-495 8000, Fax: +46-11 495 8001 5.3 VALIDATING TIME SERIES DATA The time series database for a certain domain may contain a large number of monitoring stations and parameters. The parameters can be related to mass concentrations of pollutants or meteorological data, traffic intensities, instrument readings of other kinds or quality control data from data loggers. For each parameter there is also a quality flag (status). Data can also arrive once per year from some other sources to be imported into the time series database with Indico Administration or Waved. Time series data can also be generated by the postprocessor in Dispersion or by statistical forecasting in Aircast or by a meteorological agency. Normaly Indico Validation is used by working through all menus from Domain & Time resolution down to Time, excluding Macros, which is a shortcut to load time series specified in the macro. 5.3.1 Domain Here you can select domain and the time resolution. If you change the domain, you may get a different set of time resolutions, times series and macros related to the selected domain (Figure 5.3.1.) Figure 5.3.1. Domain and Time Resolution. It is possible to work with several instances of Indico Validation simultaneously. Multiple users can work at the same time without any substantial risk for interference. 5.3.2 Selecting time series Airviro 3.20 Page 10(24) June 2010 In Time Series, you will see a list with all stations - active or inactive - and all observed parameters in the parameter database, regardless the station. When you select a station in the station list, you will get a list of parameters belonging to that station. Clicking Re-scan will update the lists from the time series database to include any new time series that have been added since the session started. Clicking Clear removes the current selections and fills the lists with all stations and parameters. On the other hand, if you are interested in all stations that measure a certain parameter, start by clicking on the parameter in the parameter list (Figure 5.3.2.) It is possible to sort the stations or the parameters in the list box alphabetically, by station key or parameter key by selecting sort key in the associated drop-down list box. Checking/unchecking Reverse re-sort the list in the order accordingly. It is also possible to promote active stations by moving them to the top of the list by checking Active first. Sorting stations also by reverse death time creates a list of increasingly older stations. Click Clear to get a full list of stations. Operational stations are shown preceded by an asterisk (*), these stations collect data automatically (Figure 5.3.2.) After you have selected both a station and a parameter, you will get the instances list for the actual parameter. The instance is used to differentiate among multiple measurements of the same parameter at the same site. There are three different kind of instances. The letter in brackets after the three letter instance specifies the type. The type [v] means a raw value that can be scaled in the validation process. Normally time series values of this type are read only and can not be changed. The type [M] means data that are editable. These can origin from raw data (scaled or copied from raw data) or be loaded directly into the time series database. The type [O] indicates an editable value with an additional standard deviation from the integration period as well as a light intensity. All types also has a status flag assigned by the quality control in Indico Administration. (Figure 5.3.3.) Once you have selected a station, parameter, instance and attribute, the time series is uniquely identified (for the current time resolution). Click New to select the time series for further processing. Please remember that the variables are numbered according to the order in which they are listed in the “Selected” list box. You can remove a highlighted selected variable from the list box with the Remove selected button or replace it with another selection and clicking Replace. If you click Clear all, all variables will be removed in the “Selected” list box. Here, the same restrictions for selecting the time series as in the Indico Presentation are used. The user can select up to 8 time series. When you are satisfied with your time series selections, click Apply to save your settings. (Figure 5.3.2.) Using the Indico Validation Figure 5.3.2. Time Series Windows. Figure 5.3.3.Example of selected time series. 5.3.3 Period The last week is selected by default but you can select any time period. Date and time can be presented in European, UK or US date format. If you want to set another starting date, you can write the date in the From box. Alternatively, you can use the double arrows between the From and To boxes to transfer a date between the boxes. You can also use the double arrows adjacent Swedish Meteorological and Hydrological Institute, SE-601 76 Norrköping, Sweden Page 11 (24) Phone: +46-11-495 8000, Fax: +46-11 495 8001 Airviro 3.20 to Year, Month, Week, Day or Hour to step forwards or backwards. You can always reset the To box to present time by clicking Present. When you have selected a period, click Apply. (Figure 5.3.4.) Figure 5.3.4. Period. The hour starting at 00:00 and ending at 01:00 is labelled 01. The hour starting 23:00 and ending 00:00 is labelled 00 the next day. The date and time given is exclusive in the From box and inclusive in the To box. 5.3.4 Macro Load Under Macro Load you can select the macro previously created using the Indico Presentation module. Figure 5.3.5. Figure 5.3.5. Macro. Page 12(24) June 2010 Using the Indico Validation Swedish Meteorological and Hydrological Institute, SE-601 76 Norrköping, Sweden Page 13 (24) Phone: +46-11-495 8000, Fax: +46-11 495 8001 5.3.5 Validate In this page the user can view or edit the Time Series Data for the selected domain. The user must select time series before starting a validation session. • Session A session consists of 1 to 8 time series of data for a limited time period. If there are Scalings available for a selected time period, the first active scaling before and after the selected period will be included. Both raw and scaled data will be loaded for each time series if available. Raw data is never modified in any way, but scaled data may be edited or changed by adding, changing or deleting scaling. If no raw data is available, no scaling can be applied and it is only possible to edit data. No changes are actually saved to the database until the Apply button is pressed. (Figure 5.3.6.) The data is normally captured from the data logger configured using Indico Administration module. Collected data is stored as raw data if this status flag is assigned to the parameter in the protocols settings within the Indico Administration • Interface Spreadsheet Spreadsheet contains an active time window, which usually is a subset of the whole session. The active window may also contain one value that has keyboard focus (marked with a black square). Graph If no time series in the spreadsheet has focus, all time series in the session will be shown. Graphs will be colour coded per time series in the same way as the time series labels in the spreadsheet. If one time series has focus, only that time series will be shown. Graph will be colour coded by value status in the same way as the data in the spreadsheet. The active window is marked with a light grey rectangle in the graph. Any Scalings are marked with triangles at the top of the graph area. Active Scalings have a dark triangle and inactive ones have a gray triangle. If a scaling is selected in the Scalings list, the corresponding marker in the graph will be slightly lower than the others. When a shorter time period is shown, each value is marked with a diamond. This disappears when there are too many values to be shown. Airviro 3.20 Page 14(24) June 2010 Scaling If there are raw data available, the Scalings interface will be shown to the right of the graph. Here it is possible to add, edit and delete Scalings. Axis If auto scaling of the x-axis is enabled, the application will try to find the best possible scaling. It is also possible to enter manual limits to the x-axis. To enter manually the Y-axis scale, just type the min and max value for it in the test boxes and press enter after that. Fonts & Columns The size of the text in the spreadsheet and the number of columns can be selected. These settings will be saved in a cookie between sessions. • Mouse Clicking on a spreadsheet value will set focus to that value. Clicking on the time series labels to the left will remove focus from an individual value. Clicking on the graph will set the focus on the nearest value and move the active window so that it is centred on the value. Pressing and holding down the left mouse button will allow interactive dragging of the active window. • Keys Page Up Scroll active window back one window size. Page Down Scroll active window forward one window size. Arrow Up Move focus to cell above (if possible). Arrow Down Move focus to cell below (if possible). Arrow Left Move focus to cell to the left (if possible). Arrow Right Move focus to cell to the right (if possible). Home Move active window to beginning of session. End Move active window to beginning of session. Esc If editing a value, this will abort the editing without changes; otherwise it will remove focus from the spreadsheet. Enter If editing a value, this will confirm the changes you have made and return to spread sheet. SHIFT+ALT+Arrow Up Copy value to cell above and move focus. Using the Indico Validation Swedish Meteorological and Hydrological Institute, SE-601 76 Norrköping, Sweden Page 15 (24) Phone: +46-11-495 8000, Fax: +46-11 495 8001 SHIFT+ALT+Arrow Down Copy value to cell above and move focus. SHIFT+ALT+Arrow Left Copy value to cell above and move focus. SHIFT+ALT+Arrow Right Copy value to cell above and move focus. d Delete current value (status 2). D Delete a number of values. Time period entered in dialog. u Undelete current value (change status 2 to 15). U Undelete a number of values. Time period entered in dialog. e Edit current value. f Apply ramp on a series of values. Time period, offsets and multiplication factor entered in dialog. The offset is applied before the multiplication factor. r Restore current value to the original value at the start of this session. R Restore a number of values. Time period entered in dialog. i Recover current value from raw value with applied scaling. Overrides status 2 and 15. I Recovers a number of values. Time period entered in dialog. s Search for values of a certain status and set the value to a specified value. Time period, status and new value entered in dialog. Note: Applies to all loaded time series, not only the active time series. t Apply a lower threshold on a series of values. Time period and the lower threshold entered in dialog. z Toggle graph between zoomed mode and whole mode. Only the data in the active window will be shown when in zoomedAirviro 3.20 mode. • Colour coding by status The following colour codes are used for statuses: 1 Blue Unchecked (should not appear) 2 Dark red Manually marked invalid. 3 Red Error from logger. 4 Red Value below configured minimum. 5 Red Value above configured maximum. 6 Red Exceeded maximum configured gradient. 7 Red Variations less than configured minimum. 8 Red Too large standard deviation (OPSIS only) 9 Red < not used > 10 Orange Logger warning. 11 Orange < not used > 12 Orange < not used > 13 Orange < not used > 14 Blue Value checked ok. 15 Green Manually changed. Figure 5.3.6.Example of the Validation. With Settings you can control some features: • Show raw values in graph Page 16(24) June 2010 Using the Indico Validation Swedish Meteorological and Hydrological Institute, SE-601 76 Norrköping, Sweden Page 17 (24) Phone: +46-11-495 8000, Fax: +46-11 495 8001 • Show supporting lines from y –axis in graph • Immediate auto-rescaling of y-axis • Auto minimum on y-axis • Show focus value into window With Apply & load prev/next you can step forwards or backwards the selected period. • Scalings In the Scaling window, the user can define new scalings for the selected stations. After having selected a station from the Station list, the user must define a Date/time. Also must choose a Type (linear, raw, scaled, linear2) for the scaling and select the time Resolution for the time series (1h, 30 min, 5 min, etc). A Comment can also be entered to identity the new scaling instance. Finally, using the button Change parameters, the parameters associated to the chosen Station, must be selected and the values obtained during the calibration of the actual analysers. (Figure 5.3.7.) To add a new Scaling, click on Add and then enter Time, Usage, Type and their characteristics. In the usage combo box different options are available (inactive, used or discarded.) In the option type, three different options can be selected (raw, scaled or linear) and for each of them their characteristics must be set up according to this detail: - Raw: Cylinder concentration, user span and user zero must be entered. - Scaled: Auto span, auto zero, user span and user zero must be entered. - Linear: Line slope and line offset must be entered. Airviro 3.20 Figure 5.3.7.Example of the Scalings. When the user defines a scaling for a TSDB, for the first time in the Validate… Scaling sub window (Scaling frame), it is necessary to click Apply to visualize the new values in the graph and tables. The values that appear on the graph with status 1-Blue-Unchecked should not appear. The Scalings are defined in Scaling subwindow if it is available. The characteristics of the validated data are: - The validated data have as instance the letter M - The raw data is zero if it is not associated to a scaling - The raw or validated data can be visualized in the Output menu. When a scaling is defined for the time series, using the Scaling windows, this scaling will be applied from the defined period until finding another scaling definition. • Scaling formula – Raw In this case the values received from the logger/instrument are unscaled, e.g. mV values. Page 18(24) June 2010 Using the Indico Validation Where: Vscaled The scaled value calculated in Airviro. Vraw The value received from the logger/instrument. Zu The zero point entered. Su The span point entered. CC Span gas cylinder concentration. • Scaling formula – Scaled (by calibration) In this case the values received from the logger/instrument are scaled using the latest valid calibration. Where: Vscaled The scaled value calculated in Airviro. Vraw The value received from the logger/instrument. Zm The zero point measured in calibration. Sm The span point measured in calibration. Zu The zero point entered. Su The span point entered. • Scaling formula – Linear In this case the values received from the logger/instrument are scaled using a linear transformation. Useful for entering multipoint calibrations. Where: Vscaled The scaled value calculated in Airviro. Vraw The value received from the logger/instrument. Lm Offset. Lk Slope. • Scaling formula – Linear 2 This is just another variant of the Linear transformation above. Swedish Meteorological and Hydrological Institute, SE-601 76 Norrköping, Sweden Page 19 (24) Phone: +46-11-495 8000, Fax: +46-11 495 8001 Airviro 3.20 Vscaled = (Vraw – Lm) · Lk Where: Vscaled The scaled value calculated in Airviro. Vraw The value received from the logger/instrument. Lm Offset. Lk Slope. 5.3.6 Ratify In this window the user can ratify data, i.e. making some fine tuning of the data and then save the data to a new instance. The "Ratify" function is more or less the same as "Validate". The main difference is that when you press "Apply", you can enter a different instance to save the data in. This means you can save it as ratified data under another instance, e.g. named RAT. When you have finished ratifying, click on Apply to save your data. See Figure 5.3.8. Page 20(24) June 2010 Using the Indico Validation Swedish Meteorological and Hydrological Institute, SE-601 76 Norrköping, Sweden Page 21 (24) Phone: +46-11-495 8000, Fax: +46-11 495 8001 Figure 5.3.8.Ratify. 5.3.7 Scaling This is an alternative way to enter scalings. Here all scalings for a certain time, station and type of scaling are grouped together and it is possible to add, edit and delete scalings for the station. The same scaling parameters can be entered here as in Validation with some additions: A comment can be entered for the whole group of scalings (scalings with the same time stamp, type of scaling and station). A list of stations are shown in the leftmost list. The actives stations are visualized by checking the Active check box. (Figure 5.3.9.) If the system has a large number of stations, two available filters (Protocol and Group) can be used to reduce the number of Stations that are displayed by default in the Station list. Select the station for which you will define a scaling. That will show all the available scalings for the selected station. Click on one of the scalings to list all the characteristics that have already been configured for that scaling type, then you can choose one for editing. Click on Add to add a new scaling. Date/time is the time from which this scaling will be applied. Type is the type of scaling. Resolution specifies the time resolution that the scalings will be applied to. Different scaling parameters can be entered depending on the scaling type that has been chosen. The first columns shows parameter and instance. The parameters included in the scalings can be changed using the Change parameters… button. In raw case (see formula, 8.3.5.6.1.) the values received from the logger/instrument are un-scaled, the user must be enter the: - The zero point entered ( Zu ) - The span point entered (Su) - Span gas cylinder concentration (CC) - Status: inactive, used and discarded Note: the columns Z and S are calculated. Z is Zu and S is (CC / Su - Zu) * 1000 Airviro 3.20 Figure 5.3.9.Scaling. In this scaled case (see formula, 8.3.5.6.2) the values received from the logger/instrument are scaled using the latest valid calibration, the user must input the following data: - zero point measured in calibration (Zm) - span point measured in calibration (Sm) - zero point entered (Zu) - span point entered (Su) In this linear case (see formula, 8.3.5.6.3) the values received from the logger/instrument are scaled using a linear transformation, the user must input: - offset (Lm) - slope (Lk) Click on Apply to save your changes. (Figure 5.3.9.) Page 22(24) June 2010 Using the Indico Validation Swedish Meteorological and Hydrological Institute, SE-601 76 Norrköping, Sweden Page 23 (24) Phone: +46-11-495 8000, Fax: +46-11 495 8001 5.4 OUTPUT Using Output time series, data are shown in a table with fixed headers. The Header includes the name of the station, the parameter, instance and attribute. The following is a typical table for the station Validation: Figure 5.4.1. Output. Airviro 3.20 Page 24(24) June 2010 5.5 DELETE Using Delete, the user can delete whole time series. For each station using the Update button the user can select, using the available filters, which time series he wants to delete. Figure 5.5.1. It is important to remember that after the delete button is pressed, there is no way back for a ordinary user to recover the deleted data. However, the system administrator can recover the data. Figure 5.5.1. Delete.