Problem Solving with InfoNet 3: Importing and exporting data


Courtesy of Innovyze

Untitled Document

The problem

Water companies typically keep their network asset data either in a GIS or in a number of different files of different formats and ownership; some companies use a combination of the two. The management of such data repositories is often an ongoing headache for water engineers, especially at the time of data entry – import into the database– and at the point of using the data – exporting the data from the database. These problems usually stem from the following issues:

The format of the data: almost certainly the data will have been collected in a variety of different formats, by different contractors and at different times. Its input can necessitate time-consuming manipulations. Equally, exporting data in the right format is often difficult and requires further processes of conversion.

Selecting subsets of data: usually water engineers working on networks want to deal with subsets of data – a part of the network. However, because there is no concept of connectivity in a GIS or most data files, such selections can be made only by guesswork, looking at visualizations of groups of pipes that, because they are clustered together, probably constitute a coherent sub-network. But there is no way to validate these guesses, even when the data is exported and in use in an analysis or a model.

Applying corrections to spatial data: if the location of an asset such as a manhole or pump is corrected with more accurate data, the location of all connected pipes must be corrected separately, once all these connected assets have been identified. This problem too results from the lack of connectivity logic in a GIS and most other files storing asset data.

Over-writing old data with new: some data items, such as the location of a Pressure Reducing Valve, have only one correct value. But others have values that change over time – depth of silting for example – and the trends on this data are of value. However, most systems allow only one instance of each data item, so old data is overwritten by new, and trends can only be tracked if the old data is taken from the database and stored in a separate file.

Another problem of overwriting can occur if data being entered into the database includes a new asset which has, by mistake, the same identifier as an existing asset. In this case, although they are in fact different assets, the new data will overwrite the old, and a whole asset record will be lost.

Interfacing to other data systems: no repository, however comprehensive, will contain all the data a water engineer requires. This raises issues of interfacing to other systems, and knowing the different formats in the different systems. The problems are twofold – the practical issues of getting access to the remote files, from both a communications and a format viewpoint, and the knowledge issues of the structure of those files

Size limitations: water data sets can be very large, with increasing use of graphics and the requirement to store CCTV clips. Many general purpose products run out of space or start to perform very slowly when faced with the volumes inherent in networks containing hundreds of thousands of manholes.

The use of InfoNet

Because InfoNet is purpose-built for water companies and for network assets, it has multiple advantages over other systems, and can address the problems identified above.

The format of the data: InfoNet, just like the InfoWorks suite, uses its Open Data Import Center and Open Data Export Center to manage the import and export of data. These Data Centers can handle many different file formats, including:

  • MapInfo (MIF) files
  • ArcView (shape) files
  • AutoCAD files , both dwg and dxf (dwg files are also supported by a Direct Importer)
  • GeoDatabase files
  • Comma separated variable (.csv) format data
  • Tab separated data
  • Jet/Access database.
The “data” referred to above need not always be simply numbers. Text and images can also be stored within the InfoNet database, including both Digital Terrain Models and CCTV images in their various formats.

InfoNet also retains each Configuration Mapping, storing the mapping of a particular file format to the InfoNet format so that, once a specific mapping has been defined, subsequent files in the same format can be imported or exported without further mapping work

Selecting subsets of data: InfoNet contains linking logic on all its assets, so it is possible to define a complete subnetwork with no danger of omissions. The data from that subset can then be exported for modeling, or can be analyzed within InfoNet.

The other function that connectivity brings is the ability to trace a flow through a series of pipes, following not just connectivity but the actual path the flow follows, another very valuable feature for the examination of subsections of the network.

Applying corrections to spatial data: using the knowledge of the connectivity of all assets, correction of the location of one asset is used to automatically correct all related assets. For example, if a GPS survey finds that the location of a specific manhole needs to be corrected, that same correction is applied to all other connected assets – the connections on the upstream and downstream pipes for example – that logically must also be corrected. When a GPS survey is making numerous corrections to old and inaccurate location data this is a major time saver and avoids the inevitable errors inherent in checking manually for all the additional corrections required.

Overwriting old data with new: a major difference between InfoNet and other repositories is that InfoNet specifically caters for multiple instances of the same data. There is no effective limit to the number of surveys of the same group of assets that can be stored in InfoNet, each stored and available for future analysis. Also, InfoNet does not assume that “latest is best” – any dataset can be flagged as the preferred version.

One of the features of InfoNet is that it maintains an up-to-date network model for viewing at any time, based on the best version of all data. If new data is considered to be less reliable than existing data, this can be flagged and the network model takes account of this preference.

A second problem of overwriting is using the same asset name for two different assets, which most systems cannot address. InfoNet is smart enough to identify that there is probable duplication, and the new asset can automatically be given a new and unique name. InfoNet has a number of functions to help the management of the naming of assets, using any naming scheme specified by the user company.

Interfacing to other data systems: when data is not stored in InfoNet (some data, such as customer service data is best owned and managed outside the engineering and operations departments), there is the option to set up an online link from InfoNet to the other system. When an InfoNet user requests that data, InfoNet passes on that request to the other system and takes the current values of the specified data item. Each of these external calls is transparent to the user, who therefore needs no knowledge of the formats of the remote files – InfoNet manages that interface.

Size limitations: InfoNet is designed and built for the large datasets that hydraulic networks generate, and is not limited by using third party software at critical points. One client using an InfoNet database is now storing data on a 900,000 manhole network, including over 40,000 CCTV clips, and is performing well in terms of response times. With the capacity for managing terabytes of data, size limitations are a long way off for most users.

Customer comments

No comments were found for Problem Solving with InfoNet 3: Importing and exporting data. Be the first to comment!