Mastering the “Black Art” of Manipulating Modeling Parameters

- By:

Courtesy of Courtesy of Innovyze

Matt White, Senior Wastewater and Drainage Modeler at Earth Tech advises fellow modelers to invest time in calibrating their models. 

Calibration is the most important phase of any sewer flow modeling exercise, but is often neglected, says Earth Tech’s Matt White. Poorly calibrated models can have a significant impact on the costs of the modeling process, but just as important, the accuracy of the model as a predictive and estimation tool, he stresses. “Poor calibration leads to poor results. Models are really only as accurate as the data that is applied and ability of the modelers who build them.”

Poor results might lead to incorrect sizing of solutions, with storage oversized or undersized, he warns. Similarly a problem might be shifted elsewhere by duplication or upsize instead of being solved. Big budgets associated with capital works programs are often involved “and we should be spending a bit more time making sure that the model instills confidence”. 

The process of calibrating a model is a “black art”, says White. He likens it to taking a lump of coal – “useful, practical, not particularly attractive” – and then turning it into a diamond “which is much more attractive, has different assets and is wanted by a lot more people”. Greater attention to calibration would improve modeling across the industry, he believes.

InfoWorks CS has many valuable features and capabilities that users may employ, says White. “InfoWorks CS is a great tool for hydraulic network modeling. It is very powerful, with a lot of useful features. Model calibration is a key, key thing - I can’t emphasize it enough.”  Water businesses must ensure that their specifications are thorough and that modelers are given adequate time to scope the work and do the job efficiently, he believes.

Traditional static modeling was quite generalized and would only give an approximate idea of the potential flooding, says White. Much more information is needed as input in the modern approach using InfoWorks CS. The modeling still requires catchments, data and inputs but many additional aspects may also need to be considered. These might include the run-off calculations, routing of the flow and antecedent conditions as well as all the ‘standard’ modeling parameters. Taking account of all the factors produces a much better indication of where the flooding might be as well as its magnitude, he stresses.

Calibration involves matching predictive model results with recorded field data based on sound engineering judgment. “That’s the key issue - sound engineering judgment,” says White. Any changes should be supported by field data, reports, operators’ comments, historical events or similar evidence. “Calibration is a difficult task,” he stresses. “It is generally unique to each catchment.  What works for one catchment doesn’t necessarily work for another.” In some cases, the calibration fits straightaway, but others might take a lot more time, effort and iterations - particularly for complex systems with numerous pump stations and ancillary controls.

Enlarge“But we don’t have an infinite pool of money or infinite amount of time to get these models calibrated,” acknowledges White. “It will always be a Time vs Quality debate. For example. there is no point spending a long time getting the lines to fit beautifully, only to find that the flow survey data has become out of date, budgets are blown and deadlines past”.

Avoiding the pitfalls

Reality checks also play an important role, believes White. The model’s general purpose is to replicate what is happening in the real system, he points out, so it is important to ask some broad questions about whether the results look acceptable and are as expected. Concern should be raised if there are anomalies that can’t be understood. 

There are a lot of pitfalls and shortcuts to be aware of, he says, including the force-fitting of data. This involves making arbitrary changes to a model to make it fit the observed data. It runs counter to sound engineering judgment, says White, and means that the causes of the discrepancies between observed and predicted are being ignored. “You’re not explaining why the model is behaving the way it is, and therefore you must ask how good it will be as a predictive tool,” he says. Force-fitting is usually done by changing key parameters, such as per capita discharge rates, land use parameters and dimensionless profiles – the diurnal profiles that are put into the model.  InfoWorks supports different profiles for trade, commercial and residential as appropriate, and therefore it is easy to ‘tailor’ model inputs to match outputs. This is where reality checks and common sense play an important role, says White. “Residential profiles should be generally uniform and similar across the catchment. In a single study area, you wouldn’t expect two profiles to be completely different without significant justification.”

It is also important to be aware of per capita discharge. These can vary across socio demographics but, as with profiles, would generally be of similar values. “You wouldn’t expect residents in a north part of your catchment to use 300 l/d and then - as I’ve seen in some examples of catchments - the population apparently using 140 l/d to the south.” Sound reality checks like correlating figures back to water consumption data are good practice.

Land use run-off parameters are important when calibrating response to storms. It is important to be aware of contributing areas and run-off coefficients, which can vary across study catchments.

Verification against historical data is also a key task. This can cover historic flooding records, operators’ reports and even anecdotal evidence from consumers/residents about prior problems. Models are often used to determine the size of a potential solution to solve an identified deficiency. It is important to be aware that solutions might be sized to a particular design standard, for instance a one in five year ARI event while the model has been calibrated to a smaller event.  Historical verification plays a key role in giving confidence about how the model will work under those larger, more extreme events.

The importance of good flow data

Due consideration must be given to the flow survey data, says White.  “It is often seen as irrefutable, as gold dust,” he says. A lot of money may be spent getting the data, but calibrating to poor data is just as bad as force-fitting the model. “You won’t have great  confidence in the results and the options you might be developing out of that model.”  He welcomes InfoWorks’ inclusion of scatter graphs as being much quicker than using spreadsheets to examine the data.

Looking back at the field recordings and viewing data over an entire period can also help. “It is very easy to look at one event in isolation, and not consider catchment response and behavior over the entire survey period.” 

Picking the best flow survey site is also imperative. “Authorities and councils should ideally get the modelers involved in the flow monitoring stage,” he advises. The ability to ensure that monitors are collecting data on aspects such as trade flows and historical flooding locations will increase the likelihood of the model performing as required. 

The chosen site should be hydraulically suitable for the particular type of study being undertaken. For instance, there might be a significant influence from trade. In one particular model, 60% of the dry weather flow was from trade waste produced by three major customers. It was essential to have confidence in the data from this area. “They made up such a big component that extra monitors were installed to pick them up,” says White.

Specifying the requirements

Specifications play a positive part in the production of well-calibrated models, says White, with major authorities such as Yarra Valley Water, Logan Water and Ipswich Water leading the way. “We don’t want the specification to write in everything – but they can control and give guidance on how things are done.” The specification gives scope for ensuring that appropriate attention is given to calibration and can also help with the scoping of works. “The consultant can give a much better idea of how much time things are going to take when they know that there are no blurred lines,” he says. The specification can be tailored to a particular authority or council’s needs or requirements.

It is essential to step back and check that the model does what is needed. “If the answer is no, then you’ve got to stop and reassess until you are happy with the model.”

Giving people a chance to check or audit their work is important, but it is not always allowed for. “It’s a standard engineering practice with design and construct jobs. Someone designing a bridge will give it to someone else to make sure that the calculations are right so that bridge doesn’t fall down,” says White. Similarly, capital programs should allow for some degree of checking or auditing. “This could be internal or external – and it is certainly easier if a specification exists. All in all, it increases confidence in the models that are provided.”

This is based on a presentation given by Earth Tech Senior Modeler and Project Engineer Matt White at the Wallingford Software InfoWorks CS User Conference in Melbourne, Australia in December 2006.

Customer comments

No comments were found for Mastering the “Black Art” of Manipulating Modeling Parameters. Be the first to comment!