Data Management and Analysis

An important element of IWRM modeling is management and processing of data. Data is the fundamental key to a successful representation of a system. Without proper data representation, the model is lost. I decided to do a quick search to see what people are using today. Doing simple Google searches on the various tools, I found HEC-DSSVue and Hydstra to be the most popular (assuming Internet search returns is a measure of popularity). Here are the search results:

HEC-DSSVue => 320 unique results
Hydstra => 390 unique results
Aquarius => 100 unique results

In looking at scholarly documents online, I was able to compare the 3 tools (shown above) by counting the different kinds of models these were used for. HEC-DSSVue seems to have been used most often for water resources and flood modeling, Aquarius for water resources and operations models, and Hydstra for water quality and runoff models. Table 1 summarizes the results from this comparison.

As part of my research, I posted a question on some LinkedIn groups to see what people had to say. You can see the comments people made by following the links to the posts I made on these groups in the links below. Note that you need to be a member of these groups to see the comments.

AWRA Group Comment
AWWA Group Comment
Assoc.Water and Enviro-Modeling Group Comment

sci.geo.hydrology Forum Comment

These are the tools I am at least somewhat familiar with:

HEC-DSSVue
This tool is a powerful data management system with over 60 math functions that can be performed on the data and many utility functions for dealing with the data, such as interpolation. The GUI provides a robust data viewing capability that can be customized to suit your needs. The obvious benefit of this tool is that it's extendable and free.

This tool is widely used in the government, academic, and consulting arenas since it has been around for a while and is free to use with decent software documentation.

1. Retrieving data
There are many ways to retrieve data for use in HEC-DSSVue. You can enter data manually, import text files, SHEF data, csv files, USGS data, NCDC data, CDEC data, DSSUTL format files, and even image files (i.e. gridded data). Another way is using a Java plug-in, which could be used to extend the software (usually by adding a menu button to the main screen of the software). The software also comes with an MS Excel add-in for retrieving and storing data directly from the spreadsheet file. An available plug-in also allows users to retrieve snotel data from the NRCS online database. Data files with sizes up to 8Gb can be stored in HEC-DSSVue.


2. Viewing data
Data can be viewed in tabular or graphical format. There is also an option to view data in groups as shown in this example, taken from the HEC website:

Plots can be customized in many ways, including 

3. Data types
The data can be represented in 4 different ways, which controls how interpolation is performed. These include:

• INST-CUM (instantaneous cumulative, i.e. precipitation mass curve) 
• INST-VAL (instantaneous value, i.e. river stages)
• PER-AVER (period average, i.e. monthly flow values)
• PER-CUM (period cumulative, i.e. incremental precipitation)

Interpolation between points in a dataset varies depending on the datatype chosen, which is demonstrated in the HEC-DSS user manual as follows:

4. Data analysis
There really is no limit to the math functions you can perform on datasets in HEC-DSSVue, but some functions may require a script to be written, which can be overwhelming to the novice user. Basic functions can be executed graphically, such as accumulation, absolute value, change units, and subtracting data.

A nice feature of HEC-DSSVue is the ability to perform hydrologic functions on the datasets. These functions include Muskingum Routing, Straddle Stagger Routing, Modified Puls Routing, Rating Table, Reverse Rating Table, Two Variable Rating Table, Decaying Basin Wetness, Shift Adjustment, Period Constants, Multiple Linear Regression, Apply Multiple Linear Regression, Conic Interpolation, Polynomial, Polynomial with Integral, and Flow Accumulator Gage Processor.

The following statistical functions can be invoked:  Basic (statistics), Linear Regression, Cyclic Analysis, Duration Analysis, and Frequency Plot.

Aquarius

Compared to HEC-DSS, Aquarius is said to be more visual and user-friendly. This is mostly attributed to the object-oriented framework in which the user can work with the data. This might be something to consider when comparing the tool to HEC-DSS since the latter is often less able to help the user visualize the data and the functions applied to them. Another major difference I can see is that Aquarius provides some functionality to streamline the acquisition of data from the field. I haven't seen enough of the tool to say whether any computational aspect presents a great benefit over HEC-DSSVue, but they seem to be comparable in this regard.

1. Retrieving data

There are many ways to retrieve data for use in Aquarius. One of the obvious benefits of Aquarius (compared to HEC-DSS is that it is very easy to acquire data from data loggers in the field. You can enter data manually via ad-hoc manual data entry of datasets such as portable field meter data, grab samples, or stage vs. discharge point pair data. You can also import the most common file format such as ASCII or text-based formats (e.g. comma separated values (.csv) files), Aquarius Optimized Packages (.aop), AquariusML (.xml) and HYDAT text files. It is also very easy to join chunks of data to the end of existing datasets but this is trivial in HEC-DSS as well.

Another important aspect of Aquarius is data acquisition from outside sources, including Distributed Control Systems (DCS), Supervisory Control and Data Acquisition (SCADA) systems, Programmable Logic Controllers (PLC), Batch execution systems, Lab Information Management Systems (LIMS), and Relational databases, and XML files.

2. Viewing data

Data can be viewed in tabular or graphical format. There is also an option to view data in groups as shown in this example, taken from the Aquarius website:

Also, here is a view of the main screen that allows data and functions to be dealt with in an object-oriented fashion.

3. Data types

The data can be represented in different ways, which controls how interpolation and accumulation are performed. This question has been asked of the company and I await their reply.

4. Data analysis

Functions similar to those described under HEC-DSS can also be performed in Aquarius. Functions and utilities also include data corrections, develop discharge curves, and statistics. It can help with paired data and performing lagged regressions, linear regression, autoregressive models, and artificial neural network. A special note needs to be made about why someone might choose Aquarius over the free HEC-DSSVue. After spending some time with support staff on the phone, I learned that even the USACE (who develops HEC-DSSVue) chooses to use Aquarius because of their robust data analysis features, such as producing rating curves for stream gages and modeling purposes. Apparently, they currently have about 200 customers.

Hydstra
Hydstra is similar to Aquarius with a couple of differences: integration with GIS and no object-oriented pallet for building the data model. Hydstra is developed by Kisters. Data can be viewed graphically and in tabular form. It appears that Hydstra is much more popular than Aquarius.

It appears that Hydstra has been used for more for real-time data management rather than dealing with larger, historic records that might be incorporated into IWRM models.

Interestingly, Hydstra datasets can be published online.

I sent the Kisters team a query into how Hydstra compares to Hec-DSSVue and got this reply:

Kisters have well over 500+ water agencies world wide using our water management  products. These range from national systems such as the UK Environmental Agency, South African Department of Water Affairs, Australian Bureau of Meteorology through to large state government water agencies and private companies such as Pacific Gas and Electric Company [PG&E California].

I would need to do a full cross check of the functionality offered by HEC-DSSVue and the various Kisters products to understand the exact differences. At face value however the Kisters products have far more functionality as our products offer extensive data management, reporting and analysis tools. Simple functions such as "batching", producing 'pdf" outputs etc are basic functionality within Kisters.

Kisters offer a "richer" suite of tools to undertake plots, data tabulation, editing, and manipulation.  


Other Tools

These are tools I am not familiar with but are possibilities:
InfoNet by MWHsoft
AllMax
Locus

Hybrid treatment plants: maintaining production

It is known that large and complex water treatment plants are most efficient when production levels are maintained at or near the design capacity over the long term. As production rates rise and fall dramatically, it is required to change operations and startup/shutdown major pieces of equipment, costing the plant more money to operate and maintain.

One of the major problems with treating stormwater is that the water rushes through during storm events then virtually disappears at other times, usually at times that are unpredictable. Regulating these flows using storage is most likely uneconomical because of the high cost to build huge storage facilities. Another problem with treating stormwater for reuse is that the water quality of the stormwater is possibly different than the traditional water supply and so a totally different treatment process might be necessary.

What if a treatment plant was able to quickly change the type of treatment process but maintain overall production level? Would this be economical? If this was found to be viable, then a treatment plant could divert it's traditional inflow stream and accept stormwater in it's place while the stormwater is available, then switch back after the storm is past.

Has this ever been done?

An article has recently been published describing how treatment plants in Singapore have been built that can switch from treating fresh water to saline water while maintaining production.  The abstract of the article says:

 When the river water has high salinity or dries up, treatment is maintained with the plant operating in seawater desalination mode instead of remaining idle, thus affording high plant utilisation. The treated water is of high grade with energy consumption half that of seawater desalination plants. The average unit production and capital cost of such variable salinity desalination plants is significantly lower than that of seawater desalination plants. (Sheah, 2010)

 So the question I have is whether or not this concept might be applied to a water supply system that contains stormwater collection facilities and catchments. Are there possibilities out there that would allow for a treatment plant that typically treats cleaner river water on a regular basis, but changes operating modes to treat stormwater on an as-need basis? If the plant could be maintained at the design capacity through the operational switch, could it be economical?

Potential for IWRM Application: Queensland, Australia

Introduction

According to an article here, a major energy project could potentially impact future water supplies in Queensland, Australia. The project is to develop coal seam gas and could be worth over $30 billion, which would make it the most significant energy project in Queensland. One man who was interviewed stated his concern over possible impacts to farmland and underground water supplies when this project is put into place. The title of the article is "Water Supply Major Concern." With a title like that, I felt it was time to investigate this a little further to see what was being done to ensure sustainable water supply for both the local farmers, the environment, and the energy needs of the state. The opposition leader John-Paul Langbroek said, "[The] onus will be on this State Government to ensure that all of the operating conditions are met by the industry to protect the communities, the farming land and water."



Background

The Queensland government has set up a coal infrastructure task force with the purpose of developing a strategic plan "for the provision of infrastructure required to meet the needs of the Queensland coal industry over the next 20 years." The responsibilities of the taskforce are to:

• oversee implementation of the Coal Infrastructure Program of Actions
• develope a long-term (20 year) strategic infrastructure plan to identify coal infrastructure and related social infrastructure needed to support increased coal exploration, mining and export
• work with coal seam gas (CSG) producers, local governments and other state government agencies to identify beneficial re-use options for coal seam gas water.

The third bullet is of interest because it could directly impact future water supplies. I looked into this a bit more and found the state has developed a policy to protect the environment and encourage the beneficial use of used coal seam gas (CSG) water. CSG operators are required to submit environmental management plans demonstrating how they plan to manage CSG water. The management plan must address 

• quantity and quality of CSG water
• proposed management of the water and criteria used to assess the management of this water
• actions taken if the management was not effective

The main thrust of the management plan is to prevent salt (produced through the CSG process) to re-enter the environment.


The original article mentioned concern over the groundwater aquifers in Queensland due to future energy projects. I looked into the work being done by the state to manage the groundwater and found they produced an information portal, fact sheet on this website, and are currently developing a statutory framework to protect the groundwater resource. The portal provides information about the groundwater levels, quality, well production, and gas production volumes.




Conclusion


It appears the government is trying to be proactive in this process. This fact was not adequately portrayed in the article. I would like to give notice to these government efforts at this time but also suggest that more might be done in regards to a more integrated approach to future water resources of the area. Rather than look only at the effectiveness of the CSG management plans in dealing with salt content, they could look to see if there are ways to benefit all parties involved, including the environment. An IWRM approach would help the stakeholders involved take a step back and look at everyone and everything that is potential affected by the CSG projects. Could an energy project actually help farmers improve their water supply reliability through conjunctive use? Could farmers make some potentially beneficial trade-offs for CSG reuse water? Could environmental mitigation strategies be included in the CSG management plans?


I wonder if the Coal Infrastructure Task Force could consider these types of questions.

Good references for learning about IWRM and modeling

How do you incorporate the IWRM framework into modeling, planning, and decision-making? Before attempting to answer this question, it is a good idea to review what others have done about it. Below is a list of references that might help shed some light on the subject.

**Added in 2011:


Rahaman, Muhammad Mizanur and Vari, Olli. Integrated water resources management: evolution, prospects and future challenges. 2005. Integrated Water Resource Management, Vol. 1, Issue 1.

U.S. Army Corps of Engineers (USACE). Building Strong Collaborative Relationships for a Sustainable Water Resources Future. National Report: Responding to National Water Resources Challenges. 2010

James, AJ. Institutional challenges for water resources management: India and South Africa. 2003. WHIRL Project Working Paper 7 (draft).

Hooper, Bruce. Integrated Water Resources Management: Governance, Best Practice, and Research Challenge. Universities Council on Water resource. Journal of Contemporary Water research & education, issue 135, pages 1-7 december 2006

**Original list:

Ahmad, S., & Prashar, D. (2010). Evaluating MunicipalWater Conservation Policies Using a Dynamic Simulation Model. Water Resour Management , DOI 10.1007/s11269-010-9611-2.

CH2M HILL, Franson Civil Engineers. (2007). Conceptual Analysis of Uinta and Green River Water Development Projects. Salt Lake City: CH2M HILL. http://www.riversimulator.org/Resources/USBR/UintaGreenRiverWaterDevelopmentReport.pdf

Costanza, R., & Ruth, M. (1998). Using Dynamic Modeling to Scope Environmental Problems and Build Consensus. Environmental Management Vol. 22, No. 2 , 183–195.

Elmahdi, A., Mainuddin, M., & Kirby, M. (2009). Water Balance Dynamic Simulation Model-WBDSim for Water Policy options Analysis. 18th World IMACS / MODSIM Congress. Cairns, Australia: http://mssanz.org.au/modsim09.

Federal Ministry of Education. (2010). Integrated Water Resource Management in the Lower Jordan Valley . Retrieved from http://www.bmbf.wasserressourcen-management.de/en/109.php

Grigg, N. S. (1996). Water Resources Management: Principles, Regulations, and Cases. New York: McGraw-Hill.

GWP, G. W. (2000). Integrated Water Resources Management, TAC Background Papers No. 4. Stockholm, Sweden: Global Water Partnership.

Hollocks, B. W. (1995). The impact of simulation in manufacturing decision making . Control Engineering Practice, Volume 3, Issue 1 , 105-112 .

International Water Association. (2010). Principles of Integrated Water Resources Management in Urban Areas. http://www.gdrc.org/uem/water/iwrm/1pager-01.html.

Islam, N. e. (2010). Callite: A California Central Valley Water Management Screening Model. Journal of Water Resources Planning and Management .

Lillywhite, J. (2008). Performance of Water Supply Operations Measured by Reliability and Marginal Cost (Thesis). Salt Lake City: University of Utah.

Martinec, J. (1975). Snowmelt-Runoff Model for stream flow forecasts. Nordic Hydrology 6(3) , 145-154.

Mekong River Commission. (2010). Basin Development Plan. Retrieved from Mekong River Commission: For Sustainable Development: http://www.mrcmekong.org/programmes/bdp.htm

NeWater. (2009). Adaptive Integrated Water Resources Management (AWM): Explicitely addressing today's challenges. University of Osnabruck, Germany: NeWater.

NeWater. (2007). Adaptive Water Management: How to Cope with Uncertainty. NeWater, Policy Brief.

Pacific Islands Applied Geoscience Commission. (n.d.). SOPAC Water, Sanitation and Hygiene. Retrieved 2010, from http://www.pacificwater.org/pages.cfm/resource-center/water-tools/iwrm-toolboxes-1/integrated-water-resource-management-planning-process.html

Rosbjerg, D., & Knudsen, J. (1983). Integrated water resources management within the Susa basin. Scientific Procedures Applied to the Planning, Design and Management of Water Resources Systems. Hamburg Symposium.

Rosenberg, D. E. (2008). Integrated Water Management and Modeling at Multiple Spatial Scales. University of California Davis.

S. R. Carpenter1, N. F. (1998). NONPOINT POLLUTION OF SURFACE WATERS WITH PHOSPHORUS AND NITROGEN. Ecological Applications , 559-568.

Simonovic, S., & Fahmy, H. (1999). A new modeling approach for water resources policy analysis. Water Resources Research, Vol. 35, No. 1 , 295-304.

Stave, K. A. (2003). A system dynamics model to facilitate public understanding of water management options in Las Vegas, Nevada. Journal of Environmental Management 67 , 303-313.

Tellinghuisen, S. (2010). Protecting the Lifeline of the West, How Climate and Clean Energy Policies Can Safeguard Water. Western Resource Advocates and Environmental Defense Fund.

UN-Water, F. a. (2010). UN Water. Retrieved September 2010, from http://www.unwater.org/statistics_use.html

USDA. (2008, November 11). An Advanced Modular Modeling Framework for Agricultural Systems and International Collaboration for Building Models of the Future. Retrieved 2010, from USDA Agricultural Research Service Unit Mission: http://www.ars.usda.gov/Main/docs.htm?docid=17727

Winz, I., Brierley, G., & Trowsdale, S. (2009). The Use of System Dynamics Simulation in Water Resources Management. Water Resources Management , 23.

WSSD, T. A. (2002). Johannesburg World Summit on Sustainable Development (WSSD).

Introduction to Integrated Water Management

Integrated Water Management (IWRM) is the practice of making decisions and taking actions while considering multiple viewpoints of how water should be managed (www.waterencyclopedia.com, 2010). IWRM has been defined by the Technical Committee of the Global Water Partnership  (GWP) as "a process which promotes the coordinated development and management of water, land and related resources, in order to maximize the resultant economic and social welfare in an equitable manner without compromising the sustainability of vital ecosystems.

IWRM is a systematic process for the sustainable development, allocation and monitoring of water resource use in the context of social, economic and environmental objectives. IWRM is based on the understanding that water resources are an integral component of the ecosystem, a natural resource, and a social and economic good. Typical components of this framework are shown in the figure below.

IWRM is a way to plan for and manage water resources with more than the single water supplier in mind but instead the view of cause and effect relationships throughout a region. IBM assisted Northern California with an Integrated Water Resources Management Plan and provided the following graphic on their webpage as a way to demonstrate possible components of a regional water system.

An example of a typical IWRM project is depicted in the following figure, which is based on real world modeling examples. In this example, surface water supplies are exported from a natural river system which happens to be affected by external human influences and the natural elements. Actions taken by the system can also impact external human actions outside the system. Each structural component is coupled with aspects of viewpoints, policy, and economics. Other influences such as recreation and the environment could also play a significant role within the system simulation.

IWRM should be viewed as a process rather a one-shot approach -one that is long-term and forward-moving but iterative rather than linear in nature. This iterative approach requires a tool that is easy to customize and refine based on changing needs. The following flow diagram shows a possible modeling scenario using this type of process.