Expansion of the CIROH Integrated Evaluation System to Accelerate Research to Operations
Objective:
This research addresses the challenge of evaluating hydrologic forecast improvements in a systematic, consistent, and accessible way so that new models and methods can be evaluated and transitioned into operations. The product being developed is called TEEHR (Tools for Exploratory Evaluation in Hydrologic Research), an open-source tool and platform that enables researchers and developers to systematically assess forecasting advancements, helping to improve water prediction accuracy for flood warnings, drought monitoring, and resource management. Key improvements include a growing list of standardized evaluation methods, scalable computing to evaluate continental scale datasets, evaluation-ready datasets published to an open data warehouse to foster community engagement, and web-based dashboards to make evaluation available to a wide range of stakeholders and decision makers.
Approach:
Our approach is to develop TEEHR through an iterative, community-driven process that keeps pace with evolving research and operational needs. This includes ongoing communication with CIROH researchers and OWP stakeholders to clarify current requirements and understand where evaluation needs are headed. Recognizing that needs and use cases are a moving target, our strategy is deliberately flexible so that TEEHR remains relevant even as R2O processes evolve.
We will enhance TEEHR’s core functionality by adding to the growing list of evaluation methods and improving computational efficiency using “big data” technologies to facilitate the analysis of continental-scale datasets. A centralized, open data warehouse will be set up to store evaluation-ready datasets, ensuring that researchers have ready access to robust data. In addition, we will improve user interfaces by creating both code-based tools (TEEHR) and web-based, real-time dashboards (TEEHR Evaluation Service), which simplify complex analyses and provide access to a wider audience.
To drive adoption and foster innovation, our plan includes extensive documentation, training workshops, community hack-a-thons, and presentations at conferences such as AMS, AGU, and CIROH DevCon. This iterative development and continuous feedback loop will ensure that TEEHR not only addresses current needs for hydrologic forecast evaluation but also sets a new standard for the continental scale research-to-operations process.
Impact:
TEEHR provides researchers, developers, and scientists a scalable, open-source framework for efficiently analyzing hydrologic simulation data, through exploratory evaluations, to gain deeper insights into model performance and accuracy. The TEEHR Evaluation System, a web-based Evaluation as a Service (EaaS) platform, extends these capabilities to a broader audience by providing standardized evaluations and ready-to-use datasets for real-time operational assessments. Together, these tools will bridge the gap between research and operations and allow decision makers to easily track and compare model/forecast performance across products and research and understand the evolution of model performance over time.Abstract:
Project Introduction (What is it?)
This project focuses on improving the way hydrologic forecasts and models are evaluated, ensuring that advancements can be systematically evaluated as part of the research-to-operations process (R2O). It builds upon the prototype evaluation system called TEEHR (Tools for Exploratory Evaluation in Hydrologic Research), an open-source Python framework designed for researchers, developers, and scientists that can be used to load, store, and analyze large volumes hydrologic simulation data. Additionally, the TEEHR Evaluation System includes a web-based Evaluation as a Service (EaaS) platform aimed at agencies, water managers, and decision-makers, offering standardized evaluation dashboards and evaluation-ready datasets.
Context (Why is it needed?)
Hydrologic forecasting is essential for public safety and resource management, helping predict floods, droughts, and water availability. However, evaluating forecasting improvements has been inconsistent, fragmented, and difficult to scale, making it challenging to determine which advancements offer real benefits. With an unprecedented surge in hydrologic forecasting research, the need for research outcomes to be systematically evaluated alongside baseline operational analyses and forecasts is more critical than ever. Before CIROH, the absence of standardized evaluation tools hindered researchers from reliably measuring success and transitioning innovations into operations. This project addresses these needs by delivering Python and cloud-based, scalable, and standardized evaluation tools that enable a broad range of stakeholders to assess forecast performance, compare models, and confidently advance research into operational use.
Product of the Research (What is being developed?)
This project will deliver a robust evaluation framework and web-based evaluation service, addressing key limitations in forecast assessment and will engage with the community to increase adoption
• TEEHR (Python-Based Evaluation Framework):
o Is a Python framework that provides robust, scalable, big data, open-source data analytics tools for loading, storing, and processing large amounts hydrologic simulation data for the purpose of exploring and evaluating the datasets to assess their skill and performance
o In addition to the standard evaluation metrics, TEEHR provides expanded capabilities including ensemble metrics, uncertainty quantification, and event-based evaluations.
• TEEHR Evaluation System (Web-Based Evaluation as a Service):
o Offers standardized evaluations via an intuitive dashboard and hosted real-time evaluation tools.
o Publishes evaluation-ready datasets in an open data warehouse to support broad accessibility.
o Will automatically ingest and process operational and experimental forecasts, published and experimental historic simulations, and integrate with NextGen Research Data Stream (NRDS) and the CIROH Hydrometeorological Test Bed (CHMT).
• Increase Community Adoption
o Training workshops, comprehensive documentation, shared feedback mechanisms, open evaluation-ready datasets (including in OCONUS), and community evaluation-focused events (i.e., “eval-a-thons”).
o Presentations at conferences such as AMS, AGU, and CIROH DevCon.
Impact (Why is it important?)
This project ensures that CIROH research outcomes are consistently and objectively evaluated, with tools that are accessible and easy to use by a diverse community of stakeholders. TEEHR, designed for researchers, developers, and scientists, provides a scalable, open-source Python framework for efficiently analyzing large hydrologic datasets, enabling robust forecasting assessments. The TEEHR Evaluation System, a web-based Evaluation as a Service (EaaS) platform, caters to agencies, operational forecasters, water managers, and decision-makers, offering standardized evaluations and ready-to-use datasets to streamline forecast comparisons and operational assessments.
Beyond serving as the evaluation com