Validation of pesticide root zone model 3.12: employing uncertainty analysis

Computer models are being increasingly used to provide an efficient cost-effective means of evaluating the fate and behavior of chemicals in the environment. These models can be used in lieu of or in conjunction with field studies. Because of the increasing reliance on models for critical regulatory decision making, the need arose to assess the validity of regulatory models via an analysis of the correlation of model response estimates with measured data. In conjunction with the evaluation of the correlation of model response estimates and measured field data, a rigorous statistically based validation was also warranted. Because of the unique nature of the correlative exercise using modeled and measured data, standard statistical analyses, while informative, failed to encompass factors associated with the uncertainty of measured environmental fate data and potential model inputs. In an effort to evaluate this uncertainty, an initial sensitivity analysis was performed where key model input parameters for runoff and leaching simulations were identified. Once the sensitive input parameters were identified, a Monte Carlo-based preprocessor was developed whereby the sampling distributions of these parameters were used to propagate uncertainty in the input parameters into error in model predictions. Importantly, assumptions about parameter distributions for input into the Monte Carlo tool were made only after a formal detailed site-specific analysis of measured field data. Employing the functionality of the Crystal Ball Pro development environment, the pesticide root zone model (PRZM) 3.12 was run iteratively for 500 trials, and model output was collated and analyzed. The model predictions were considered reasonably accurate for most regulatory requirements, and the model prediction error was considered acceptable.     

Development of a Monte Carlo sampling shell for the pesticide root zone model and its application by the Federal Insecticide,Fungicide, and Rodenticide Act Environmental Modeling Validation Task Force

A user interface to the U.S. Environmental Protection Agency pesticide root zone model (PRZM) was constructed to allow Monte Carlo sampling of input parameter distributions. The interface was constructed employing the Visual Basic for Applications development environment, along with the functionality of the Crystal Ball Professional forecasting and risk analysis package. The tool has been utilized by the Federal Insecticide, Fungicide, and Rodenticide Act (FIFRA) Environmental Model Validation Task Force to perform detailed statistical analyses of model input parameter uncertainty and the propagation of this uncertainty on the model outputs as well as comparisons of modeled and field-measured data.     

Comparison of pesticide root zone model 3.12: runoff predictions with field data

As part of a process to improve confidence in the results of regulatory modeling, predictions of pesticide root zone model (PRZM) 3.12 were compared with measured data collected in nine different runoff field studies. This comparison shows that PRZM 3.12 provides a reasonable estimate of chemical runoff at the edge of a field. Simulations based on the best choices for input parameters (no conservatism built into input parameters) are generally within an order of magnitude of measured data, with better agreement observed both for larger events and for cumulative values over the study period. When the model input parameters are calibrated to improve the hydrology, the fit between predicted and observed data improves (results are usually within a factor of three). When conservatism is deliberately introduced into the input pesticide parameters, substantial overprediction of runoff losses occur. Recommendations for future work to improve regulatory models include implementation of more sophisticated evapotranspiration routines, allowing for seasonal variation of various model parameters (such as curve numbers, crop cover, and Manning's surface roughness coefficients), better procedures for estimating site-specific degradation rates in surface and subsoils, and improved sorption routines.     

Comparison of pesticide root zone model 3.12: leaching predictions with field data

As part of a process to improve confidence in the results of regulatory modeling, predictions of the pesticide root zone model (PRZM) 3.12 were compared with measured data collected in nine different field leaching studies. Reasonable estimates of leaching were obtained with PRZM 3.12 in homogeneous soils where preferential flow is not significant. The PRZM 3.12 usually did a good job of predicting movement of bromide in soil (soil and soil pore-water concentrations were generally within a factor of two of predicted values). For simulations based on the best choices for input parameters, predictions of soil pore-water concentrations for pesticides were usually within a factor of three and soil pore-water estimates within a factor of 11. When the model input parameters were calibrated to improve the simulation of hydrology, predicted pesticide concentrations in soil pore water were usually within a factor of two of measured concentrations. Because of the sensitivity of leaching to degradation rate, the most accurate predictions were obtained with pesticides with relatively slow degradation rates. When conservative assumptions were used to define input pesticide parameters, predictions of pesticide concentrations were usually a factor of two greater than when using the best estimate of input parameters without any built-in conservatism.     

Sensitivity analysis for validating expert opinion as to ideal data set criteria for transport modeling

Environmental fate modeling results are often used in risk assessment without adequately considering uncertainty in exposure predictions. Sensitivity analysis is fundamental to model validation and error prediction since sensitive model input parameters account for the largest variance in model prediction. Once identified, sensitive model input parameters can be used to propagate parametric uncertainty in numerical predictions. Output sensitivity to variation in input code sequences was investigated for the pesticide root zone model (PRZM 3) using Plackett-Burman analysis for six runoff and leaching data sets. The analysis utilized an incomplete block factorial design with even parameter weighting and uniform proportional input perturbation. Timing and duration of key period rainfall were assumed a priori to be dominant sensitive inputs. Thus, meteorological data were fixed, allowing identification of additional input components contributing to model sensitivity. Results validated expert modeler assumptions concerning parameters most critical for model validation. For leaching data sets, the application rate, soil bulk density (an indicator of available water-holding capacity), chemical partition coefficient, and pesticide degradation rates were commonly the most sensitive inputs. For runoff data sets, the in-crop runoff curve number was the most significant input governing pesticide loss in runoff and erosion flux. The chemical partition coefficient, soil and foliar decay rates, and soil bulk density were also common sensitive components for runoff predictions. These commonly observed sensitive components for runoff and leaching prediction need to be carefully considered in the design and conduct of relevant field studies, modeling assessment of such studies, and future improvements in algorithms for environmental transport modeling.     

Review of the validation of models used in Federal Insecticide,Fungicide, and Rodenticide Act environmental exposure assessments

The first activity of the Federal Insecticide. Fungicide, and Rodenticide Act (FIFRA) Environmental Model Validation Task Force, established to increase confidence in the use of environmental models used in regulatory assessments, was to review the literature information on validation of the pesticide root zone model (PRZM) and the groundwater loading effects of agricultural management systems (GLEAMS). This literature information indicates that these models generally predict the same or greater leaching than observed in actual field measurements, suggesting that these models are suitable for use in regulatory assessments. However, additional validation research conducted using the newest versions of the models would help improve confidence in runoff and leaching predictions because significant revisions have been made in models over the years, few of the literature studies focused on runoff losses, the number of studies having quantitative validation results is minimal, and modelers were aware of the field results in most of the literature studies. Areas for special consideration in conducting model validation research include improving the process for selecting input parameters, developing recommendations for performing calibration simulations, devising appropriate procedures for keeping results of field studies from modelers performing simulations to validate model predictions while providing access for calibration simulations, and developing quantitative statistical procedures for comparing model predictions with experimental results.     

Estimating the predictive quality of dose-response after model selection

Prediction of dose-response is important in dose selection in drug development. As the true dose-response shape is generally unknown, model selection is frequently used, and predictions based on the final selected model. Correctly assessing the quality of the predictions requires accounting for the uncertainties caused by the model selection process, which has been difficult. Recently, a new approach called data perturbation has emerged. It allows important predictive characteristics be computed while taking model selection into consideration. We study, through simulation, the performance of data perturbation in estimating standard error of parameter estimates and prediction errors. Data perturbation was found to give excellent prediction error estimates, although at times large Monte Carlo sizes were needed to obtain good standard error estimates. Overall, it is a useful tool to characterize uncertainties in dose-response predictions, with the potential of allowing more accurate dose selection in drug development. We also look at the influence of model selection on estimation bias. This leads to insights into candidate model choices that enable good dose-response prediction.     

Application of Mathematical Models in Combination with Monte Carlo Simulation for Prediction of Isoflurane Concentration in an Operation Room Theater

Applicability of two mathematical models in inhalation exposure prediction (well mixed room and near field-far field model) were validated against standard sampling method in one operation room for isoflurane. Ninety six air samples were collected from near and far field of the room and quantified by gas chromatography-flame ionization detector. Isoflurane concentration was also predicted by the models. Monte Carlo simulation was used to incorporate the role of parameters variability. The models relatively gave more conservative results than the measurements. There was no significant difference between the models and direct measurements results. There was no difference between the concentration prediction of well mixed room model and near field far field model. It suggests that the dispersion regime in room was close to well mixed situation. Direct sampling showed that the exposure in the same room for same type of operation could be up to 17 times variable which can be incorporated by Monte Carlo simulation. Mathematical models are valuable option for prediction of exposure in operation rooms. Our results also suggest that incorporating the role of parameters variability by conducting Monte Carlo simulation can enhance the strength of prediction in occupational hygiene decision making.     

Evaluating and expressing the propagation of uncertainty in chemical fate and bioaccumulation models

First-order analytical sensitivity and uncertainty analysis for environmental chemical fate models is described and applied to a regional contaminant fate model and a food web bioaccumulation model. By assuming linear relationships between inputs and outputs, independence, and log-normal distributions of input variables, a relationship between uncertainty in input parameters and uncertainty in output parameters can be derived, yielding results that are consistent with a Monte Carlo analysis with similar input assumptions. A graphical technique is devised for interpreting and communicating uncertainty propagation as a function of variance in input parameters and model sensitivity. The suggested approach is less calculationally intensive than Monte Carlo analysis and is appropriate for preliminary assessment of uncertainty when models are applied to generic environments or to large geographic areas or when detailed parameterization of input uncertainties is unwarranted or impossible. This approach is particularly useful as a starting point for identification of sensitive model inputs at the early stages of applying a generic contaminant fate model to a specific environmental scenario, as a tool to support refinements of the model and the uncertainty analysis for site-specific scenarios, or for examining defined end points. The analysis identifies those input parameters that contribute significantly to uncertainty in outputs, enabling attention to be focused on defining median values and more appropriate distributions to describe these variables.     

Issues of replicability in Monte Carlo modeling: a case study with a pesticide leaching model

Sensitivity and uncertainty analyses based on Monte Carlo sampling were undertaken for various numbers of runs of the pesticide leaching model (PELMO). Analyses were repeated 10 times with different seed numbers. The ranking of PELMO input parameters according to their influence on predictions for leaching was stable for the most influential parameters. For less influential parameters, the sensitivity ranking was severely influenced by the seed number used. For uncertainty analyses, probabilities of exceeding a particular concentration were significantly influenced by the seed number used in the random sampling of values for the two parameters considered, even for those cases in which 5,000 model runs were undertaken (coefficient of variation of 10 replicated analyses, 5%). A decrease in the variability of exceedance probabilities could be achieved by further increasing the number of model runs. However, this may prove to be impractical when complex deterministic models with a relatively long running time are used. Attention should be paid to replicability aspects by modelers when devising their approach to assessing the uncertainty associated with the modeling and by decision makers when examining the results of probabilistic approaches.     

Reliability of a new biokinetic model of zirconium in internal dosimetry: part I, parameter uncertainty analysis

The reliability of biokinetic models is essential in internal dose assessments and radiation risk analysis for the public, occupational workers, and patients exposed to radionuclides. In this paper, a method for assessing the reliability of biokinetic models by means of uncertainty and sensitivity analysis was developed. The paper is divided into two parts. In the first part of the study published here, the uncertainty sources of the model parameters for zirconium (Zr), developed by the International Commission on Radiological Protection (ICRP), were identified and analyzed. Furthermore, the uncertainty of the biokinetic experimental measurement performed at the Helmholtz Zentrum München-German Research Center for Environmental Health (HMGU) for developing a new biokinetic model of Zr was analyzed according to the Guide to the Expression of Uncertainty in Measurement, published by the International Organization for Standardization. The confidence interval and distribution of model parameters of the ICRP and HMGU Zr biokinetic models were evaluated. As a result of computer biokinetic modelings, the mean, standard uncertainty, and confidence interval of model prediction calculated based on the model parameter uncertainty were presented and compared to the plasma clearance and urinary excretion measured after intravenous administration. It was shown that for the most important compartment, the plasma, the uncertainty evaluated for the HMGU model was much smaller than that for the ICRP model; that phenomenon was observed for other organs and tissues as well. The uncertainty of the integral of the radioactivity of Zr up to 50 y calculated by the HMGU model after ingestion by adult members of the public was shown to be smaller by a factor of two than that of the ICRP model. It was also shown that the distribution type of the model parameter strongly influences the model prediction, and the correlation of the model input parameters affects the model prediction to a certain extent depending on the strength of the correlation. In the case of model prediction, the qualitative comparison of the model predictions with the measured plasma and urinary data showed the HMGU model to be more reliable than the ICRP model; quantitatively, the uncertainty model prediction by the HMGU systemic biokinetic model is smaller than that of the ICRP model. The uncertainty information on the model parameters analyzed in this study was used in the second part of the paper regarding a sensitivity analysis of the Zr biokinetic models.     

Uncertainties analysis for the plutonium dosimetry model, doses-2005, using Mayak bioassay data

The Doses-2005 model is a combination of the International Commission on Radiological Protection (ICRP) models modified using data from the Mayak Production Association cohort. Surrogate doses from inhaled plutonium can be assigned to approximately 29% of the Mayak workers using their urine bioassay measurements and other history records. The purpose of this study was to quantify and qualify the uncertainties in the estimates for radiation doses calculated with the Doses-2005 model by using Monte Carlo methods and perturbation theory. The average uncertainty in the yearly dose estimates for most organs was approximately 100% regardless of the transportability classification. The relative source of the uncertainties comes from three main sources: 45% from the urine bioassay measurements, 29% from the Doses-2005 model parameters, and 26% from the reference masses for the organs. The most significant reduction in the overall dose uncertainties would result from improved methods in bioassay measurement with additional improvements generated through further model refinement. Additional uncertainties were determined for dose estimates resulting from changes in the transportability classification and the smoking toggle. A comparison was performed to determine the effect of using the model with data from either urine bioassay or autopsy data; no direct correlation could be established. Analysis of the model using autopsy data and incorporation of results from other research efforts that have utilized plutonium ICRP models could improve the Doses-2005 model and reduce the overall uncertainty in the dose estimates.     

Substantial effective sample sizes were required for external validation studies of predictive logistic regression models

BACKGROUND AND OBJECTIVES: The performance of a prediction model is usually worse in external validation data compared to the development data. We aimed to determine at which effective sample sizes (i.e., number of events) relevant differences in model performance can be detected with adequate power. METHODS: We used a logistic regression model to predict the probability that residual masses of patients treated for metastatic testicular cancer contained only benign tissue. We performed standard power calculations and Monte Carlo simulations to estimate the numbers of events that are required to detect several types of model invalidity with 80% power at the 5% significance level. RESULTS: A validation sample with 111 events was required to detect that a model predicted too high probabilities, when predictions were on average 1.5 times too high on the odds scale. A decrease in discriminative ability of the model, indicated by a decrease in the c-statistic from 0.83 to 0.73, required 81 to 106 events, depending on the specific scenario. CONCLUSION: We suggest a minimum of 100 events and 100 nonevents for external validation samples. Specific hypotheses may, however, require substantially higher effective sample sizes to obtain adequate power.     

User subjectivity in Monte Carlo modeling of pesticide exposure

Monte Carlo techniques are increasingly used in pesticide exposure modeling to evaluate the uncertainty in predictions arising from uncertainty in input parameters and to estimate the confidence that should be assigned to the modeling results. The approach typically involves running a deterministic model repeatedly for a large number of input values sampled from statistical distributions. In the present study, six modelers made choices regarding the type and parameterization of distributions assigned to degradation and sorption data for an example pesticide, the correlation between the parameters, the tool and method used for sampling, and the number of samples generated. A leaching assessment was carried out using a single model and scenario and all data for sorption and degradation generated by the six modelers. The distributions of sampled parameters differed between the modelers, and the agreement with the measured data was variable. Large differences were found between the upper percentiles of simulated concentrations in leachate. The probability of exceeding 0.1 microg/L ranged from 0 to 35.7%. The present study demonstrated that subjective choices made in Monte Carlo modeling introduce variability into probabilistic modeling and that the results need to be interpreted with care.     

Error analysis for the in-vivo measurement of radionuclides in wounds: Monte Carlo study

This paper describes calculation of error associated with the direct in-vivo measurements of radionuclides in a wound. A typical radiation injury to a hand with Am radionuclide is illustrated for error analysis. A Monte Carlo model was developed and the detector pulse spectrum studied with a custom-designed HPGe detector. A pinhole collimator was designed, and its performance with a wide area detector was studied. The results show that significant errors might propagate if the lowest energy peaks of Am are used during in vivo measurements of the wound. In comparison to that, less uncertainty was found for 26.3 and 59.5 keV gamma peaks, and those levels are recommended for estimation of wound depth and activity.     

Statistical analysis of Clewell et al. PBPK model of trichloroethylene kinetics

A physiologically based pharmacokinetic model for trichloroethylene (TCE) in rodents and humans was calibrated with published toxicokinetic data sets. A Bayesian statistical framework was used to combine previous information about the model parameters with the data likelihood, to yield posterior parameter distributions. The use of the hierarchical statistical model yielded estimates of both variability between experimental groups and uncertainty in TCE toxicokinetics. After adjustment of the model by Markov chain Monte Carlo sampling, estimates of variability for the animal or human metabolic parameters ranged from a factor of 1.5-2 (geometric standard deviation [GSD]). Uncertainty was of the same order as variability for animals and higher than variability for humans. The model was used to make posterior predictions for several measures of cancer risk. These predictions were affected by both uncertainties and variability and exhibited GSDs ranging from 2 to 6 in mice and rats and from 2 to 10 for humans.     

Longitudinal variable selection by cross-validation in the case of many covariates

Longitudinal models are commonly used for studying data collected on individuals repeatedly through time. While there are now a variety of such models available (marginal models, mixed effects models, etc.), far fewer options exist for the closely related issue of variable selection. In addition, longitudinal data typically derive from medical or other large-scale studies where often large numbers of potential explanatory variables and hence even larger numbers of candidate models must be considered. Cross-validation is a popular method for variable selection based on the predictive ability of the model. Here, we propose a cross-validation Markov chain Monte Carlo procedure as a general variable selection tool which avoids the need to visit all candidate models. Inclusion of a 'one-standard error' rule provides users with a collection of good models as is often desired. We demonstrate the effectiveness of our procedure both in a simulation setting and in a real application.     

Two‐Locus Modeling of Asthma in a Hutterite Pedigree via Markov Chain Monte Carlo

Bayesian Markov chain Monte Carlo (MCMC) segregation analysis for asthma was performed on the whole 1,544‐member Hutterite pedigree. Heterogeneous and epistatic two‐locus models and complex one‐locus models were investigated, with trait loci postulated to be linked to markers in regions previously found to be possibly linked to asthma or atopy. The epistatic two‐locus dominant‐dominant model provided the best estimates, among the models investigated, in terms of prediction of population prevalence and relative risk for sibs of the affecteds. © 2001 Wiley‐Liss, Inc.