Validation of Computer Codes and Calculation Methods

Title of document

ONR GUIDE

Validation of Computer Codes and Calculation Methods

Document Type:

Nuclear Safety Technical Assessment Guide

Unique Document ID and Revision No:
Date Issued:

NS-TAST-GD-042 Revision 5

March 2019

Review Date:

September 2023

Approved by:

Rob Exley

Professional Lead

Record Reference: Revision commentary:

CM9 Folder 1.1.3.978. (2020/262106) Rev 4: Major review including addition of new appendices Rev 5: Updated Review Period

TABLE OF CONTENTS
1. INTRODUCTION.................................................................................................................. 2 2. PURPOSE AND SCOPE ..................................................................................................... 2 3. RELATIONSHIP TO LICENCE AND OTHER RELEVANT LEGISLATION.......................... 2 4. RELATIONSHIP TO SAPS, WENRA REFERENCE LEVELS AND IAEA SAFETY
STANDARDS ADDRESSED................................................................................................ 3 5. ADVICE TO INSPECTORS ................................................................................................. 3 6. REFERENCES................................................................................................................... 13 7. GLOSSARY AND ABBREVIATIONS ................................................................................. 14 APPENDICES .......................................................................................................................... 15

© Office for Nuclear Regulation, 2019 If you wish to reuse this information visit www.onr.org.uk/copyright for details. Published 03/19

Template Ref: ONR-DOC-TEMP-002 Revision 3

OFFICIAL

Page 1 of 22

Office for Nuclear Regulation

1. INTRODUCTION

1

ONR has established its Safety Assessment Principles (SAPs) [1] which apply to the

assessment by ONR specialist inspectors (assessors) of safety cases for nuclear

facilities that may be submitted by licensees, or other duty-holders – e.g. a

requesting party in Generic Design Assessment (GDA). This technical assessment

guide (TAG) is one of the guides developed by ONR to further assist ONR’s

inspectors in their assessment work in support of making regulatory judgements and

decisions.

2. PURPOSE AND SCOPE

2

This TAG provides advice to ONR assessors on the interpretation of the SAPs

covering the validation of computer codes and other calculation methods used to

perform plant analysis in support of the safety cases. The guide also explains what

the ONR assessor should expect to see in a validation report produced by a licensee

in support of a code or calculation method used for analysis within a nuclear safety

case.

3

This TAG is intended to apply primarily to the assessment of the validation of

physics, thermal and structural analysis computer codes and calculation methods

used within the design basis safety studies. Any validation submission for beyond

design basis calculation methods should conform in a general way to the guidance

given in this TAG. The principles outlined here are also generally applicable to

transient, radiological and other analyses forming part of fault analysis and also in

other areas of the safety case underpinned by analysis and/or data, e.g. engineering

substantiation. Advice to inspectors assessing nuclear installations chemistry, C&I

and other studies is provided in the relevant specific TAGs.

4

With respect to Computational Fluid Dynamics (CFD) analysis methods, which have

special application, Appendix 1 provides guidance on the main issues which should

be covered in CFD reports, and their significance.

5

Appendix 2 provides base information on the Code Scaling, Applicability, and

Uncertainty evaluation (CSAU) methodology [10], which was developed by the US

NRC and is considered relevant good practice for validation of computer calculations

in the safety analysis area.

6

Practical guidance for the validation of Finite Element Analyses in the structural

integrity area is presented in Appendix 3.

7

This TAG is not directly applicable to the assessment of the validity of software used

for control and protection of operational nuclear plant and processes. These are

covered in ONR’s NS-TAST-GD-046 Computer Based Safety Systems, which

provides references to the applicable standards, e.g. IEC 60880:2006 Nuclear power

plants - Instrumentation and control systems important to safety - Software aspects

for computer-based systems performing category A functions.

8

This TAG contains guidance to advise and inform ONR inspectors in the exercise of

their professional regulatory judgement. Comments on this guide, and suggestions

for future revisions, should be emailed to the HOW-2 team.

3. RELATIONSHIP TO LICENCE AND OTHER RELEVANT LEGISLATION

9

Licence Condition 14 - Safety documentation requires the implementation of

adequate arrangements for the production of safety cases. Computer code analysis

Report NS-TAST-GD-042 CM9 Reference – 2020/262106

Page 2 of 22

Office for Nuclear Regulation
of plant design and operation forms an important part of a modern safety case. The computer codes should be validated prior to their use in the production of safety cases.
10 Licence Condition 19 - Installation of new plant requires “adequate documentation to justify the safety of the proposed construction or installation”. In order to be adequate – i.e. suitable and sufficient - the documentation (safety case) should be based on safety analyses carried out by appropriately validated computer codes.
11 Licence Condition 22 –Modification or experiment on existing plant requires the implementation of adequate arrangements for the modification or experiment on existing plants. Computer code analysis of plant modifications forms an important part of the supporting safety cases. The computer codes should be validated prior to their use in the production of safety cases.
12 Licence Condition 23 – Operating Rules requires the production of an adequate safety case which identifies the necessary operating rules and operating conditions. The adequacy of these rules and conditions are often demonstrated by computer codes analyses of design basis faults.
4. RELATIONSHIP TO SAPS, WENRA REFERENCE LEVELS AND IAEA SAFETY STANDARDS ADDRESSED
13 ONR SAPs are benchmarked against the WENRA reference levels for nuclear installation safety and against the applicable IAEA safety standards. Computer codes validation is addressed in SAPs AV.1 to AV.8, ECE.15, etc. This guide provides additional information on the subject, assuming that the inspector is already familiar with the SAPs.
5. ADVICE TO INSPECTORS
5.1 DEFINITIONS 14 Many of the terms used in this TAG are already defined in the SAPs at a general
level. This section provides more detailed interpretations relevant to the TAG specifics.
15 Validation is the process of testing and evaluation of the whole computer code or calculation method after the completion of its development and prior to its application to ensure compliance with the requirements of the intended application. Validation provides the evidence that the computer code or calculation method is fit for purpose by comparison of their results with data from experiments or other trusted sources.
16 Verification is the process of ensuring that the model specification has been complied with and that controlling physical equations have been correctly translated into the computer code or, in the case of hand calculations, correctly incorporated into the calculation procedures.
17 Quality Assurance is the process of reviewing, inspection, testing, checking, auditing or otherwise determining and documenting whether or not items, processes, services or documents which support the computer code or calculation method conform to specified requirements.
Note: Software Verification & Quality Management is not subject to this TAG. Useful information on this subject is provided in (Refs. 2 – 6).

Report NS-TAST-GD-042 CM9 Reference – 2020/262106

Page 3 of 22

Office for Nuclear Regulation

18 The safe state of a nuclear facility is provided by the following Fundamental Safety Functions:
a. Control over criticality and heat generation;
b. Cooling of safety critical components;
c. Containment of the radioactive materials.
19 Safety Parameters are plant parameters which indicate the state of the facility in terms of safety (core power, peak clad temperature, containment pressure, etc.). A safe state is established when the safety parameters are within their safety limits.
20 Fault studies predict the safety parameters behaviour during postulated events to check whether the facility will remain in a safe state during the event (or not). The prediction is based on computer simulation of the relevant equipment and processes.
21 Safety margin is the difference between: the worst value predicted for a safety parameter occurring during the postulated event, and the relevant safety limit.
22 Fault studies computer code is a computer code which applies specific physical correlations and mathematical methods to model (partly or completely) a facility and simulate its behaviour during postulated fault transients. The input data for such codes is usually separated in two groups: (1) parameters of the facility structures, systems and components (SSC) and (2) assumptions for the sequence of events (systems operation/failures, operator actions, etc.). The results provide predictions of the facility safety parameters and allow for evaluation of the safety margin available in each case.
23 The Conservative codes/approach to fault studies aims to calculate a system response using limiting values of model parameters bounding the uncertainty arising from measurement errors, knowledge insufficiency and modelling simplifications by modification of the known physical correlations, measured data and accident conditions in a way that tends to reduce the estimated safety margins. The purpose is to provide a robust demonstration that adequate safety margins are preserved with a high level of confidence. For example, SAP FA.7 recommends: “Analysis of design basis fault sequences should use appropriate tools and techniques, and be performed on a conservative basis to demonstrate that consequences are ALARP”.
24 The main advantage of the conservative approach is its capability to construct an integral model of a complicated system while keeping the cost of model development and running at a reasonably low level.
25 The main disadvantage of this approach is that excessive pessimisms could lead to grossly unrealistic results and prevent a balanced assessment of risk. The inspector must clarify the grounds for selection of the conservative values and assumptions applied in each case.
26 The Best Estimate codes/approach applies the available realistic information about plant parameters, behaviours and phenomena so as to provide an unbiased estimate of the system response. It is used primarily in Probabilistic Safety Assessment (PSA) for evaluation of DBA Initiating Fault Frequencies (IFFs), and for Severe Accident Analysis (SAA).
27 The main advantage of this approach is the production of a realistic view of the accident development. From the licensee’s perspective, the combination of a realistic model with a statistical treatment of uncertainties can provide a larger demonstrated

Report NS-TAST-GD-042 CM9 Reference – 2020/262106

Page 4 of 22

Office for Nuclear Regulation
safety margin than using a set of pessimistic data. This method is internationally referred to as “Best Estimate Plus Uncertainty” (BEPU).
28 The main disadvantage of this approach is related to the risk of arriving to unreasonably optimistic results due to inadequate treatment of uncertainty. It is worth noting that uncertainty quantification demands significant resources.
29 The inspector should compare the treatment of uncertainty against established good practices e.g. the CSAU method recommended in a US NRC guide (Ref. 10).
5.1 GRADED APPROACH 30 The graded approach to assessment of computer calculations is applied in line with
the ONR principle of proportionality, i.e. application of effort and expenditure of resources in proportion to the safety significance of the subject.
31 The assessor should be aware of where the calculation fits in the overall estimate of risk presented in the safety case and should have an understanding for the level of results confidence and uncertainty qualification of any calculated results. For example, SAPs (Figure 1) illustrate the inter-relationship between the three types of fault analysis, DBA, PSA and SAA, and how, in combination, they address the range of potential initiating events with nuclear safety significance off the site.
32 Where a deterministic safety case is made, analysis will demonstrate to a high level of confidence that safety measures are effective for events initiated within the domain of operating conditions permitted by the plant operating rules (see TAG 35 [14]). The analysis should ideally quote and substantiate confidence levels for its calculations. Where this is not practicable, the safety case should provide enough information to allow a judgement to be made that the results are suitably conservative.
33 However, it may not be reasonable to consider an event where all extreme conditions occur simultaneously. For example, fault sequences occurring in extreme conditions can reasonably be limited to events with a predicted frequency (probability) in the region of 1.E-7 per year. For events of such low probability it is reasonable to account for only the most significant uncertain parameters. The required level of validation rigor should be proportionate to the risk significance of the analysis.
34 In the case of frequent events (which are anticipated within the life of the facility or within the life of a fleet of similar facilities) it may be appropriate to apply statistical techniques to demonstrate that the protection provided by the plant design meets the selected acceptance criteria.
35 Some fault sequences which fall outside the design basis still require analysis to demonstrate that the available safety measures are reasonably practical. Such analysis will normally consider the likely plant response and therefore the use of best-estimate approach may be suitable.
36 The inspector should consider the level of statistical rigor appropriate to the study of each particular fault and compare that presented against established practice. The cost of data needed to substantiate analysis methods should also be considered in the formulating of regulatory expectations.

Report NS-TAST-GD-042 CM9 Reference – 2020/262106

Page 5 of 22

Office for Nuclear Regulation

5.2 VALIDATION REPORT - GENERAL EXPECTATIONS 37 ONR expects a licensee to present a validation report for each of the computer
codes/calculation methods used in a safety case. The ONR assessor needs to be satisfied with this report in a number of areas as explained below.
38 The validation report should identify the shortcomings in the computer code method of solution, the uncertainties of the associated physical models and the inaccuracy in the experimental data used in the validation work. This information should be used to define the sensitivity analyses to be performed as part of the safety case. The object of such analyses is to confirm and demonstrate that “cliff-edge effects” do not exist. The sensitivity analyses should cover the uncertainties/approximations in the mathematical models, input data and boundary conditions of the analysis.
39 A validation report which complies with the established good practices is expected to cover the following areas:
1. limits of application;
2. details of models used;
3. details of numerical methods;
4. correlations used;
5. treatment of uncertainty:
a) comparisons with experimental data; b) comparisons with plant data; c) comparisons with analytical solutions; d) comparisons with other calculation methods; e) bias calculations; f) review of new data;
6. uncertainty of best estimate calculations;
7. user’s proficiency
8. quality assurance;
40 Each of these items is discussed in more detail in the following sub-sections.
5.2.1 LIMITS OF APPLICATION
41 The validation report should define the limits for application of the calculation method and indicate the dominant processes which are expected to occur in any situation to which it is applicable. The limits of applicability are often based on an identifiable change in the dominant processes which are predicted to take place. Calculation methods are often developed to cover a limited range of plant states (e.g. flow regimes). Consequently, different calculation methods may be used for modelling different phases of the analysed fault.
42 The inspector should make sure that the validation report defines the processes which the calculation method is designed to model and identifies the changes in those processes which make the method no longer applicable outside the identified limits.
5.2.2 DETAILS OF PHYSICAL MODELS USED
43 The derivation of the equations used to model the various processes and the simplifying assumptions made should be fully described. Modelling a fault progression requires the development of mathematical equations to describe the processes which are believed to occur. In general, a number of simplifications are made to enable a tractable formulation. For example, complex three-dimensional

Report NS-TAST-GD-042 CM9 Reference – 2020/262106

Page 6 of 22

Office for Nuclear Regulation
geometries may be reduced to one- or two-dimensional approximations in order to simplify the modelling. This is particularly true in modelling turbulence and neutron transport. The validation report should enable the assessor to follow the derivation of the controlling equations and should justify any simplifying assumptions.
5.2.3 NUMERICAL METHODS
44 In many cases the physical complexity of the process being modelled means that an analytical model of the entire domain cannot be derived. Solution of the controlling equations requires numerical approximation techniques, such as finite differences and finite elements methods. The validation report should justify the solution methods used and should demonstrate the accuracy of the numerical approximation. Numerical problems that can occur with such techniques should be listed along with an explanation as to why they will not invalidate the calculations for the safety case.
45 The codes should apply internal procedures to check that the numerical schemes employed provide compliance with the basic conservation laws throughout the calculation. There should be a demonstration, by successive refinement, that the nodalisation used is fine enough to provide a converged solution. Where such approach is not practicable, the report should provide substantiation against integral experiments and explain why any lack of convergence does not invalidate the results.
46 Deterministic methods may be inherently more accurate for the range of conditions upon which the physics is based but validation outside of this domain is questionable. Monte Carlo methods may be subject to good validation in their domain but still fail to converge under certain scenarios, which should be a focus for additional validation effort. Multi-physics assessment is subject to the use of several methods and, just as importantly, several interfaces between physical domains. For this reason, only limited credit may be claimed in safety analysis for the use of multi-physics analysis (e.g. hydrodynamic codes), and there should be an expectation of diverse and sound safety arguments in addition to code results.
5.2.4 CORRELATIONS USED
47 Sometimes the physical complexity of the process being modelled means that the full set of governing equations is not tractable or that it is not practicable to derive them from first principles. In these cases, empirical correlations may be used to represent the essential parts of the physical process and so enable the problem to be 'closed'.
48 The validation report should present the important empirical correlations along with their ranges of applicability. This should be supported by a description of the technical basis and justification for the use of each correlation in the range of interest to the safety case. The report should also explain what steps have been taken to prevent the correlation being used outside that range.
5.2.5 TREATMENT OF UNCERTAINTY
49 The simplified check-list presented below is neither mandatory nor exhaustive. Still the inspector should check for:
 Identification of the input parameters with highest impact on the analysis results (operational experience, experiments, previous calculations, expert judgment,).
 Establishment of the expected uncertainty intervals of the most important input parameters (e.g. based on measurement precision).

Report NS-TAST-GD-042 CM9 Reference – 2020/262106

Page 7 of 22

Office for Nuclear Regulation

 Sensitivity calculations with input variations within the uncertainty intervals to establish the confidence bands for the key safety parameter(s).
 Comparison of the confidence bands for the key safety parameter(s) against the identified safety margins.
 Consistency of the calculated results with the conclusions of the validation report.

Comparison with experimental data
50 One way of testing the combined effect of the various elements of the mathematical modelling is to compare the calculated predictions against experimental results. Two types of experiments are used: 'separate effects' tests are designed to examine at the most a few phenomena, while 'integral' tests are designed to enable most of the phenomena of interest to the nuclear installation to occur in an interactive way. Commissioning test results could also be a useful source for code validation and may be necessary to demonstrate compliance with SAP AV.1.

51 Integral tests are usually limited to fairly small scales by considerations of cost and complexity. Data from both types of experiments should be used to validate the predictive capabilities of the computer code/computation method.

52 Assessors need to be particularly aware of the potential selective use of “comfortable” experiments in the validation report and should seek justification for the exclusion of other experiments which seem relevant.

53 When analysing separate effects tests, the correlations that are being tested should be identified and reference to the claimed accuracy of prediction should be made. A distinction should be drawn between any data base that was used to develop the correlations and that which is being used for the validation exercise. Wherever possible, comparisons should be made with integral experiments at a range of scales to explore the ability of the calculation method to extrapolate from small scale tests to the conditions of the nuclear installation in relation to such integral experiments.

54 Many calculation methods are 'tuned' to a greater or lesser degree to results from a specific experimental facility. Tuning is the process of recalculating the same test case with adjustments, for example, in input parameters, user options or nodalisation until the best possible agreement is obtained. A calculation method that has been gradually tuned to a succession of slightly differing test cases may show excellent agreement with results from a particular facility. However, its actual predictive capabilities can only be established by calculations for a range of different facilities.

55 Code results are sure to fit with the experiments used as a basis for code tuning. Hence such experiments should be excluded from the set used for code validation.

56 If additional experimental data is required as part of a study, the inspector should consider whether it is appropriate to carry out 'pre-test', 'blind' or 'double-blind' calculations. The delivery of formal definitions of these terms is beyond the scope of this TAG, but brief clarifications are provided below to guide the inspector’s work:



A 'pre-test' calculation is carried out prior to the test being done and has to

assume appropriate initial and boundary conditions.



A 'blind' calculation is usually carried out after the test and will employ initial

and boundary data from the actual test.



A 'double-blind' calculation is a more restricted blind calculation on a facility

for which the user has no prior modelling experience.

Report NS-TAST-GD-042 CM9 Reference – 2020/262106

Page 8 of 22

Office for Nuclear Regulation

Comparison with plant data
57 Tests carried out in full sized plant during commissioning or start-up procedures, as well as operational transients or accidents, can be a useful source of data and should, where practical, be included in the validation report. In general plants are now well instrumented and licensees should be encouraged to retain and analyse data from plant transients.
Comparison with analytical solutions
58 Certain well defined problems may have established analytical or numerical solutions. Also asymptotic analytic solutions may be available for limiting cases. In the areas of structural mechanics and neutron physics for instance, numerical 'benchmark' problems already have a long tradition. The use of numerical benchmark problems will provide information on the mathematical solution ability of the calculation method rather than on the physical modelling and their value may be limited. Nonetheless it is important to ensure that numerical solution errors are small compared with modelling errors and benchmark problems may be a way of establishing bounds on these errors, albeit for limited types of problems. A numerical benchmark problem requires:
a) the model equations to represent a well-posed mathematical problem with an unique solution;
b) every term in the equations to be defined and written down explicitly;
c) the initial and boundary conditions to be defined explicitly.
Comparison with other calculation methods
59 In addition to comparing the calculations with experiments, useful information can be obtained by comparing one calculation method against another. The comparison calculation method should have been developed independently of that used in the safety case and should be sufficiently different from it in either numerical methods or physical modelling to make the comparison worthwhile. Clearly, comparison with a calculation method which is a derivative of or very similar to that used in the safety case would not necessarily yield useful results.
60 The calculation method used for comparison will also need a statement about its validation, and the inspector should consider whether it has been subject to appropriate testing and practical application.
Uncertainty of best estimate calculations
61 A best-estimate calculation employs modelling that attempts to describe realistically the physical processes occurring in the plant. The modelling should provide a realistic calculation of any particular phenomenon to a degree of accuracy compatible with the current state of knowledge of that phenomenon.
62 Deriving the overall uncertainty for a best-estimate calculation method may be a difficult undertaking. Uncertainties are not restricted to the combined effect of uncertainty in input data. This can also come from applying models derived from small scale experiments to the full-sized plant (scaling uncertainties), as well as from the uncertainties associated with the initial and boundary conditions. The overall calculation uncertainty estimation should take all such factors into account.
63 The methodology used to combine the various sources of calculation uncertainty should be described and justified. The bias of any such judgements should be clearly

Report NS-TAST-GD-042 CM9 Reference – 2020/262106

Page 9 of 22

Office for Nuclear Regulation
stated. Justification should be provided for the assumed uncertainty distribution of each parameter which is judged to be of relevance to the overall uncertainty.
Inherent calculations bias
64 Uncertainties in the representation of important physical processes may be such that models of these processes have biases built in the calculation procedure. Any bias in the calculation needs to be quantified based on experimental data and allowed for in the final evaluation of the available safety margin.
5.2.6 QUALITY ASSURANCE
65 Additional to the justification of the modelling process, there is a need to establish that the computer code correctly represents the physical model by ensuring that a systematic approach has been adopted for designing, coding, testing and documenting the computer program. In this respect the American Nuclear Society has produced a useful guide against which the degree of assured quality can be judged, namely, ANSI/ANS-10.4-2008 [2].
66 A computer code should be validated and verified for the particular engineering application, hardware and software configuration used in the safety analysis. The validation report should present details of the hardware on which the code was run and version numbers for the supporting software such as compiler, linker, loader and library routines. Evidence that the hardware and software have been suitably qualified should be provided. The process of change from one software platform to another should be supported by an appropriate set of regression tests.
67 User manuals should be suitable for their purpose and of an appropriate standard: IEC Standard 26512-2011 [3] provides guidance on the content of software and user documentation against which the report can be assessed.
68 Evidence that the software has been produced and maintained to the required standard for the application should be sought. For example, conformance with ISO 9000 series [4, 5 & 6] will indicate that good programming practices have been used.
69 The validation report should demonstrate that the sections of code used in the generation of the results have been adequately tested by sample problems and benchmark calculations (Reference [5] provides guidance on this subject).
70 Evidence should be supplied that adequate procedures are in place to control the production and maintenance of the computer code used in the safety case. In particular, there should be auditable controls over how source code can be amended and new versions issued. Collectively these procedures are known as Configuration Management. Compliance with relevant company, national and international standards, codes of practice or guidelines should be demonstrated.
71 The preparation of input data for the calculations should also follow rigid validation procedures and should be auditable so as to assure their quality. Each item of data should have a clearly defined origin within the plant documentation or else its source should be identified and justified. Details and justification should be given of data embedded in the code. Since it is often impossible to check manually the integrity of all input data, there should be suitable measures within the computer code to identify input data errors and erroneous results.

Report NS-TAST-GD-042 CM9 Reference – 2020/262106

Page 10 of 22