1.1 Purpose

This user's guide describes the Bayesian regression methodology developed by the Canadian Strategic Highway Research Program (C-SHRP) for the Canadian Long Term Pavement Performance (C-LTPP) monitoring program. The purpose of this guide is to fill the need for a single, comprehensive overview of the methodology. It provides the reader with the essential theory of Bayesian regression, a step-by-step guide to applying C-SHRP's 10-step Bayesian regression template, and an example modelling application based on analysis of the C-LTPP experimental data.

The sponsors and participants of the C-SHRP program are the provincial and territorial transportation agencies. The user's guide is aimed principally at staff from these agencies, but it will also be of interest to anyone wishing to apply Bayesian regression to a linear modelling application. It is assumed that the reader is reasonably familiar with statistical concepts and linear regression modelling.

This guide is also available from C-SHRP on CD-ROM. In addition to the written material in this guide, the CD contains complementary references linked to specific sections of the guide. Content includes the final reports from a number of Bayesian modelling projects, notes from training seminars on Bayesian regression, and spreadsheets which demonstrate use of the software.

1.2 Scope of the Guide

Bayesian statistics is typically given a highly theoretical and mathematical treatment in published references. C-SHRP's Bayesian regression software and modelling methodology are somewhat different in that they have been application-oriented from the outset to enable provincial transportation agencies to develop their own Bayesian regression models. This Guide therefore stresses the application of Bayesian regression in the C-SHRP context, specifically using the C-LTPP 10-step model building template.

The 10-step template is essentially a recipe for developing linear models using a Bayesian regression approach. While analysts may inevitably pursue a variation on this approach, the Guide documents a methodology that has been successfully used by C-SHRP. The guide uses the C-LTPP rutting model as an example throughout each of the 10 steps. The model was developed for the C-LTPP modelling project using Bayesian regression and data obtained from the C-LTPP database (C-SHRP 1996).

Chapter 1 provides a background for the guide. Chapters 2, 3 and 4 cover the C-LTPP 10-step model building template. Chapter 2 covers Steps 1 through 5 which focus on design of the linear model. Steps 6 and 7 are covered in Chapter 3 which deals with gathering the data and performing analysis using Bayesian regression software. Lastly, Chapter 4 details Steps 8 through 10 which cover analysis of results and planning subsequent iterations of the model.

Appendix A provides an introduction to Bayes theorem including its history and two examples. In Appendix B, the basic theory for Bayesian regression is provided. While this guide does not focus on the theory of Bayesian regression, it may nonetheless be used as a starting point for those seeking a deeper understanding of the theory of Bayesian Methods.

Chapters 2, 3 and 4 borrow freely and extensively from the working documents and training material produced as part of the C-LTPP modelling project. In particular, The C-SHRP Training Sessions in Bayesian Methods and Software (Vemax Management Inc./ Decision Focus Inc. 1995) (available on the CD-ROM version of the users guide) and The C-LTPP Bayesian Analysis Project Consolidated Working File (Vemax Management 1994) were used.

1.3 Rationale for Using Bayesian Regression in C-SHRP

The Canadian Long-Term Pavement Performance (C-LTPP) project was initiated in 1989 by C-SHRP with the goal of evaluating and improving Canadian practice in the rehabilitation of flexible pavements. C-SHRP plans to monitor the performance of the 65 C-LTPP test sections located across Canada over one life cycle (approximately fifteen years).

During the initial phase (1989-1994) of the C-LTPP experiment, C-SHRP identified the need for early and ongoing analysis of the C-LTPP performance data. While recognizing that a more informed analysis would have to wait until deterioration of the test sections reached an advanced stage, the motivation for early analysis was to advance the point at which agencies would receive benefits from participation in C-LTPP. This would allow agencies to use the knowledge gained from C-LTPP in their decisions as quickly as possible. Hence a comprehensive data analysis project was initiated in 1990 to develop pavement performance prediction models using the first few years of data collected.

A Bayesian regression approach was adopted as opposed to a classical regression approach for a number of reasons. A complete discussion on the rationale for employing the Bayesian approach is contained in the report, Design of Long Term Pavement Monitoring System for The Canadian Strategic Highway Research Program (Nesbitt and Sparks 1990). This report can be accessed in the CD-ROM version of this user's guide. In essence the Bayesian approach allowed the development of prediction models even though the C-LTPP database was premature in that it contained only a few years of performance data. This was achieved by combining, using Bayesian regression, the early performance data with prior experience and knowledge that was available through the professional engineering staff in the provincial agencies. The prior experience ensured that the resulting performance models spanned the service life of the pavements.

1.4 Overview of the Bayesian Regression Approach

In its simplest sense, Bayesian regression is a specialized adaptation of Bayes' Theorem involving the development of multivariate regression models which explicitly consider two disparate sources of information:

• A Priori Information: Information that is known prior to an experiment

• Experimental Data: Information that is derived from an experiment

The interpretation and conclusions drawn from experimental data can be quite different depending on what other (i.e. previous) evidence exists on the subject at hand. However, this difference in interpretation does not simply mean a biasing of results. Interpretation of results using Bayes' theorem is a mathematically consistent way to interpret new evidence/information. Bayes' Theorem is explained in Appendix A.

The evidence/information known prior to collecting new data is known in Bayesian terminology as 'the prior'. The prior used in the C-LTPP modelling project was elicited from experts. The new information was performance data from the C-LTPP experiment database.

The concept of Bayesian statistics originated with Reverend Thomas Bayes (1702-1761) who was born in London and attended the University of Edinburgh. Little is known about where Bayes obtained his knowledge of mathematics, but it is known that he was skilled in geometry, mathematics, and philosophy.

Bayesian statistics can trace its roots to Bayes' Essay Towards Solving a Problem in the Doctrine of Chances. This work involved proof of a special case of what has become known as Bayes' Theorem. This paper was not actually published until after his death, when it was discovered in reviewing his effects (Dale 1991).

While Bayes originated the concept of Bayesian statistics, it was Laplace that developed it into a formal mathematical context. The work of Laplace was significant enough that some have said that Bayesian statistics might also have been named after Laplace. Based on the work of Laplace and others, statistics in general grew into a science with Bayesian methods as the dominant form.

Bayesian methods were predominant in statistics until the late 19th century. At this time the English statistician Fisher developed simpler statistical methods. These methods did not involve the use of prior knowledge and evidence in reaching conclusions. Today Fisher's methods are the dominant form of statistics, classical regression being an application from this school (Hively 1996).

Similar to classical regression, Bayesian regression analysis involves the development of models or predictive relationships between variables. As an example, a predictive equation may take the following linear form.

In this equation, y represents the dependent variable to be predicted from x_i, the independent variables which contribute to y. The problem is to establish quantitative estimates for the coefficients b_j.

The classical statistical approach to the problem is to gather data and estimate the coefficients by regressing y on x to best fit the data. However, there are a number of drawbacks with the classical approach. First, the collection of data can be expensive. In particular, collecting data through the observation of in-service pavements can prove difficult. Pavements change so slowly that nothing substantive may be learned for many years. Furthermore, the estimates of b_j based on small sample sizes may not be meaningful. Finally the classical approach to model development does not incorporate judgement in any way. In practice however, results are sometimes modified to reflect the judgement or experience of the analyst.

The Bayesian statistical approach to model development, represented in Figure 1-1, is to systematically combine prior knowledge and experience with data to improve the predictive relationship. The Bayes approach calculates a meaningful and credible answer without relying solely on the small sample size database. In doing so, the Bayes technique allows decisions to be made in the short term while improvements to the data, judgement, and the model continue to be made.

The Bayes solution achieves a balance between two solutions based on data or judgement alone. This balance calculation is mathematically rigorous, based upon the theorem first developed by Bayes. Since the 1950s, the Bayes concept has been refined for regression analysis by mathematicians including Raiffa, Schlaiffer, Press, Pratt and Zellner.

In assembling information for Bayesian regression, data collected in the traditional manner is supplemented with prior knowledge. This approach is summarized Figure 1-1 (after C-SHRP Training Notes). The so-called prior may be drawn from expert judgement, "old" data sets, or knowledge that is generally accepted in the field. Expert judgement can also be encoded by polling experts and asking them to estimate the value of the dependent variable for a combination of contributory variables. Once collected, the experts' 'observations' are interpreted similar to traditional data.

The larger view of Bayesian regression is summarized in Figure 1-2 (revised, after Smith et al, 1979). This flowchart shows Bayesian regression as a continuous process of updating the existing 'partial state of knowledge' with new data. This yields a posterior which is another partial state of knowledge. With a process similar to this in mind, the combination of prior and data is also sometimes known as a Bayesian update.

A Bayesian regression analysis based on the 10-step methodology includes a detailed analysis of the prior, a classical regression of the data, and a Bayesian regression which combines the two. The conclusions that would be reached based on data alone are always known and consideration of the validity and consistency of the prior is part of the analysis process. The effect of the prior is also known and can be readily described and evaluated.

The assessment of the prior view and formal comparison of this with the classical regression result is usually at least as important as the final Bayesian regression result. As it has evolved in C-LTPP, Bayesian regression is a formalized comparison tool at least as much as it is a formalized combining tool. The process of determining the prior and assessing its impact amounts to a built-in review of the experimental results for validity. Large discrepancies between the prior and the data prompt the analyst to question, for example, whether significant contributory variables have been excluded or whether the experimental sample is as representative of the broader case as it originally appeared. Where the prior and the data are consistent, there is an increased certainty in the results. As a result, the conclusions and models that arise from the Bayesian regression process are well reasoned and can be used with increased confidence.

The Bayesian regression theory and software used in the C-LTPP analysis project is based in large part on the work of Professor Arnold Zellner of the University of Chicago. Professor Zellner was a consultant to the contractors who developed the C-LTPP methodology. Zellner's work on Bayesian regression has been applied mainly in the field of economics and includes a textbook on the subject: An Introduction to Bayesian Inference in Economics (Zellner 1987).

1.5 Bayesian Regression Software

One of the problems faced in the C-SHRP Bayesian modelling project was the lack of a 'user friendly' Bayesian regression software package. To this end, two Bayesian regression software packages were developed by C-SHRP, B-STAT and XLBayes.

B-STAT provides an EXCEL spreadsheet interface to a FORTRAN based Bayesian regression program, PC-BRAP, which was developed under the direction of Professor Zellner (Abowd et al. 1985). XLBayes is a much faster Bayesian regression program based entirely in EXCEL. The analysis features and numerical results of the two programs are identical.

Both software packages were provided to all C-SHRP sponsors. Other organizations can purchase the software through VEMAX Management.

The mechanics of running BSTAT and XLBayes are relatively straightforward and operation is demonstrated in this user's guide with numerical examples. A guide to using XLBayes is provided in Chapter 3 (operation of BSTAT is essentially the same as XLBayes). Example spreadsheets are also provided on the CD-ROM version of the user's guide.

1.6 Overview of the Bayesian Regression Template

A 10-step template was developed to provide a step-by-step procedure for developing Bayesian regression models. The template is based on C-SHRP's Bayesian modelling experience and is designed to ensure that key issues and questions are considered at each stage in the modelling process. The fundamental strategy outlined in the 10-step template is similar to the procedure which would be followed in a classical regression analysis. However, certain tasks are unique to Bayesian regression. The template contains the following steps:

Step 1: Decide What You Want to Model

The first step in the process is to ask some key questions about the purpose of the model that is being sought. What will the regression model be used for? To what situations (i.e. inference space) is the model desired to be applicable in? What experimental data is available or can be readily obtained for the model. What other information exists in addition to the experimental data (i.e. what potential sources exist for the Bayesian prior)?

Step 2: Select a Dependent Variable

With the purpose of the model identified, the next step is to identify the dependent variable in more specific terms. What variable should the model predict to meet its objectives? What are the units of measurement for the dependent variable? What procedure will be used to measure the dependent variable? In specific terms, what inference space is sought? How far into the future are forecasts required?

Step 3: Select Model Type

Mechanistic model forms and cluster variables often offer advantages over simple empirical forms. Does sufficient theory exist to create an empirical-mechanistic model for the forecasts that are being sought? What 'cluster terms' might be useful to incorporate as independent variables? What empirical relationships have been found by other researchers? What types of models have been used by others?

Step 4: Select Independent Variables

Numerous variables may influence the performance of the dependent variable and there may be a large number of candidate independent variables for the regression model. A smaller number of independent variables is usually more desirable for the model and the list of candidate variables often needs to be pared down. Different methods for selection of independent variables are discussed in the template including a subjective assessment process based on ranking, evaluation of correlation coefficients and the use of scatter diagrams.

Step 5: Postulate Functional Form

After considering model type and choosing independent variables, the precise functional form of the model can be chosen. Different potential functional forms for linear regression models need to be considered ranging from simple linear forms to those involving log transformations and cluster terms.

Step 6: Develop Prior and Assemble Data

A number of methods exist for developing regression model priors. Non-subjective methods considered in the template include deriving priors from old databases and from existing models. Subjective methods are based on interviews with experts on the performance model in question. Three different methods for developing a table of 'pseudo-data' are discussed. These methods are based on the experts' estimates of dependent variable performance for different model inputs (i.e. values of the independent variables). A method based on the sorting of a deck of cards in order of expected performance also exists. The final form of the prior is a regression model with associated statistics. The experimental data is also put into a suitable form for regression.

Step 7: Perform Bayesian Regression

The Bayesian regression software is used to perform a classical regression on the experimental data and a Bayesian regression using both prior and experimental data inputs. A comparison of the prior, classical and Bayesian regression results is output by the software. The guide provides an overview of using the software and example spreadsheets are provided on the CD-ROM version of this guide.

Step 8: Use Model to Predict Performance

Where prior information exists from several different sources, such as several different experts, it is useful to evaluate the consistency between these sources. Techniques used in the template are a direct comparison of prior model coefficients and a comparison of the sensitivity of outputs from the various prior models to changes in input. Other comparisons which may be relevant for particular models are also discussed. The goal of this step is to arrive at a single prior model that represents a consensus of the various priors.

Step 9: Evaluating Model

A series of comparisons are made between the consensus prior, the classical regression result and the Bayesian posterior. This includes evaluation of model coefficient probability density function plots, coefficient sign, coefficient magnitude and the t-statistic. The tendency of the posterior to rely more heavily on the prior or the data is also evaluated as an indicator of how conclusive the experimental data is compared to the prior evidence.

Step 10: Iterate Model

Additional model iterations may be desirable. These iterations may be based on changes to the model form, collection of additional experimental data, and/or collection of additional prior information.

References

Abowd, J.M., Moulton, B.R. and Zellner, A, User's Guide to PC-BRAP, H.G.B Alexander Research Foundation, Graduate School of Business, University of Chicago, 1984.

Vemax Management Inc., C-LTPP Bayesian Analysis Project - Consolidated Working File, Canadian Strategic Highway Research Program, Ottawa, 1994.

Dale, Andrew I., A History of Inverse Probability - From Thomas Bayes to Karl Pearson, Springer-Verlag, New York, 1991.

Hively, Will, The Mathematics of Making Up Your Mind, Discover Magazine Vol.17-No.5, Disney Publishing, New York, May 1996.

Nesbitt D. and Sparks G., Design of a Long Term Pavement Monitoring System for the Canadian Strategic Highway Research Program, Canadian Strategic Highway Research Program, Ottawa, 1990.

Smith, W., Finn, F., Kulkarni, R., Saraf, C and Nair, K, Bayesian Methodology for Making Recommendations to Minimize Asphalt Pavement Distress, NCHRP Report #213, Transportation Research Board, Washington, 1979.

Vemax Management Inc. & Decision Focus Inc., Training Sessions in Bayesian Methods and Software (Training Notes), Canadian Strategic Highway Research Program, Ottawa, 1995.

Zellner, A, An Introduction to Bayesian Inference in Econometrics, Robert E. Krieger Publishing Co., Malabar, Florida, 1987.

Continue to Chapter Two

Return to Table of Contents