Improving the process for public transport patronage forecasting

Gary McGregor¹ and Tim Raimond^{2 1} Rail Development, RailCorp NSW

²Transport and Population Data Centre, NSW Dept of Infrastructure, Planning and Natural Resources
 

1           Introduction

Public transport demand forecasting is a challenging and controversial process and both Australian and international forecasts have frequently been criticised for overestimating likely demand. It is important that organisations managing these forecasts develop sound processes to provide assurance to senior management, boards and key external stakeholders that the risks of inaccurate or inappropriate forecasts are being properly managed.

In this paper, our goal was to develop recommendations for modelling practitioners and clients at both a strategic and project level to assist in the management of forecasting risks. To reach this goal we undertook a risk assessment of the patronage forecasting process for public transport considering:

·          the key stakeholders;

·          the objectives of a good public transport forecast;

·          the risks to achieving these objectives; and

·          management of these risks

Our methodology involved a literature review and in-depth discussions with 17 stakeholders in the NSW forecasting environment.
 

2           The study problem

The recent Australian experience of public transport patronage forecasts is of over- forecasting of patronage. A brief review of publicly available information on public transport infrastructure projects completed since 2000 (Table 1) shows only one out of four projects not significantly over - forecasted.

Table 1: Public Domain Forecast Accuracy of Public Transport Projects – Post 2000

 

Project                                       Forecast Accuracy               Source

Airport Rail Link (Sydney)            30% of forecast in 2003        Vince Graham, RailCorp CEO to

Budget Estimates Committee 3 Sept 2003 (4)

Brisbane Airtrain                         Not publicly available although project promoter states “it is no secret that patronage has fallen well short of projected”

 

Vince Scully in Sydney Morning Herald, June 24 (2004) in Mac Bank - Fat and Hungrier than Ever

Liverpool Parramatta Transitway

(Sydney)

 

South East Busway (Brisbane)

 

22% of forecast patronage in 2003

    Not publicly available although said to be “well in excess of expectations during the first year”

 

Hon Michael Costa, Minister for Transport, to Budget Estimates Committee, 3 Sept 2003 (Hansard

2003)

Hon, Steve Bredhauer, Minister for Transport and Main Roads, Press Release 29 April 2002

(1)      Only major new network additions analysed. New stations or fleet not considered.

(2)       Perth Clarkson extension opened in late 2004, but is considered too early in operation to draw conclusions

(3)      Authors unaware of any publicly available information on Sydney Inner West Light Rail Extension. 

(4)      Hansard Transcripts (2003), General Purpose Standing Committee No.4. 3 Sept 2003 

International literature confirms that the perceived over-forecasting trend in Australia is by no means unique. The main stream of investigation of forecast errors in patronage in recent years has been through Danish academic Bent Flyvbjerg (2005, 2003, 1996 with Skamris). In 2005, Flyvbjerg, Skamris and Buhl published a comprehensive review of patronage forecasts for 210 projects in 14 countries across 5 continents. The conclusions from this study, which were consistent with earlier work by Flyvbjerg (1996), were that:

 

·          on average, the actual patronage on the sample rail projects was 40% lower than forecast;

·          at the 95% confidence interval, the patronage was between 19% to 60% lower than forecast; and

·          road automobile patronage was underestimated by an average of nearly 9%, although nearly 50% of road forecasts are different to that observed by more than +/-20%.

 From these results, Flyvbjerg et al (2005) concluded that simple uncertainty could account for the type of uncertainty found with road, but not rail, forecasts. A significant limitation of Flyvbjerg’s work is that it wa focused purely on the first year of operation, and so part of the effect may be ramp up error. In addition, many of the rail projects in the US were based on the work of Pickrell (1992) and may in part relate to the particular funding situation in the US in the 1980s which encouraged production of higher forecasts in order to obtain funding.

 

Brinkman (2003) undertook an extensive literature review on forecasting error as part of a dissertation on ethics in forecasting and argues that apart from Pickrell and Flyvbjerg¹, few studies provide any robust and comparative information on forecasting error, and that no peer reviewed work would seem to contradict their findings.

 

Whilst the Australian experience deserves greater research, perhaps with detailed case studies, the authors believed there to be enough evidence of a least of perception of over - forecasting to warrant an investigation into the risks in the forecasting process and for some suggestions to reduce these risks

 

3           Study approach

 

 

3.1          Study process

 

This study is concerned with managing the risk that a patronage forecast will not meet stakeholder objectives. In considering this aspect of risk management, we follow a process broadly consistent with AS/NZS 4360:2004: Risk Management Guidelines, as illustrated in Figure 1.

 

As an initial scoping exercise, we did not undertake a detailed risk analysis. Our approach was to ask stakeholders what they considered major risks and to highlight consensus or divergence rather than to quantify likelihood and probability. We then explored the options available to manage the major risks that were identified consistently by stakeholders or the literature.

 

 

 

 

 

 

  

¹ Although published before Flyvbjerg (2005), Brinkman had access to Flyvbjerg et al’s unpublished work

  

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Figure 1: Study Process (modified from approach in AS/NZS 4360:2004)

 

3.2          Stakeholder interviews

 

In determining appropriate stakeholders to interview, our own experiences and a review of literature (eg Flyvbjerg et al 2003) identified six broad groups of stakeholders that should be covered. Figure 2 shows the different groups interviewed, the number from each group, and in recognition that some stakeholders had multiple roles over the past 3 years some stakeholders were classed as belonging to two stakeholder groups. Accordingly, we had more stakeholder roles than interviews.

 

 

 

  

 

 

 

 

 

Figure 2: Stakeholder Groups and Interviews

 

Our approach to interviewing stakeholders from these groups was to devise a “pragmatic sample” where the majority of stakeholders interviewed were people who the authors were in contact with on a professional basis at the time of writing.

 

Data collection was by 17 semi-structured interviews, each approximately one hour long. Interviews were guided by a common question framework broadly covering:

·         role in relation to patronage forecasting;

·         objectives of patronage forecasts;

·         top risks to achieving objectives; and

·         methods to mitigate risks .

 

These in-depth stakeholder interviews cover a very substantial sub-set of the industry in NSW. As a pragmatic sample, however, the approach has several potential limitations. The most significant limitation being over-representation of government employees or contractors and potentially narrower and conforming views.

 

With these limitations, we consider the stakeholder interviews to be an important guide to inform and give context to risk management, rather than a clearly representative sample of the views of all stakeholders in patronage forecasting in NSW.

 

4             Objectives of patronage modelling

The first question in considering patronage risk is understanding the objectives of forecasts. The key patronage modelling objectives identified by stakeholders were:

 

·         getting it right/getting the most accurate result;

·         robust, transparent, defensible, credible modelling process;

·         flexible model allowing what if analysis/multiple scenario testing/testing of alternative options; and

·         understanding of travel behaviour and risk

 

Stakeholders diverged in the level of importance ascribed to these objectives. The practitioners tended to focus on the objective of accuracy. Academics and clients of modelling were more likely to express the view that there was no right answer and that the robustness and defensibility of results was more important.
 

This discussion on objectives has highlighted a number of important considerations in establishing a context for managing risks in patronage forecasting. Firstly, there is more to forecasting than simply getting the “right number”, with sound process and flexibility also key considerations. Secondly, there is debate as to whether the “right number” is indeed a meaningful objective.
 

5           Risks to meeting objectives of patronage modelling

The risks identified by all stakeholders can be broadly classified into five groups, as outlined in Figure 3, which shows the risks to patronage modelling in a cascading format. The hierarchy, starting with project bias, does not necessarily imply that bias is the most probable risk, but shows that it can influence all aspects of the project, from the way it is conceived, to the way the models are specified.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

  

 

Figure 3: Key patronage modelling risks

 

5.1          Forecasting bias

 

Bias in forecasting can simply be defined as a systematic deviation from an optimal forecast (Harvey 2001). Flyvbjerg (2005) argues that such bias is evident in rail travel forecasts as they are overestimated too consistently for an interpretation in terms of simple uncertainty to be statistically plausible. Almost all stakeholders interviewed for this paper also indicated that forecasts for public transport were more likely to be overestimated than underestimated.

 

 

From our review of literature and stakeholder discussions, three potential forms of bias were identified. Firstly, there is strategic misrepresentation which is a bias best described by Wachs (1990 p143) who argues that “forecasts are presented to the public as the result of unbiased scientific procedures yet they are in reality often highly subjective exercises in advocacy”. Attention in the literature has primarily focused on the agency/ client demanding the advocacy forecast. Brinkman (2003), however, also argues that where engineering firms undertake patronage forecasting for a project and also the potentially more lucrative engineering work on that transport project, there is a financial incentive to provide optimistic forecasts.

 

A second category is judgement bias which relates to a range of forms of bias that are not deliberate but are the result of limitations in human abilities to make judgement. Makridakis (1995) suggests that these limitations can manifested via a practitioner’s predisposition to remember information that confirms their beliefs better than information that disproves beliefs. In this regard, Lave (cited in Brinkman 2003 p37) warns of the transport planners who: “envisage a better environment in which increased transit use could solve many of our urban problems … but they are so certain about how people ought to commute that they have talked themselves into believing it is possible to make them behave that way”.

 

Finally, there is methodological bias where modelling practice has tended to ignore or coarsely model some of the complex to model variables such as the full door to door multi- purpose trip or reliability and comfort. It is possible that not modelling these variables does lead to a bias to over - forecasting of public transport versus car usage, particularly in new modes where the issue cannot be addressed through base year calibration. Whilst theoretically this should be overcome over time with experience, it may take some time, particularly for new modes

 

Stakeholders more frequently cited examples of judgement or methodological bias although the interview format may not have been conducive to discussion of strategic misrepresentation. Irrespective of the cause of the bias, there is consensus that it exists with the symptom perhaps best encapsulated in one stakeholder’s view that clients are frequently suffering from “project fever”.

 

5.2          Project definition

 

There is a significant risk that the transport project delivered is very different from the project specified for forecasting and that these differences are directly relevant to patronage outcomes. One stakeholder suggested that a reason for the low patronage for the Liverpool Parramatta transitway was that modelling was undertaken under the assumption that transitway services were integrated into local feeder bus operations, whereas in reality they are less attractive as they operate only as trunk transitway services.

 

There are a range of reasons why this risk may emerge. One factor identified by a number of client and central government stakeholders is that too much detailed modelling is attempted early in project development rather than well developed strategic followed by operational studies. Another factor is that without prior planning, it is often difficult in public sector procurement systems to quickly ramp up modelling efforts when project scope is changing.

 

A related issue is that the modelling may not be set up to produce the outputs that are relevant for economic, financial and environmental assessment. One practitioner argued that public transport forecasts have generally focused on spatial aspects (eg: trip distribution) when the decision making criteria have been temporal (eg: will the peak spread and the associated implications for rail service crowding).

 

Practitioners also stressed a lack of time and budget to undertake quality modelling . In many cases of poor specification of resource requirements. the ultimate budget and timeframe may end up being appropriate or even in excess of true requirements, but are countered by unproductive negotiations on variations on scope, budget and time.

 

 

5.3          Market definition / segmentation

 

A challenge in all patronage studies is defining the likely markets for the public transport service, both geographically and socio-demographically. This is especially difficult for projects conceived to serve markets which are developing or likely to change significantly before the transport project is delivered. Current examples include the proposed rail links to new release areas in North Western and South Western Sydney, where several hundred thousand new households may be established before the rail links are operational. There is considerable debate about the mix of households likely to choose to live in these areas and whether early commitment to a transit project would in itself alter these characteristics.

 

Another issue related to market definition is that population and employment projections are seen by almost all stakeholders as critical risks. Several stakeholders suggested that one of the likely reasons Sydney’s Airport Rail Link has not reached projected patronage is because the project was conceived based on major redevelopment around Green Square and other sites in the corridor, but many of these proposed developments have yet to be built.

 

5.4          Market research

 

Aside from market definition, failure to collect the right data, and enough data, from the right market was probably the second most frequently discussed risk by stakeholders. In particular, key variables likely to affect people’ s decisions about using public transport were often excluded from patronage models in favour of variables which best explained the base year situation. Many questioned whether this was appropriate, particularly for new modes to a particular market which may not have been experienced by that market. They also questioned whether excluding variables which may influence choice, such as crowding, reliability, multi-purpose trip requirements and even respondent personality type may ultimately be leading to systematic over-estimation of public transport patronage by traditional models. This is in part a manifestation of “project fever”, where a project is conceived in a perfect world, but is actually delivered in the real world.

 

A further point, suggested by two internationally experienced modelling practitioners as well as private sector clients, was the “paltry” level of investment in project specific data collection by the public sector. While the public sector in NSW was acknowledged to be well served with strategic level data, project specific revealed and stated preference data collection was seen to be poorly resourced in relation to international experience, which was that up to 5% of project development costs were invested in understanding the market for the project. That percentage is well below 1% in NSW , based on the authors’ and stakeholders’ experiences.

 

5.5          Fitness for purpose of models

 

Modelling practitioners and their clients both raised concerns about the appropriateness of many current modelling approaches for public transport modelling. Most strategic models used in the public and private sector in Australia began their life as traffic models. The public transport modelling capabilities of many of these models have been largely developed as add ons, and are for the most part crude. Important issues such as the full door to door trip, crowding and reliability are generally not modelled. There was also a criticism that strategic models appeared over - utilised for project specific investigations. A risk is that models being used to evaluate public transport project patronage are simply not fit for purpose.

 

There was also the possibly contradictory criticism that too much effort is going into complex model development and into procedural aspects of modelling with not enough resourcing in the areas of risk assessment, communication of meaning and sanity testing. The response of model practitioners to the perception that their patronage forecasts have historically not been very accurate has been to attempt to model more and more aspects of travel behaviour, leading to increased model complexity, and reduced accessibility to non-practitioners.

 

Clients of patronage forecasting noted that their objective was for a transparent, robust modelling approach that allowed the testing of multiple scenarios. This is not something many clients felt they were getting, and this was a risk to them being able to make an effective business case for projects, and a case which adequately evaluated all the risk factors for a project. There was a view that even when multiple scenarios were forecast, numbers were coming out of black box without context rather than informing the client about where risks lie.

 

5.6          Modelling methods – skills in modelling

 

Many stakeholders noted a lack of public transport modelling skills in Australia. A factor influencing the skill levels and skills pool is the relative rarity of major public transport projects in comparison with road, and particularly toll road, modelling. The private sector in NSW is particularly adept at toll road modelling not only because of the number of projects over recent years, but also because these projects have involved private sector bids, therefore many modelling teams have been working for different construction and finance companies supporting toll road bids.

 

The lack of private sector involvement in public transport projects, coupled with the relative scarcity of new public transport projects, means that there are few consultancies with considerable experience in public transport forecasting. In addition,                           there is a well documented dearth of transport modelling skills in Australia (Taylor et al, 2004), and the existing batch of modellers is ageing. Coupled with this lack of modellers is the longer term issue that several university-based transport programs are not attracting quality students and are in danger of changing focus.

 

5.7          Communication

 

Clients almost universally criticised modellers for their inability to communicate results in a clear and concise fashion. No matter how good the modelling was, a risk was that clients were unable to understand the modelling results or present them to decision makers in a way that could help facilitate a decision.

 

Two particular themes emerged in communication. Firstly, potentially too much energy was devoted to the detailed modelling process when clients wanted “sanity checks” and benchmarking of the forecast results against comparable situation. While clients were generally getting very complex technical reports on model calibration and validation they wanted more emphasis on whether the forecasts pass a test of reasonableness compared to comparable situations. This situation is similar to that called for by Flyvbjerg, Skamris, Buhl (2005) in recommending “reference class forecasting”.

 

A second communication problem was that the forecast results were often presented as aggregate numbers which communicated little about travel behaviour or risk. Figures such as total new transit trips or traffic through cordons may be important for some aspects of project evaluation but gave limited insight into the underlying sensibility or robustness of the forecasts.

 

6           Risk mitigation at the strategic level

Now that the key areas of risk have been identified, it is important to turn attention to mitigation. There is a natural separation in risk mitigation between actions that can be implemented in an individual project by a study project manager, and actions that would require significant strategic thinking above the level of individual projects. Risk mitigation is firstly considered at the strategic level and then at a project level as summarised in Figure 4.

 

 

R isk category

 

S trategic m easures

 

P roject m easures

 

 

 

  

 

  

 

  

 

 

 

 

 

 

Figure 4: Strategic and Project Risk Mitigation Measures

 

 

6.1          Strategic framework for project appraisal

 

Most stakeholders agreed that unclear project objectives were often a source of risk, particularly in project definition. Clear objectives mean the right product for the right market can be determined and evaluated in a systematic way. The best way to achieve this to have a strategic framework which enables consistency in the evaluation of transport projects. This framework may be in the form of a transport and land use strategy, outlining the objectives of the government of the day, where the priorities lie, and what evaluation criteria are important. Within such a framework, it is then possible to develop an approach, such as the concept of guidelines discussed in the next section, to ensure consistency, comparability and ability to measure projects against each other and community/government objectives.

 

6.2          Guidelines for data, modelling and communication

 

Several countries, including New Zealand and the United Kingdom, use guidelines to assist practitioners in developing patronage forecasts. These guidelines include parameter value ranges and modelling approaches for different situations. In NSW we are quite familiar with this approach for roads. The Roads and Traffic Authority produces a Guide to Traffic Generating Developments, and an Economic Analysis Manual to guide assessment of road needs. Victoria has recently attempted to move a step further by developing guidelines that discuss appropriate sensitivity tests and measures of model quality/accuracy (Vicroads 2005).

 

Many stakeholders thought a set of guidelines covering public transport modelling had merit, especially because of the relative infrequency of work in this area and resulting inexperience of modellers. Guidelines could potentially provide guidance in the following areas:

·         current best practice approaches

·         appropriate approaches for projects of different scales or different stages of conception

·         important inclusions for modelling briefs

·         methods for using peer review approaches

·         standards for documentation of approach and assumptions

·         examples of clear communication techniques for modelling results

·         advice on open model architecture, model flexibility, and sensitivity testing

·         a library of standard model parameters and values; and

·         risk management approaches.

 

Concerns were raised about aspects of guidelines including stifling innovation and becoming out of date. However, we believe that carefully implemented and regularly updated guidelines could overcome these concerns. Stakeholders emphasised that it is important that guidelines be endorsed by industry and allow for departure for soundly justified reasons.

 

 

6.3          Common data

 

In order to ensure consistency in the assessment of projects both within the transport portfolio and across government, it is important that consistent data inputs are used across projects. In NSW we are relatively lucky in having a central source of demographic data and forecasts, and a central source for strategic travel data and public transport networks.

Nationally, there are moves towards a National Transport Data Framework which is intended to achieve greater consistency of data across Australia (National Data Network 2004) .

 

There are, however, basic data needs which are not currently centrally stored or accepted. Reasonable Australian parameter value ranges, values of time, and elasticity values are not reported in any standard way accessible by modelling practitioners. As mentioned previously, these ranges could be published and regularly updated in some form of modelling guidelines.

 

Most stakeholders strongly supported the concept of expanding existing common data sources, perhaps supplied in the form of guidelines. The main dissenting view was from private sector clients. They wished to have full control of model inputs and modelling approaches to minimise the risk of relying on sources of data of unknown quality and to potentially add advantage to their bids through better intellectual property in market research and modelling.

 

6.4          Common modelling platform/approach

 

As already discussed, NSW does not have enough new public transport projects to make it worthwhile for transport consultants to develop and maintain their own quality multi-modal model, or the in-house skills to develop such a model. Exactly the opposite is the case for toll road modelling where there is considerable Australian experience and expertise, and this expertise is exported on occasion.

 

One way of addressing the issue for public transport modelling is to pool the modelling and skills in some way. A way to do this is for government to require all models to be developed on a single platform, with all model and network improvements made to this common model.

 

Stakeholders were split in their support for this idea. While several thought specifying a common modelling platform had the potential to stifle innovation, many, especially government clients, felt that there were advantages in terms of being able to validate models, to use models in the future to assess alternative scenarios, and to enable each project to improve the common model. There was a considerable variation in opinion, even among private sector practitioners, some of whom were seeking guidance while others considered that maintaining individual firm intellectual property would lead to more advances in modelling practice.

 

One step most stakeholders supported was the concept of specifying in study briefs that consultants be asked to deliver a fully working and documented model, along with training as appropriate, such that their client can understand the model, validate it, and run alternative scenarios as required. The authors have had very positive experiences with this approach, being able to find and correct model errors and quickly run different scenarios as the project evolves.

 

However, for this approach to be truly effective, the client needs to be familiar with the modelling approach and platform developed by the consultant. Even with improving compatibility between models, there are such complexities in the way models deal with different issues that the only way to truly replicate results is on the same platform.

Accordingly, it would be attractive for government to specify in briefs the modelling platform to be used. It is a matter of weighing up the benefits of a common platform with the potential costs of stifling innovation, restricting the market to those with skills in the particular modelling platform and needing to select a single platform which may not always be the most appropriate for all modelling tasks.

 

A solution with some potential may be collaboration between government and the private sector on patronage modelling projects. The government may use its own modelling platform, but work with consultants on project specific data collection, model estimation, application and presentation of results. The consultants could provide their advice, but the model itself would be implemented by government on a government platform. This solution raises a dilemma in that some stakeholders, including central government agencies, saw the independence offered by consultant forecasts as desirable in addressing bias risk.

 

7           Managing Risks at the Project Level

 

 

7.1          Independent and dissenting advice

 

Harvey (2001) suggests that different people (or possibly organisations) be responsible for project planning and for forecasting to reduce judgement bias. Stakeholders also generally considered that forecasts will be more credible when prepared by “reputable “consultants as opposed to in the agency or by the consultants acting as contractors to the agency.  An

 

important caveat should be that the patronage consultancy not stand to benefit from other large engineering contracts that are contingent on the transport project proceeding.

 

Makridakis (1995) argues that the key to avoiding judgement bias is setting up procedures that encourage the search for disconfirming information and allow for devil’s advocate roles. A number of clients and practitioners (some who had been peer reviewers) considered that peer review was much more appealing in theory than in practice in addressing this issue. The key problem was the small number of suitably qualified parties and the motivations of the parties. It was suggested that key motivators are often “point scoring” against the lead consultant or being unwilling to “rock the boat” due to the influence of the lead consultant.

 

Several stakeholders suggested the early involvement of a stakeholder with different perspectives would ultimately assist with the objective identified by many stakeholders of ensuring a more credible, robust and transparent process. Organisations such as Treasury may bring a more risk adverse approach to counter the potential optimism of the transport agency and may be useful in the role of challenging agency beliefs. Accordingly, it was considered that the stakeholder’s perspective was more important than their technical knowledge in fulfilling a devil’s advocate role.

 

7.2          Separate strategic and detailed feasibility studies

 

 

A number of clients highlighted the problems of bundling up too much modelling effort while the project is still relatively undeveloped, leading to model and network specifications not reflecting the developing project. This is best managed by a tiered approach of a first stage strategic level study which may facilitate a political decision on a project, before embarking on the more robust work necessary for economic and environmental appraisal.

 

 

7.3          Early technical and project management assistance.

 

In many cases, it may be more important for technical assistance to be given before the issuing of briefs than in peer review of modelling work. A particular issue is ensuring that the outputs specified in the study brief actually meet the needs of downstream users including the economist, environmental impact assessor and operational planner.

 

A range of clients also identified skill gaps in managing large patronage studies and understanding the true cost and timing as well as technical issues. In this situation, there is a strong argument for the engagement of a skilled project manager in this field as the starting point in the forecasting process. As noted previously, however, there is a skill shortage in this area, at least if understanding of demand modelling is a pre-requisite for this role.

 

 

7.4          Flexibility to update model

 

In many cases there will be circumstances that lead to significant project scope change and the need for rapid updating of forecasts. A first step is acknowledging in work programs that forecasting is dynamic and will not stop once a particular report is issued. Following from this recognition, a number of steps should be undertaken including:

 

·         requiring (and budgeting dollars and time) for a sound audit trail and documentation in model development;

·         programming and budgeting for regular updates of the model; and

 

·         specifying in study briefs that consultants be asked to deliver a fully working and documented model in a platform capable of being modified by the agency, enabling modelling, audits or running of additional options.

 

Whilst acknowledging the potential of consultant proprietary models, this lends support to the issues raised in section 6 of enabling the client to use and modify the model. This is particularly relevant in the public sector where re-tendering timeframes may be incompatible with the rapid evolution of the project.

 

7.5          Improving market research

 

With respect to project specific market research, it was suggested that public transport projects generally have a small number of key markets where travel change is likely. These markets could be geographic (eg: travel to destination in the Sydney CBD) or relate to the characteristics of the user (eg: worker in single car household). These core markets should be carefully targeted and sampled rather than using more ‘broad brush’ market research. Market research should also assess how important “soft” issues are such as reliability, crowding and safety perceptions, et al.

 

7.6          Conservative assumptions

 

Pickrell (1992) in a critique of US public transport forecasts argued that assumptions regarding land use and the performance of alternative modes have generally been too bullish and forecasts should be undertaken on the basis of the status quo.

 

The appropriate use of optimistic versus conservative input assumptions is dependent on the use of forecasts and whether they are for design or project evaluation. However, we believe the downside risk of forecast error could be reduced if the patronage assuming no significant land use change (or conservative assumptions for release areas) was explicitly reported and generally used as a central case. This would avoid a situation like on the Sydney Airport Rail Line where much of the under - forecasting of, was a result of the extent of inner Sydney land use change envisaged but unrealised 4 years after project opening. Challenging optimistic assumptions should be a key role of the “devil’s advocate” referred to in section7.1.

 

7.7          Model complexity

 

It is clear that modelling briefs should require sound representation of key aspects of public transport service quality. In many situations, however, more complex network, behavioural or land use representations may not assist in ultimate decision making. In cases such as Sydney’s South West Rail Link, we would question the importance of detailed representations of land use, transport networks or new market research, given the uncertainty in many of these factors in what is still a greenfields site. In this situation resource effort may be better focused on scenario testing and the issue of communicating the respective implications to stakeholders.

 

7.8          Understanding of results
 

True meaning is often lost at aggregate levels such as annual trips and total system usage. Reports should accordingly provide information about what market groups are benefiting and their size. There is also a need for sensibility checks that the output from the model passes a common sense test of reasonableness. A number of practitioners considered that the public sector was much more focused on demonstrating good modelling process than on the sanity testing of the output and could learn from private sector clients in this regard.

Study briefs should require results to be presented and benchmarked against comparable projects. This approach should also be applied at the acceptance of input parameters stage, with benchmarking against comparable markets.

 7.9          Risk analysis 

The general view of practitioners was that public sector modelling projects placed a much greater percentage of resource effort into model methodology and less into understanding risk than private sector projects. More emphasis is required on understanding risk in the uncertainty of input variables. A number of client and practitioner stakeholders considered that greater sophistication in risk analysis should be required through risk analysis techniques such as computer simulations (eg refer to Patrick et al 2004 for a greater exploration of this issue).

 

In many projects agencies may be expending too much effort in formulating and achieving signoff on a definitive land use forecast or future transport network specification, when the resource effort is better placed into understanding uncertainty in these variables and how they influence results. This requires communication at all levels of the agency and government to deal with what is generally the client and approval stakeholder and public expectation of a forecast producing a single best estimate number for conditions when the project is implemented.

Conclusions

Stakeholders were unified in their concern regarding the potential for public transport forecasts to be overestimated, at least in the short term. There was also a high degree of commonality in the nominated sources of risk. In many instances, however, there was divergence on both the objectives of forecasts and how to manage the forecasting risk.

 

The key area of agreement was on the development and use of guidelines to clarify and improve the process of patronage forecasting for major public transport projects The caveats suggested were that the guidelines must be updated, be accepted across the broad spectrum of practitioners, allow for well considered divergence of views and not stifle advances in practice.

 

There was a sharp polarisation of views on the issue of brief requirements for a common modelling platform. Stifling innovation, reducing the pool of modelling teams and possibly having an inferior platform for a particular job were seen as the negatives with auditability, greater flexibility in updating forecasts and pooling investment in improvements seen as the benefits. We believe that further consideration should be given to this approach as recent Australian history suggests that public transport forecasting models have commonly been relatively unsophisticated add ons to road based models. However, given the concerns raised by many stakeholders, a review of international experience in this area would be instructive prior to the formulation of a final view on this matter

 
Another area of divergence in views was on the issue of simpler versus more sophisticated modelling. What was clear, however, was that clients were looking more for better communication of results, risk analysis, reasonableness checking and benchmarking, than for technical advances in modelling.

 In concluding, it is important to consider the incentives an agency has to implement the risk mitigation measures. Many of the measures such as guidelines and more commonality in modelling platform will require inter-agency agreement. Some measures such as more simplistic but transparent approaches are intuitively appealing to agencies. Other measures such as actively promoting dissenting and independent opinion may initially seem to have less appeal. However, clients for agencies were very concerned with the objectives of robust and credible forecasts and such measures will become increasingly important in demonstrating to management, boards and key internal stakeholders that these objectives are being met.

 References

Brinkman, P (2003), The ethical challenges and professional responses of travel demand Forecasters, PhD Dissertation, University of California, Berkeley

Flyvbjerg, B, Skamris, M, Buhl, S (2005), How (in)accurate are demand forecasts in public works projects, Journal of the American Planning Association, Vol 71, No.2, Spring

Flyvbjerg, B, Bruzelius, N and Rothengatter, W (2003) Megaprojects and risk: an anatomy of risk, Cambridge University Press, Cambridge.

Harvey, N (2001), Improving judgement in forecasting, in Armstrong, J (ed), Principles of forecasting, A handbook for researchers and practitioners, Kluwer, New York