MARKET FORECASTING - Monte Carlo-Based Forecasting: How to Deal With Uncertainty

Introduction

Imagine that you have just wrapped up the Phase II trial of your new drug. The results are promising, yet the launch is several years out. R&D costs are significant, and healthcare budgets are under pressure. Your CEO is expressing concerns about critical success factors like FDA approval, potential changes of payer policies, and a chance that competition beats you to market. A market forecast may help your business decide whether to continue with the development. Yet, when so many factors going into the forecast are uncertain, there is doubt surrounding the value of conducting a forecast at all. Before you know it, conditions may change, the forecast could be rendered invalid, and you could be forced to start anew. But why not turn uncertainty into a virtue? In this paper, we show how Monte Carlo-based forecasts are better at handling uncertainties, turning them into valuable tools for product managers and marketers engaged in strategic business planning.

What is a Monte Carlo-Based Forecast?

A traditional market forecast of a new drug or treatment produces a single revenue number, say $3 billion over a 5-year period following launch. It is computed by a multiplicative exercise: Revenue = Population x Awareness x Acceptance x Price x Compliance. Awareness is most likely to increase over time because of investments in above or below the line communication and sales force effectiveness. Acceptance is the healthcare professional’s expected prescribing behavior of the new drug at the given price and taking into account payer policies for eligible patient groups. Usually, we also take into account the competitive context of alternative drugs and treatments, showing how much the new drug helps grow the market or how it can steal share from competition.

Unlike a traditional forecast, a Monte Carlo-based forecast does not produce a single number, but instead provides a range of possible outcomes at a probability distribution. For example, it indicates that sales over the 5-year period are likely between $2.5 and $3.5 billion. The probability distribution may show there is a 100% likelihood that sales will be $2.5 billion and still a 50% chance of sales being $3.5 billion (Figure 1).

FIGURE 1. From a Point to a Range Forecast.

A Monte Carlo simulator produces output ranges because the inputs are data ranges instead of data points. For example, instead of working with a single awareness number of 50%, we work with a range of 40% to 60%. Or, availability could be determined at 20%, 40%, or 60%, depending on payer policies. Taking into account all factors, the simulator multiplies range values instead of point values, producing an output range.

We believe that Monte Carlo-based forecasting is more realistic and intuitive. After all, who is expecting to make exactly $3 billion? Forecasts may always be slightly off, and most likely, they are off because of errors or uncertainties in the input variables. Therefore, it is better to explicitly account for them. In fact, Monte Carlo-based forecasting delivers a superior way of dealing with uncertainty because it helps provide a structure for contemplating what uncertainty looks like. It does so by allowing us to set the shape of the distribution of the range. A common example of this is the “normal distribution.” For example, it says that the mean awareness at moment t is 50%, but the bulk of awareness values are distributed between 40% and 60% (Figure 2A). Its most common counterpart is the uniform distribution: it sets two extreme values (eg, 40% and 60% for awareness); every value in between has an equal likelihood of occurring (Figure 2B).

These ranges are used to deal with variables that have continuous values. Similarly, we can work with the likelihood of discrete events. For example, we can use this to model the effect of competitive launches. Imagine that we expect a first mover advantage; the first to enter the market puts up an entrance barrier and sets followers at a disadvantage. We can set the likelihoods that either we or the competition move first, and we assign the first mover advantage. This will impact the distribution of the forecasted variable. The rule is the higher the uncertainty in inputs, the wider the input ranges, and the more likely we choose a uniform instead of a normal distribution.

FIGURE 2A. Normal Distribution A normal distribution is used when the uncertainty associated with a variable is low. A normal distribution can either be specified by setting the mean and standard deviation, or by setting the minimum and maximum value. Values are randomly generated based on the normal distribution. The red line describes the expected distribution; the blue dots describe the frequencies of compliance values in the 10,000 runs of a Monte Carlo simulation.

FIGURE 2B. Uniform Distribution A uniform distribution is used when the uncertainty associated with a variable is high. A uniform distribution can be specified by setting the extreme values. Values in between the two extremes are randomly generated. The red line indicates the expected distribution; the blue dots indicate the frequencies of compliance values in the 10,000 runs of a Monte Carlo forecast.

Why Do We Account for Uncertainty?

We account for uncertainty because the input variables in our forecasts are not always of good quality, and because specific events can render a forecast invalid if not accounted for. In the United States and other established markets, reliable data are widely available. Launches of new drugs and treatments in common disease states are well documented and provide for great analogues. However, there are several situations for which there are no analogues or secondary data. Yet, that’s where opportunities for new drugs and treatments often lie. Examples include emerging markets, less-explored disease states, and rare or orphan diseases.

Another reason is that inputs in forecasting often come from a variety of sources from different origins: syndicated research, expert interviews, government statistics, internal sales data, or simply anecdotal evidence and the hunches of business people. Data from different sources often contradict each other, and it may be impossible to get to a single, valid number. This may lead to researchers or business units to agree on just one data source to keep metrics aligned and avoid having to deal with conflicting data. This may work at face value, but is it really the way toward an accurate forecast and the best description of reality? Working with ranges or scenarios is a better solution. Finally, one can be sure that competition will launch new products and that payer policies will change; the only question is when? Especially in a situation in which we believe that the order of events will impact our performance, it is important to do scenario planning and include the scenarios and their corresponding likelihoods in our forecast. Scenario planning needs to include input from business owners and stakeholders and involves answering three questions: (1) what could happen, (2) when might it happen, and (3) what is the likelihood of it happening? True to the nature of scenario planning in a situation of uncertainty, the first two questions can have conflicting answers.

How Else Do We Deal With Uncertainty?

Monte Carlo-based forecasting does not help to deal with uncertainty by itself. On the contrary, the fact that it produces an output range instead of a single number can contribute to the uncertainty. That is why we set action standards. An action standard consists of two things: a threshold sales value or other KPI (key performance indicator) that the forecast needs to exceed before the business decides to proceed with the initiative (eg, $2 billion over 5 years), and the likelihood of making this number (eg, 100% sure to make $2 billion).

Action standards serve two purposes. First, they help us to set the expectations from the business. We do this as part of the study design and before sharing the results. Because the expectations are set beforehand, decision-making is usually more informed and of better quality. Second, we suggest setting the values, shapes, and ranges of the input variables in close collaboration with the business. It is here that the business learns to cope with the uncertainties associated with the input variables.

Through this process, we learn about the business’ appetite for risk. Some (companies, functions, or persons) are rather risk averse and are more willing to  accept a lower result at a higher likelihood  (eg, 100% sure to make $2 billion). Others  accept more risk and set a higher result at a lower likelihood (eg, 50% sure to make $3 billion). One only pursues an initiative if one meets or exceeds the action standard.

FIGURE 3. A Monte Carlo Forecast of Revenue In a Monte Carlo forecast, thousands of scenarios are run. In each scenario, a value for each variable is drawn from the distribution of values, and the values are multiplied to compute a revenue number. The thousands of scenarios result in a distribution of revenue values, responsible for the likelihood curve used to determine if we make the target value set by the action standard.

Exhibit 1: The Case of Periculum, a New Type 2 Diabetes Drug

Further, we present a fictitious business case to show the relevance of the approach in a business context. Ducendi Incorporated is a big pharmaceutical corporation, and type 2 diabetes is one of its main focus areas. Launching a new molecule will ensure Ducendi remains a strong player in this market. Always on the look-out for up-and-coming biotech companies with interesting innovations in the pipeline, Ducendi has focused its attention on Novus Pharmaceuticals.

Novus has just met its Phase II clinical trial endpoint for a new drug to treat patients with type 2 diabetes called Periculum. Periculum is an oral drug with a novel mode of action that is believed to ensure minimal adverse events with an outstanding tolerability profile.

Novus and Ducendi are looking into a partnership agreement to take the drug from the current Phase II clinical trial to its launch a few years out. Novus asks Ducendi to fully fund the Phase III trial set up a production plant, and launch Periculum globally. But there are uncertainties. While the efficacy data coming out of the Phase II trial look promising, the Phase III outcomes may disappoint. Also, rumor is that competition is going to launch a similar drug right around the same time. While Novus is keen on signing a license agreement, Ducendi wants to first assess the opportunity of Periculum.

FIGURE 4. Revenue Forecast This chart shows a Monte Carlo revenue forecast with on the horizontal axis the revenue value and on the vertical axis the likelihood value of making the revenue.

Usually, Ducendi validates business cases in the US only, because succeeding there almost certainly generates a positive ROI. However, because of expected changes in the healthcare landscape in the US and global markets, Ducendi also wants to validate the business case in emerging markets. While validating a business case is easy in the US and other established markets because of the availability of analogues, it is more difficult in emerging markets. Analogues are less prevalent, and fewer data are available and of uncertain validity. Hence, the need for a Monte Carlo-based forecast.

The team sets out to collect data about key emerging markets. First, they commission a study to validate interest in the drug. They estimate the willingness of healthcare professionals to switch to the new drug under target product profile scenarios describing variations in efficacy and tolerability profiles. They also include similar drugs by different pharmaceutical brands to help the team get insight into the peak share of the new drugs under various launch scenarios. They define the size of the patient population and set the expected awareness curve, patient compliance, and persistence levels. Awareness is a compounded measure accounting for satisfaction with current drugs and treatment and the awareness or “buzz” associated with the new product. As input, they collect government statistics and syndicated data enriched by expert interviews with practitioners, opinion leaders, policy makers, and payers. Because there is no such thing as one truth, the team decides to include range forecasts of awareness curves (Table 1) and patient compliance (Table 2).

Next, they align about the likelihood of events and outcomes. They set the likelihood of Periculum to be less, equally, or more efficacious in the Phase III clinical trial compared with Phase II (Table 3).

They also set the likelihood of the competitor drug to be launched before or after Periculum, or for both to be launched  at the same time (Table 4).

Then, acceptance levels for Periculum are determined under the various scenarios of efficacy and the competitive launch. Next, they convert study results into a market forecast (Figure 3).

The best way to ensure management buy-in for the outcome of the forecast is have them take part in the preparation. The team sits with Ducendi management to agree upon the expected revenue and whether or not to proceed with Periculum. Ducendi’s gold standard is to break even in 5 years after launch. The 5-year investment in Periculum is estimated at $1.5 billion, including payments made to Novus, royalties on sales, an upfront fee due upon signing the deal, and milestone payments. Therefore, the 5-year revenue of Periculum must exceed $1.5 billion to generate a positive ROI.

We know that a Monte Carlo forecast provides a range estimate (Figure 4).

So, the second thing management needs to decide is what risk they want to accept not to meet or exceed the target value. Having a low appetite for risk, we can set the likelihood of making $1.5 billion at 100%. But management is willing to take a higher risk of not making a large profit (eg, a 75% likelihood of making $2 billion). We take these values as the action standards to decide whether or not to proceed with a partnership between Ducendi and Novus.

When running the numbers including the uncertainty factors, it becomes clear that there is a 93% likelihood of making $1.5 billion and an 81% likelihood of making $2 billion (Table 5). The team makes a positive recommendation for an in-licensing deal between Novus and Ducendi.

Summary

The reality of the healthcare industry is that what happens around us is stochastic, but the state of mind is still deterministic. Decision-makers often want one answer and are not inclined to commit to a range nor to the associated probability. Deciding on action standards at the outset of a research study and accurately setting ranges for the parameters are crucial to the success of the forecast. Both of these require collaboration and careful thinking among market researchers and business intelligence managers. Ultimately, Monte Carlo simulations are a way to obtain more accurate forecasts in areas of great complexity and uncertainty – never deleting the uncertainties, but helping us to cope with them.

Jemma Lampkin
Senior Project Manager
SKIM

Jemma Lampkin (j.lampkin@skimgroup.com) is a Senior Project Manager in SKIM’s Hoboken, New Jersey office. She has over 8 years of experience in the research industry in the healthcare sector. She has extensive experience designing and conducting global quantitative and qualitative market research studies in a wide range of healthcare indication areas. Ms. Lampkin earned her BA in Psychology from Columbia University.

Dr. Gerard Loosschilder
Chief Methodology
Officer & Partner
SKIM

Dr. Gerard Loosschilder (g.loosschilder@skimgroup.com) is Chief Methodology Officer and partner at SKIM, responsible for innovation in methodology. He has over 20 years of experience in market research in various positions in academia and on the client and the agency side. He earned his PhD in Market Research from Delft University of Technology in Delft, the Netherlands. Before joining SKIM, he was Senior Director of Market Intelligence at Philips Domestic Appliances and Personal Care.