A Path to Improved Pharmaceutical Productivity

Carl M. Cohen, Ph.D.

Abstract

Attempts to improve the productivity of pharmaceutical drug discovery efforts by applying currently used metrics-driven approaches will fail. This article presents an approach to the use of performance metrics that has the potential to guide fundamental industry improvement though cooperation.   

Drug discovery is a trial and error process that relies on biological science, yet is hampered by the incompleteness of our biological scientific knowledge. The consequence is that, more than any other industry, drug discovery relies on making fundamental scientific discoveries and applying those discoveries according to rules that are only partly known.  Imagine trying to design a modern aircraft with the knowledge that there might be rules of aerodynamics that are yet undiscovered, and that the only real test of the aircraft will come when passengers are placed on board. This is the problem that the  pharmaceutical industry faces daily. 

We undertake drug discovery with awareness of some general principles of the scientific method and a very incomplete understanding of the rules that biological systems obey. What is typically left unsaid when we observe that as many as 50% of the genes in the human genome have yet to be characterized is that there are many levels of biological complexity that remain to be understood. For every biochemical pathway that we think we understand there are many more that impinge on a given physiological process that we do not understand.  Because of this, the front end of drug development is more closely allied to discovery research than in any other major industry.  As a consequence, attempts to abstract learnings from other industries, where advances or  improvements can be achieved by fine tuning the application of known principles, are flawed. Only if we accept the fundamental difference between the state of our knowledge of the biological universe and the state of our knowledge of the physical and chemical universe will we develop appropriate paradigms for improving drug development.

The reliance on discovery in the early stages of drug development frustrates rational efforts to improve the process because there are no formulae for how to accelerate discovery on an industrial scale, or for that matter on any scale. 

A common approach that has been applied to improving drug discovery is that of empiricism - determining what works best by trial and error.   Two principal problems plague this approach. First is the absence of validated metrics for drug discovery performance, and second is the small size of the data set from which conclusions are drawn.  Without metrics one cannot know whether changes in process or technology affect performance or outcome. Moreover performance, however it is defined, must be measured in a time frame short enough that it can be used as feedback to guide the modification of practices. Thus, quantitating drugs to market is obviously a poor measure of current discovery performance because the time between discovery and market is too long to be practical. Moreover even if it were used, in the 10-12 years it takes to get a drug to market technologies change, knowledge increases and more importantly staff and management turn over. One cannot rationally apply feedback to a system that changes autonomously before the feedback can be applied. Much earlier, "surrogate markers" for performance must be developed that are more tightly coupled to discovery. 

However, even if such markers were available current practices suggest that most organizations would use them inappropriately. A key mistake made by organizations in the use of performance metrics is to use them as goal posts. Targets validated, leads generated, and NCEs produced become quotas to be met.  The universal experience with meeting quotas in discovery is that they can always be met. Unlike manufacturing quotas, where quality can be assessed almost immediately, the quality of deliverables in discovery is difficult to assess by traditional methods. Leads can be generated in any quantity, but what of quality?  

Box 1

Current use of metrics: a missed opportunity?

Many pharmaceutical  and biotechnology companies set explicit annual goals for targets validated, leads optimized, new chemical entities developed, etc.  These are often posted in conspicuous places throughout the organization to remind researchers of their productivity objectives . From my own observations such numerical goals are viewed with cynicism both by those who produce them, and by those who are supposed to be guided by them.  Company productivity managers acknowledge that "people produce what you measure."  A frequent complaint in organizations in which drug development is segmented is of the existence of the  "throw it over the wall" syndrome.  This results from one group tossing less than optimal product "over the wall" to the next group in the development line in order to meet its quota. The recipients of such products  become resentful and are tempted to perpetuate the process further downstream. 
The good news is that companies that collect accurate data on productivity and economic metrics already have in their hands the most difficult to collect subset of the data needed for the proposal presented here.  

Some existing organizations do a credible job of tracking performance both of the pharmaceutical industry as a whole and of individual companies.  The Tufts Center for Drug Development is a highly regarded organization that provides valuable industry-wide data on costs and other metrics associated with drug development (1, 2).  Similarly the Center for Medicines Research (CMR International) conducts detailed surveys that enable companies to compare selected discovery and development metrics of their company with peer organizations and with themselves over time (3).  The fact that the industry is obsessed with the development and use of performance metrics is attested by the number of national and international conferences each year devoted to this theme. Phacilitate, Cambridge HealthTech Institute  and IBC  all sponsor one or more conferences each year devoted entirely or partly to these themes(4) and numerous consulting organizations weigh in on industry metrics and performance on a regular basis (5). These efforts, while useful and necessary, do not go far enough. Knowing that the productivity of the industry as a whole or of an individual company has declined from one year to the next or that your company is less productive than peer companies provides motivation for improvement but no guidance or direction. What is needed is a map pointing to a path to improvement. 

This leads to the second problem with empiricism, that of the size of the data set on which decisions to change are based.  Such decisions are typically based on an assessment of the expected or measured impact of modifications in scientific or organizational practices on success within a given company. While this might work when the markers for success are temporally tightly coupled with outcomes, ie surrogate markers as discussed above, the approach suffers from the fatal limitation that any given company can only try a few different approaches in any given area of endeavor.  While following path B might give better performance metrics than you had while on path A, how might performance be affected  by following path C, D or E? 

Solutions?

How can we get around the use of metrics as goal posts? One possible solution is to assess the success of target validation not by the number of targets validated but by validated targets for which leads can be generated. Similarly one can assess  lead generation not by numbers of leads generated but by the fraction of leads that generate compounds with appropriate ADMET properties, and so on.  The principle is to assess the processes of group A not by its output, but by the output of the group it supplies. Even with this advance, what is being measured is still output, and if output is not as expected there is no guidance for improvement. 

How do we generate guidance for improvement? One solution is to expand the data set to include not one but many organizations. In this case the analysis consists not of measuring the performance of one organization trying different paths sequentially, but of many organizations trying different paths simultaneously. Here, performance metrics are used to identify high performing organizations. Their paths are characterized by data that enables a comparison of what these organizations do differently from others. Conceptually, such information can be categorized as scientific and technical behaviors or practices, process and procedures, and organizational structure.  

If such a data set were sufficiently large and the performance metrics sufficiently robust valuable guidance may be obtained (see Figure 1).

Box 2

How will it work in practice?

The process will have three phases: completion of survey instrument; quality control follow up; data analysis; feedback.  Completion of the survey will likely be accomplished by several people within each organization, for example the head of target validation or functional genomics for one survey section, the head of screening or lead optimization for another section, and a senior manager or Vice President for overall  company statistical information.  Quality control follow-up is necessary because the principal pitfall in such a study  lies in the inherent ambiguity of the questions.  Terms such as "target validation" "lead optimization" and even "budget," no matter how carefully defined, mean different things in different companies, and even to different people in the same company.  Thus, a rigorous (and time consuming) quality control process must be instituted to ensure that  questions are understood and answered consistently across companies. This process would involve in depth follow up interviews focused on the most ambiguous subset of the survey questions.   Data analysis will involve multiple approaches, from simple correlations between the use or adoption of specific organizational characteristics or technical practices or (eg. do you use virtual screening technologies?) and outcome metrics relevant to those practices (eg average cost and time per "lead") to more complex analyses of the relationship of clusters of practices or characteristics with outcomes.  However, it would be naïve to think that a mindless series of multivariate analyses of this type will of necessity yield profound insights. For this reason a third component of the analysis must consist of thoughtful review of the data by individuals experienced in drug discovery who could develop insights, paradigms and testable hypotheses from the data.  The feedback phase would consist of two elements. First a set of overall conclusions and insights relating technical and organizational characteristics to performance and outcome as defined by the survey and second, a series of customized reports specific to each participating company detailing their performance in each category relative to the other (un-named) participants (both as a whole and subdivided by peer-group) as well as a detailed elaboration of that company's technical and organizational characteristics relevant to each outcome (e.g. "it costs your company 50% more/less than average to develop lead series, but you do it in half/twice the time, and the key features that distinguish what you do and what others do  are rigorous inventory and quality control of archived combinatorial compounds and strict application of predictive algorithims for gate-keeping decisions on compounds.")  

What type of organization could perform such a service? First, any organization that undertakes such a project will need staff with deep domain expertise in drug discovery to design the study instrument and interpret the results.  Second, such an organization must be viewed as a trusted entity and have rigorous procedures in place to protect the participants' privacy and ensure anonymity of the data collected. Third, the entity should have sufficient stability to ensure its ability to carry out this project for a period of at least 5 years.  This should be a sufficient time for participants to be able to judge whether or not the results have utility.  

Collecting such data requires agreement by the major pharmaceutical companies that the complexity of the problems with which they're struggling are too great to be solved by any company individually.  Pooling non-proprietary information can enable the discovery of practices that will benefit the entire industry.  Such sharing will in no way create a level playing field.  Individual companies will still be differentiated by their ability to execute on what is learned, by the strength of their team and by their intellectual property estate. But a rising tide of understanding will improve the industry as a whole. This is an industry that has exhibited admirable but isolated episodes of cooperation in a variety of consortia that generate and share scientific data. Perhaps the time has come to develop and share the science of drug discovery.

Carl M. Cohen, Ph.D.

Carl M. Cohen has held executive positions in the biotechnology industry since 1997, prior to which he was Professor of Medicine and Cell Biology at Tufts University School of Medicine. He provides consultation to the biotechnology industry on organizational and technical matters.  

References

  1. Grabowski, H., Vernon, J. Di Masi, J.A. (2002) Pharamcogenomics, 20, Suppl. 3, 11-29
  2. DiMasi, J.A. (2002), Pharmacogenomics 20, Suppl.3: 1-10.
  3. Sim, P., Gill, J. Results of the 2002 Drug Discovery Performance Metrics Survey CMR02-188R,  CMR International.
  4. CHI "Strategies to optimize R&D Performance" June 9-19, 2003, Philadelphia, PA;  Phacilitate, Inc. "R&D Leaders Forum" October, 2003, Geneva Switzerland; IBC "Drug Discovery Technology Conference"  August 10-15, 2003, Boston, MA.
  5. "Parexel's Pharmaceutical R&D Statistical Sourcebook, 2002/2003," Parexel International;  "A Revolution in R&D: How genomics and genetics are transforming the biopharmaceutical industry"  Boston Consulting Group,  2001;  "Pharma 2010: The threshold of innovation" IBM Business Consulting Services, 2002.

Figure 1.  An illustration of how the ideas presented might be implemented.

The process

Step 1: Determine the productivity and efficiency of discovery efforts from target identification through lead optimization in a cross section of companies.

The following are some  of the "surrogate markers"  that can be  collected to track discovery productivity 

  • Average time from start of validation to start of lead-finding
  • Average time from start of lead finding to final lead series
  • Average costs for pursued targets; Average costs for killed targets
  • Proportion of validated targets for which successful screens or assays are developed
  • Number of validated targets generated and proportion for which leads are generated
  • Number of lead series generated and proportion that are successful in pre-clinical efficacy studies

Step 2. Gather detailed information on company science and technology, how the discovery process is conducted  and on discovery organizational characteristics .

Step 3 (the hard part).  Identify relationships between science/technology, process and organization with productivity and efficiency.  

The result

The following are illustrative of  the types of specific questions than could be addressed by the above process:

Impact of Science and Technology 

  • Which biological techniques are most cost and time effective in helping make target validation decisions? 
  • What is the impact of using an internal compound registration system that automatically predicts pharmaceutical properties on the time and cost of lead optimization?
  • What impact do virtual screening technologies have on time and cost for lead identification? 
  • What sources of compounds for screening yield the highest frequency of lead series that are pursued into lead optimization? (list includes pure compounds from archive,  combichem libraries, compound mixtures, etc.; may be target class specific)

Role of Discovery Process

  • Do companies that rely on the use of ADME/Tox prediction algorithms in compound selection have higher success rates in lead optimization?
  • Does the incorporation of formal checkpoints to eliminate poor targets improve productivity?
  • How does your company's policy for dealing with a compound series that is found to have development liabilities impact efficiency?
  • Are some aspects of drug design such as potency overemphasized at the expense of other important attributes? If so what impact does this have on lead finding and optimization? 

Effects of Organizational Structure and Behavior

  • Who makes target selection decisions? At what point does input from preclinical, clinical, legal, and marketing experts have the most positive impact on the output of the discovery process?
  • Do companies that have clearly defined ownership and management of the target validation process have better discovery track records than those that do not?
  • Define how high throughput screening groups are organized that have their own  budgets less productive than those that have a back-charge relationship with internal customers?
  • Do companies that report greater divisiveness and conflict in the discovery organization have poorer performance that those that do not?