Abstract

Objectives: To demonstrate the feasibility of a population-based measure of physician services utilization by type of service as a tool for physician workforce planning.

Setting: The Northern Health Region of British Columbia.

Design: Retrospective descriptive statistics are compiled about the regional population's physician services utilization by the specialty of the service, irrespective of the specialty or location of the provider. These are compared to norms based on provincial average utilization, adjusted for age and sex, and norms based on population-to-physician ratio recommendations.

Metrics: By specialty type of service: actual utilization; age-sex expected utilization; in-region, out-of-region and out-of-province utilization; full-time equivalency (FTE) values of actual and expected utilization; and FTE requirement to meet a set of recommended population-to-physician targets. Specialty substitution by general practitioners (GPs) is also quantified.

Results: The overall estimated deficit in physician numbers is similar between the two methods (51 versus 54), but the magnitude of surplus or deficit by specialty is greater with the population-to-physician method.

Conclusion: The method targets an equitable distribution, rather than normative ideal physician supply. The magnitude of estimated surplus or deficit at the level of each specialty is greater with the population-to-physician ratio approach. The latter fails to consider interregional flow and specialty substitution. A population-based utilization approach is demonstrated to be a feasible, and in many ways superior, tool for physician resource planning.

[To view the French abstract, please scroll down.]

In Canada, physician-to-population recommendations from the Royal College of Physicians and Surgeons of Canada (1988) and the Advisory Committee on Health Manpower (Health and Welfare Canada 1985) have guided physician workforce planning for a number of years. The utility of these metrics has been called into question for rural or regional workforce planning because services are often obtained outside the geographic region (Pong and Pitblado 2001) and there may be specialty substitution of services (Pong and Pitblado 2005).

There is a systematic and deliberate policy in Canada and other countries to train general practitioners (GPs) to provide "enhanced" or "advanced" services. (The term GP is inclusive of MDs with certification from the College of Family Physicians of Canada as well as non-specialist MDs not so certified.) Several specialty societies have issued joint statements with the College of Family Physicians regarding formalizing training for advanced or enhanced practice by general practitioners (Working Group of the Society of Rural Physicians of Canada 2001; Humber and Iglesias 1999; Iglesias and Hutten-Czapski 1999). All medical schools in Canada offer, to varying extents, a third-year addition to the Family Medicine Residency Program for advanced skills. A physician who wishes to provide procedural services in a hospital must provide evidence of his or her training to the responsible Credentials Committee and Medical Advisory Committee. Tepper (2004, 2006) also notes that there has been steady growth in opportunities for family physicians to gain training in areas of further specialization. Despite the training opportunities, the number of GPs providing obstetrics, procedural service and hospital care is in decline (Chan 2002).  

In 2000, the National Health Service in the United Kingdom set a target of training 1,000 "general practitioners with special interest" by 2004 (Nocon and Leese 2004). There are few reports of the outcomes of specialist-type services provided by GPs in Canada. However, a number of studies elsewhere demonstrate that general practitioners providing caesarean section in the United States (Deutchman et al. 1995a,b), other procedural rural medical practice in Australia (Hays et al. 2005a,b) and dermatologic practice in the United Kingdom (Salisbury et al. 2005) achieve similar clinical outcomes to specialists. Hays et al. (2005b) observed that the bias of GP-provided procedures is towards the "relatively simple." The relatively lower complexity of specialist services provided by GPs is reflected in data from British Columbia, though not reported in this paper. Thus, no analysis of physician supply is complete without attention to specialty services provided by GPs and specialty substitution in general.

The report of the Canadian Institute for Health Information (CIHI) on geographic distribution of physicians (Pong and Pitblado 2005) clearly shows that rural GPs work differently from their urban counterparts. They provide a broader scope of service and more specialty care. The methods used in this paper provide a consistent way to adjust for urban and rural differences in scope of practice and to handle specialty substitution issues across the spectrum of providers.  

Several important reviews point out that ideally, physician resource planning would start with measures of population need (Canadian Policy Research Networks 2002; Ontario Expert Panel on Health Professional Human Resources 2001). Although Denton et al. (2001, 2003) also acknowledge that a measure of need would be the ideal, their papers fall back on demographics as a proxy for need. Chan and Barer's (2001) presentation to the Fifth International Workforce Conference cites an expert panel with respect to the relationship between poor health status as evidenced by standardized mortality ratios and the need for more healthcare providers. The panel reported standardized mortality rates (SMRs) alongside physician-population ratios, but stopped short of recommending any threshold at which a community could be defined as underserved. Chan and Barer (2001) state: "This is perhaps not surprising, given the well-known problems with inferring anything very informative from physician supply-mortality relationships."

A comprehensive literature review and gap analysis was conducted in support of Task Force Two (Canadian Labour and Business Centre 2003). Among the findings in this report that pertain to this paper are the following:  

… there is no consensus for a method to measure need, [nor to quantify how need translates into numbers of providers required] (p. ii); exploration and development of needs-based planning models require further research (p. iv;) little information exists on what an informed public might want to spend on medical care (p. 63); there is a lack of consensus on the best planning approach to use (p. 67); no technical formula exists for determining the number of doctors needed (p. 68); the assessment of effective supply of physicians and how this supply meets population health care needs requires more solid data and methodologies (p. 68); and we need to look beyond aggregate numbers and ratios of physician-to-populations and adopt a methodology that can find broad acceptance in Canada (p. 73).

The CIHI Guidance Document 2005 (Tomblin and O'Brien-Pallas 2005) offers sound advice on priority information needs, indicators and data elements required to support compilation of health human resources in Canada, but offers little in the way of methodological advice to measure demand, or adjust for interregional flow and specialty substitution.  

Another literature review was conducted for Health Canada (2004). The editors' summary identifies a paucity of decision support tools. The authors called for a shift from planning for the "supply of health professionals" to "planning for health services" and identified substitution and expanded scope of practice and team delivery models as key deployment strategies. The use of a utilization target, as demonstrated here, is consistent with both the "access modelling" and "benchmarking" methods referenced by Chan and Barer (2001). This paper takes counting services as the starting point, rather than providers, consistent with the above-cited advice. Although this case study illustrates its application to a rural region, where specialty substitution is a primary focus, the method is also relevant and readily applicable to analysis of inter-regional flow of patients between predominantly urban regions.  

Methods

Study setting

The Northern Health Authority (NHA) was the setting for this study. The NHA is a vast area covering 588,344.8 square kilometres with a population of approximately 330,000. Over 35% of the NHA population resides in settlements of less than 1,000. Prince George, with a population of approximately 77,000, is the NHA's largest municipality.  

Data sources

The BC Ministry of Health granted permission to use the data. All data management practices conform to the Freedom of Information and Protection of Privacy Act. The population and demographic information comes from the registration files of the provincial healthcare claims system, called the Medical Services Plan (MSP). Physician payment data are for paid claims by date of service from the fiscal year ending March 31, 2003. Payment data regarding non-fee-for-service arrangements were obtained from the Alternate Payment Programs (APP) database in the Ministry of Health's health data warehouse. While the APP data are specialty-specific and reflect total compensation, there are no service-specific data in APP. For the purposes of the analysis, it is assumed that 100% of the services provided and dollars expended through APP contracts administered by the NHA flow to patients that reside in the NHA and that they are provided in the NHA.  

Physician functional specialty

Each physician service ("fee item") code described in the MSP Payment Schedule (Medical Services Commission 2003) was grouped to a mutually exclusive functional specialty service type. Specialty consultations are "restricted" in that they can be paid only to a certified specialist. "Unrestricted" codes may be paid to any licensed physician. Physicians get the same fee for these unrestricted items regardless of their certified specialty. For unrestricted services, the specialty group that submits the greatest total claims for the service item determines the functional specialty type. For example, an appendectomy is unrestricted. Appendectomies are most commonly provided by general surgeons and are therefore deemed to be within the general surgery functional specialty type of service, irrespective of whether the provider was a gynaecologist, a GP or a general surgeon.  

The functional specialty of a service can differ from the registered specialty of the provider. In this method, an individual physician can contribute to the supply of services - and thus to full-time equivalency (FTE) values - in more than one functional specialty. GP services and anaesthesiology can be assigned a fractional FTE for each function. The sum total of fractional parts can exceed 1.0, as explained below. This is a major distinction from population-to-physician ratios, in which physicians' contribution to supply are counted in only a single specialty and do not exceed 1.0. Owing to the limitations of the data (the lack of subspecialty-specific billing codes), the subspecialties of internal medicine and the subspecialties of paediatrics (such as gastroenterology and nephrology) are all combined in the internal medicine category or the paediatric category, respectively. Laboratory medicine and diagnostic imaging were also excluded for data quality reasons. The MSP Payment Schedule was amended partway through the period of analysis. Separate sections of cardiac surgery and thoracic surgery were created from the previously combined items covering "cardiothoracic surgery."

Incentives and perquisites (such as the Northern Isolation Allowance, the Medical On-Call Availability Program and after-hours service charges and surcharges) were excluded to ensure comparability from place to place.

FTE estimates

Although physicians may contribute several types of service to the population, they were nonetheless assigned to a mutually exclusive functional specialty peer group corresponding to the type of service from which they derive the greatest portion of their revenue. For example, if the greatest portion of a GP's revenue comes from general surgery services, he or she is assigned to the general surgery peer group.  

The total population expenditures for each respective specialty-type service were converted into notional FTE values by dividing each of these totals by mean earnings by members of the corresponding peer group. Thus (hypothetically), if the NHA population consumed $1 million worth of gerontology services and the average gerontologist earned $100,000, then the NHA population is said to have consumed 10 FTEs worth of gerontology services.  

Here the method departs from the CIHI formula (Tomblin-Murphy and O'Brien-Pallas 2005), which uses a logarithmic expression for physicians with earnings above the third quintile in their peer group. An arithmetic rather than logarithmic expression simply answers the question "How many physicians, earning an average amount for their functional specialty, would it take to deliver the expected level of service, if the population used the average level of that type of service, controlling for their age and sex?" To put it simply, in this method an FTE physician is an average physician. The actual number of FTEs minus the expected number of FTEs is the measure of physician surplus or deficit.  

Specialty substitution

A population-based approach is used to quantify specialty substitution as well. Owing to space limitations, only the analysis of specialty service substitution by GPs is reported. The sum of the service type utilized by members of the resident population and supplied by GPs is reported as a percentage of all services obtained by the resident population in the respective service type. The same method was used for the general (provincewide) population as well. The results are then compared.

For the purposes of population-to-physician ratios, the physician count is a head count based on registered specialty, consistent with past practice. To ensure that only active physicians are included, minimum inclusion criteria of at least 20 distinct patients and at least $25,000 in payments were used. Of physicians with an NHA address only, two were thus excluded.  


[Figure 1]


[Table 1]

 

Results

The results in Table 1 show how the resident population's actual use of services (column 'b') compares to age-sex expected use (column 'a') where the expected use is based on service utilization by the provincial population, and to the physician requirement calculated using population-to-physician ratios (column 'd'). Column 'e' shows the head count. Columns 'c' and 'f' show the difference between actual and expected use based on the utilization method and the population-to-physician method, respectively. Figure 1 shows actual utilization of each type of service as a percentage of age-sex expected utilizaton, where 100% is the provincial age-sex adjusted average utilization. For instance, in general surgery, the head count is 12 (column e) and the expected (target) is 12.5 (column a). The actual utilization of general surgery services is 34.4% above the expected utilization, as depicted in Figure 1. The 12 general surgeons were relatively more productive than average, having provided the equivalent of 13.42 FTEs worth of general surgery services (a subset of the total of 16.8, shown in column b of Table 1). There are 15 practitioners identified as internal medicine by functional specialty, compared to an expected or target of 42 by the service-utilization method, and 36 by the ACHM recommendation. Table 1 also contrasts the estimates derived by the service utilization method with the population-to-physician ratio method. Although the overall estimate of physician shortage is nearly the same for the two methods (51 versus 54 FTEs), the estimates by specialty are substantively different. The utilization method indicates a surplus of nine GPs, while the population-to-physician method indicates a GP surplus of 77 FTEs. In general, the estimates of shortage and surplus are more extreme for the population-to-physician method compared with the service utilization method.

Among the data graphically depicted in Figure 1 are the following: the population obtained 31 FTEs worth of internal medicine service. Of the 31 FTEs worth of service, physicians in the region provided 19.68 FTEs worth. Although there are only seven individuals identified as anaesthetists by functional specialty in the region, the population received over 25 FTEs worth of anaesthetic services, of which 18 FTEs worth was provided within the region.

In Figure 1, 100% is the level of service the regional population would have utilized - from all sources - had their utilization been the same as the provincial average, adjusted for age and sex. Figure 1 also shows the proportion of services that residents of the NHA received outside their geographic region. For explanatory purposes, a few examples follow:  

  • The largest proportions of out-of-region services were for surgical specialties.
  •  
  • In-region physicians provided negligible amounts of cardiothoracic and neurosurgery services. Despite that, the total utilization of these services was close to expected levels.  
  •  
  • In contrast, practitioners out of the region provided all physical medicine and rehab services, and the total utilized was only 10% of the expected level of just over 4 FTEs worth.

Comprehensive data tables and calculated values corresponding to Figure 1 are available upon request


[Table 2]


An analysis of specialty substitution was done for every specialty group and type of service. Only the analysis of specialty substitution by GPs is reported. Table 2 shows the proportion of GP-billed services categorized by specialty type of service and actually utilized by the NHA population. These proportions are compared to the corresponding proportions and providers supplying them in the province as a whole. For example, Table 2 shows that the NHA population obtained 51% of its obstetrical service from GPs, versus 39% for the provincewide population. The GP service contribution was approximately sixfold greater for orthopaedics, threefold greater for plastic surgery, and more than fourfold greater for anaesthetics for the NHA population than for the provincial population.  

The utilization of psychiatry services was less than 50% of expected. Specialty substitution may have occurred, but was not captured. There are no specific fee items for psychiatric management billable by GPs in the BC MSP Payment Schedule.

Discussion

There are limitations to a utilization-based method. Utilization is the quantity at which supply intersects with demand. This analysis assumes that demand is relatively uniform across British Columbia, adjusting for age and sex. Lower utilization than expected therefore implies a lower practical supply, irrespective of physician numbers. A utilization-based method does not provide any objective assessments of the "ideal" physician workforce benchmarks needed to maintain and improve population health. Nevertheless, it could be argued that provincial average utilization rates represent a "fair share" for regions within the province, assuming a reasonable homogeneity of age- and gender-specific needs.

The specialty restrictions on all fee items in the psychiatric section of the Payment Schedule lead to a weakness in the analysis of specialty substitution. GPs are obviously involved in the primary care of patients with mental health diagnoses, but whether they provide care that would normally have been provided by specialist psychiatrists cannot be measured by the methodology based on "ownership" of the fee item.

On a population-to-physician basis, the total supply of physicians in British Columbia is above the Canadian average (CIHI 2005), although not uniformly so, specialty by specialty. Where physician numbers are higher, this may not necessarily reflect excessive levels of service use. The supply of physician services also depends on physician productivity, hours worked and scope of services provided (Chan 2002).  

The health needs of a geographic region or subregion population may differ from provincewide averages. Standardized mortality ratios (SMRs) and potential years of life lost (PYLL) are also used as indicators of underlying population health. Given the pattern of mortality seen in the NHA, it is likely that age and sex adjustment alone somewhat underestimates need. PYLL and SMRs due to causes of death deemed "amenable to medical care" (also referred to as "treatable causes") are statistically significantly higher in parts of the NHA compared to the province as a whole (BC Statistics 2004).

Diagnosis-based risk adjustment groupers, such as the ACG method developed by Johns Hopkins University (Reid et al. 2001, 2002a,b), hold promise of relating health status to expected utilization. The application of ACGs to workforce analysis requires further study.  

In setting manpower targets in a region, consideration must also be given to the economic viability and sustainability of providing specialty services locally. Targeting improved physicians' lifestyle and/or improved access may result in low-volume practices and must be balanced with the relationship between volume and quality. Where the population-based requirements for a particular service type constitute less than a "critical mass" of FTEs, emphasis on telemedicine and visiting specialists may be more appropriate. Having quantified the FTEs required to deliver a target level of service, the resources needed to teach and do research can be explicitly added to the "bottom line" as fits the circumstances. If physician service targets are used as a starting point, the associated facility and ancillary staffing required can be estimated. The method depicted here can be - indeed, was - also applied at much smaller geographic levels, though not reported here.

Population density must be taken into account simply because people will have to travel greater distances to see their physician in sparsely populated large areas. Therefore, in some subregions, even if the actual population-to-physician ratios were brought to a provincial ratio, it would not necessarily create the same level of service access, again departing from the assumptions inherent in population-to-physician ratios. Planning for such sparsely populated areas must also take into account the standby capacity necessary to ensure access to emergency and primary care services. Changing the models of care delivery, such as shared care arrangements and the use of nurse practitioners, midwives and other professionals, will also change the target levels.

The use of the BC population as the reference population, or any utilization-based approach, suffers from the possibility that the average supply of services is more (or less) than "necessary." Total provincial utilization, however, is a mix of over- and underservicing. It is debatable whether the "average patient" is overserved or underserved, or whether "average physicians" over- or underserve their patients. Benchmarks, in contrast to utilization-based targets, theoretically set minimum service levels required to achieve acceptable outcomes. However, consensus around what these minimums should be is difficult to find.

Conclusion

This paper demonstrates both the feasibility and greater utility of a service-based and population-based approach compared to an approach based on population-to-physician ratios. No single methodology exists that can address all the uncertainties in physician resource planning. Any method that converts physician revenue into FTE values suffers from the same embedded assumption that the fees for services accurately reflect physicians' human resource input. The method demonstrated has the advantage of reflecting the reality of the services that patients actually consume and the services that physicians actually supply.


Compter des têtes ou compter des services? Comparer les méthodes population-médecin aux méthodes fondées sur l'utilisation des services pour la planification des effectifs médicaux

Résumé

Objectifs : Démontrer la faisabilité d'une mesure - fondée sur la population - de l'utilisation des services des médecins par type de service comme outil pour la planification de la main-d'œuvre médicale.

Cadre : Région sanitaire du Nord de la Colombie-Britannique.

Conception : Des statistiques descriptives rétrospectives sont compilées au sujet de l'utilisation des services des médecins par la population régionale, par la spécialité du service, sans égard à la spécialité du fournisseur ou à son emplacement. Ces statistiques sont comparées aux normes fondées sur l'utilisation provinciale moyenne, ajustées en fonction de l'âge et du sexe, et à des normes fondées sur les recommandations concernant le ratio population-médecin.

Mesures : Selon le type de spécialité du service; l'utilisation réelle; l'utilisation attendue en fonction de l'âge et du sexe; l'utilisation intra-régionale, extra-régionale et extra-provinciale; les valeurs des équivalences temps plein (ETP) de l'utilisation réelle et attendue; et les exigences en matière d'ETP pour atteindre un ensemble de cibles recommandées en ce qui concerne le ratio population-médecin. L'exercice d'une spécialité fonctionnelle par les généralistes est également quantifié.

Résultats : Le déficit global estimé du nombre de médecins est semblable pour les deux méthodes (51 c. 54), mais l'ampleur du surplus ou du déficit par spécialité est supérieure avec la méthode population-médecin.

Conclusion : La méthode se rapproche davantage d'une distribution équitable plutôt que d'un nombre normatif idéal de médecins. L'ampleur du surplus ou du déficit estimé dans chaque spécialité est supérieure avec la méthode du ratio population-médecin. Cette méthode ne tient pas compte des mouvements interrégionaux ni de la spécialité fonctionnelle. On a démontré qu'une approche fondée sur l'utilisation des services et axée sur la population constituait un outil faisable et à bien des égards supérieur pour la planification des effectifs médicaux.  


Disclaimer
The opinions and conclusions stated in this paper are those of the authors and do not necessarily represent the positions or policies of the Ministry of Health of British Columbia.

About the Author

Lorne Verhulst, MD, MPA
Senior Medical Consultant, Primary Health Care Branch
British Columbia Ministry of Health Services
Victoria, BC
Visiting Scholar, Post-doctoral Fellow
Department of Health Policy and Management
Bloomberg School of Public Health, Johns Hopkins University
Baltimore, MD

Christopher B. Forrest, MD, PhD
Professor of Pediatrics, Children's Hospital of Philadelphia
Philadelphia, PA

Mike McFadden, MA
Health Economist, Knowledge Management and Technology Division
British Columbia Ministry of Health
Victoria, BC

Correspondence may be directed to: Dr. Lorne A. Verhulst, Senior Medical Consultant, Medical Services Division, Ministry of Health, 1515 Blanshard Street, Victoria, BC V8W 3C8; tel.: 250-952-1425; fax: 250-952-1417; e-mail: Lorne.Verhulst@gov.bc.ca.

References

BC Statistics. 2004. Selected Vital Statistics and Health Status Indicators. 132nd Annual Report 2003. Retrieved April 10, 2007. < http://www.vs.gov.bc.ca/stats/annual >.

Canadian Institute for Health Information (CIHI) et al. (1998; 2005). Full-Time Equivalent Physicians Report, Canada. Ottawa: Author.

Canadian Institute for Health Information (CIHI). 2006 (January). Full-Time Equivalent Physicians (FTE) Report, Canada, 2002-2003 and 2003-2004. Ottawa: Author.

Canadian Labour and Business Centre. 2003. Physician Workforce in Canada: Literature Review and Gap Analysis. Ottawa: Task Force Two.

Canadian Policy Research Networks Inc. and C. Fooks. 2002. Health Human Resource Planning in Canada: Physician and Nursing Work Force Issues. Ottawa: Commission on the Future of Health Care in Canada.

Chan, B.T. 2002. "The Declining Comprehensiveness of Primary Care." Canadian Medical Association Journal 166(4): 429-34.  

Chan, B.T. and M. Barer. 2001. "Access to Physicians in Underserviced Communities in Canada: Something Old, Something New." Presentation at the 5th International Medical Workforce Conference, Sydney, Australia, November 2000. Vancouver: Centre for Health Services and Policy Research.

Denton, F.T., A. Gafni and B.G Spencer. 2001. "Population Change and the Requirements for Physicians: The Case of Ontario." Canadian Public Policy 27(4): 469.  

Denton, F.T., A. Gafni and B.G. Spencer. 2003. "Requirements for Physicians in 2030: Why Population Aging Matters Less Than You May Think." Canadian Medical Association Journal 168(12): 1545-47.  

Deutchman, M., P. Connor, R. Gobbo, and R. FitzSimmons. 1995a. "Outcomes of Cesarean Sections Performed by Family Physicians and the Training They Received: A 15-Year Retrospective Study." Journal of the American Board of Family Practice 8(2): 81-90.  

Deutchman, M.E., D. Sills and P.D. Connor. 1995b. "Perinatal Outcomes: A Comparison between Family Physicians and Obstetricians." Journal of the American Board of Family Practice 8(6): 440-47.

Hays, R.B., R.J. Evans and C. Veitch. 2005a. "The Quality of Procedural Rural Medical Practice in Australia." Rural Remote Health 5(4): 474.  

Hays, R.B., C. Veitch and R.J. Evans. 2005b. "The Determinants of Quality in Procedural Rural Medical Care." Rural Remote Health 5(4): 473.  

Health Canada Health Policy Branch and North South Group Inc. 2004. Literature Review and Environmental Scan of Preferred Practices for Deployment of Health Human Resources and Decision Support Tools. Ottawa: Author.

Health and Welfare Canada. Federal/Provincial/Territorial Advisory Committee on Health Manpower. 1985. Physician Manpower in Canada 1980-2000. A Report of the Federal/Provincial/Territorial Advisory Committee on Health Manpower. Ottawa: Author.

Humber, N., et al. 1999. "Position Paper on Training for Family Physicians in General Surgery Society of Rural Physicians of Canada." Health Canada.

Iglesias, S. and P. Hutten-Czapski. 1999. "Joint Position Paper on Training for Rural Family Practitioners in Advanced Maternity Skills and Cesarean Section." Canadian Journal of Rural Medicine 4(4): 209.  

Medical Services Commission of British Columbia. 2003. MSP Physician Payment Schedule. Victoria: Author.

Nocon, A. and B. Leese. 2004. "The Role of UK General Practitioners with Special Clinical Interests: Implications for Policy and Service Delivery." British Journal of General Practice 54(498): 50-56.  

Ontario. Expert Panel on Health Professional Human Resources. 2001. Shaping Ontario's Physician Workforce: Building Ontario's Capacity to Plan, Educate, Recruit and Retain Physicians to Meet Health Needs. Toronto: Author.

Pong, R.W. and J.R. Pitblado. 2001. "Don't Take 'Geography' for Granted! Some Methodological Issues in Measuring Geographic Distribution of Physicians." Canadian Journal of Rural Medicine 6(2): 103.  

Pong, R.W. and J.R. Pitblado. 2005. Geographic Distribution of Physicians in Canada: Beyond How Many and Where. Ottawa: Canadian Institute for Health Information.

Reid, R.J., L. MacWilliam, L. Verhulst, N. Roos and M. Atkinson. 2001. "Performance of the ACG Case-Mix System in Two Canadian Provinces." Medical Care 39(1): 86-99.  

Reid, R.J., N.P. Roos, L. MacWilliam, N. Frohlich and C. Black. 2002a. "Assessing Population Health Care Need Using a Claims-Based ACG Morbidity Measure: A Validation Analysis in the Province of Manitoba." Health Services Research 37(5): 1345-64.  

Reid, R.J., L. Verhulst and C.B. Forrest. 2002b. "Comparing Apples with Apples in Clinical Populations: Applications of the Adjusted Clinical Group System in British Columbia." Healthcare Management Forum 15(2): 11-16.  

Royal College of Physicians and Surgeons of Canada. 1988. National Specialty Physician Review (reprint ed.). Ottawa: Royal College of Physicians and Surgeons of Canada.

Salisbury, C., A. Noble et al. 2005. "Evaluation of General Practitioners with Special Interest Service for Dermatology: Randomised Controlled Trial." British Journal of Medicine (Clinical Research Ed.), 331(7530): 1441-46.  

Tepper, J. and Canadian Institute for Health Information. 2004. The Evolving Role of Canada's Family Physicians, 1992-2001. Ottawa: CIHI.

Tepper, J. and Institute for Clinical Evaluative Sciences. 2006. Physician Services in Rural and Northern Ontario ICES Investigative Report. Toronto: Institute for Clinical Evaluative Sciences.

Tomblin-Murphy, G. and L. O'Brien-Pallas. 2005. Guidance Document for the Development of Data Sets to Support Health Human Resources Management in Canada. Ottawa: Canadian Institute for Health Information.

Working Group of the Society of Rural Physicians of Canada, College of Family Physicians of Canada and the Canadian Anesthesiologists Society. 2001. Joint Position Paper on Training for Rural Family Physicians in Anesthesia. Shawville, QC: Society of Rural Physicians of Canada.