Citizens' Reference Panel on Health Technologies - Ontario Canada

Citizens' Reference Panel on Health Technologies - Ontario Canada



In December 2008, a 14-member Citizens’ Reference Panel on Health Technologies (CRPHT) was established to inform the work of the Medical Advisory Secretariat (MAS) and the Ontario Health Technology Advisory Committee (OHTAC), which produces evidence-based recommendations regarding the use of health technologies throughout Ontario’s health system. A specific emphasis of the panel’s work was to elicit the societal and ethical values that should guide OHTAC in developing its recommendations about whether and how technologies should be used in Ontario. The creation of the CRPHT represents early experimentation with a deliberative engagement method to incorporate public values and citizen perspectives into the highly expert-driven health technology assessment processes. The panel was established through a research collaboration between McMaster University, led by Dr. Julia Abelson (Department of Clinical Epidemiology & Biostatistics) and the Medical Advisory Secretariat (MAS) of the Ontario Ministry of Health and Long-Term Care, headed by Dr. Les Levin.

Problems and Purpose

In Canada, several major justifications existed to encourage a greater public role in both HTA (Health Technology Assessment) and HT coverage policymaking. First of concern, the public’s  confidence in the Canadian health system was eroding and had motivated governments to place principles of accountability and transparency on par with core health system principles of universality and equity. Second, increased scrutiny of the science community and public investments in scientific research funding prompted calls for greater accountability from all publicly funded research, including technology assessment agencies and basic researchers.  Also, citizens and stakeholders expected greater clarity about how this information was used to approve and fund health technologies. Third, the proliferation of process-oriented decision-making frameworks to guide coverage decisions emphasized the importance of seeking greater public legitimacy in rationing decisions through a variety of mechanisms including participation, deliberation, publicity, and appeal. In the HTA context, this had been articulated generally as the need to “attend to the procedural features that are also relevant in the technology assessment process” [8]. Finally, a growing consumerist orientation to health care policy was routinely drawing health system ‘users’ into consultations, evaluations, and decision-making processes about health technology by health planners and managers. [2]

In Canada, both HTA and HT decisions traditionally were invisible and inaccessible to the public, especially during their creation. Canadians normally learned of specific assessments and decisions as fait accompli (already decided and presumed irreversible). However, Canadian policy began to consider how the public should contribute to and be apprised of HTA and HT coverage decisions. Questions included whom to involve (e.g., which publics), how (e.g., through what public involvement or accountability mechanisms), and for what purpose (e.g., to inform or to be apprised of decisions and their rationales). While some other nations were well ahead of Canada in their public involvement efforts, the HTA community remained in need of a general framework for mapping the activities of HTA/HTP onto opportunities for public. [8]


Over the last decade, the ‘deliberative turn’ in health care has been rooted on stronger grounding in several high-profile areas in Canada. In 2002, for example, the principles of democratic deliberation were used to design the citizens’ dialogue sessions that were held across the country as part of the work of the Commission on the Future of Health Care in Canada. The purpose of the dialogues was to elicit, through deliberative dialogue, citizens’ values toward four options for health care transformation, which would in turn be used to inform the Commission’s final recommendations. [3]

Then in 2006, an almost identical citizens’ council was established by the Government of Ontario, which aimed to provide public input into the governance of the province’s pharmaceutical program. In 2007, the position of “Commissioner of Health and Well-Being” was introduced for advising the government on the performance of the Quebec health system. In general, these deliberative experiments have been on the rise in Canada. [4]

In 2001, the Ministry of Health and Long-Term Care (MOHLTC) established the Medical Advisory Secretariat (MAS) to conduct evidence-based analyses to help stakeholders make policy and funding decisions about health technologies in Ontario. The MAS, which is comprised of health care specialists including physicians, clinical epidemiologists, policy analysts and health economists, is committed to ensuring that residents of Ontario have access to the best available new health technologies that will improve patient outcomes (MOHLTC). [4]

The MAS produces evidence-based analyses of health technologies that are then reviewed by the Ontario Health Technology Advisory Committee (OHTAC), an arm’s length expert committee established in 2003 that makes recommendations to the Ontario health care system, and the Ontario Ministry of Health and Long-Term Care (MOHLTC) about the best health technologies for Ontario (MOHLTC). MAS and OHTAC have had a long-standing interest in involving relevant stakeholders in their goal of incorporating ethical and societal perspectives into the health technology policy analysis process. In 2007, OHTAC formed a public engagement sub-committee, which identified the need for ‘added vehicles for communication and consultation’ with three stakeholder groups: patients, families and caregivers; advocacy groups; and the general public and Ontario taxpayers. [4]

Originating Entities and Funding

In 2001, the Ministry of Health and Long-Term Care (MOHLTC) established the Medical Advisory Secretariat (MAS) to conduct evidence-based analyses to help stakeholders make policy and funding decisions about health technologies in Ontario.

The originating entities:

  • Ontario Ministry of Health and Long-Term Care (Medical Advisory Secretariat)
  • Ontario Health Technology Advisory Committee (OHTAC) [1]

Participant Recruitment and Selection

MASS LBP was contracted to recruit the 14-member panel. A ‘civic lottery’ methodology was used which was inspired by Canada’s first Citizens’ Assemblies in Ontario and British Columbia. A civic lottery is a random-representative selection process designed to identify and encourage citizens to step forward and volunteer to participate in a wide range of deliberative and civic initiatives.

A randomly generated list of thirty-five hundred Ontario residents were mailed detailed invitation letters, information sheet and postage-paid response form. These letters expressed the opportunity to opt into a pool of respondents. From this sample, 163 expressions of interest in panel membership were received, from which 14 panel members — selected by geographic region and stratified by gender and age, with one participant coming from each of Ontario's 14 Local Health Integration Network regions — were selected blindly from the respondent pool. All travel expenses associated with meeting attendance were covered and each panel member received a $65 honorarium after attending each meeting. [1]

Methods and Tools Used

This case used the Citizens' Lottery and Citizens' Reference Panel methodologies characteristic of other participatory events designed by MASS LBP.

Deliberation, Decisions, and Public Interaction

The 14-member panel was convened for 5 one-day meetings at McMaster University, in Hamilton Ontario, between February 2009 and June 2010. Topics were selected in consultation with the research team and MAS and OHTAC representatives. At each meeting, panel members participated in structured discussions covering two broad topic areas: 1) the core values that should guide the evaluation of health technologies; when and by whom should this be done? And, 2) reviews of selected health technologies (e.g., screening for colorectal and breast cancer, serological testing for celiac disease, personalized medicine) to provide ethics and social values input to OHTAC deliberations. The technologies selected for discussion at each meeting were jointly agreed to by the research team, Medical Advisory Secretariat staff and OHTAC members.

Information about the topics under discussion at each meeting was pre-circulated in a meeting workbook which included background information about the topic, explanation of key concepts relevant to the topic, and a list of discussion questions to be addressed which formed the basis of the input to be provided to the decision-maker organization. Meetings were facilitated by the research team and included a combination of large- and small-group interactions organized around a pre-circulated agenda, discussion topics and questions. The meetings were audio-recorded. Data collected included qualitative verbatim transcripts, quantitative surveys administered before and after the meeting, and observation notes recorded by members of the research team. Representatives from MAS and OHTAC attended two of the five meetings to describe the MAS-OHTAC process, to explain the rationale for the creation of the Citizens’ Panel and to answer panel member questions.

The panel met 5 times over an 18-month period between February 2009 and June 2010. Over the course of five meetings convened over the two-year study, panel members progressed toward the identification of a set of core ethical and social values – universal access, choice and quality care – at stake in evaluating health technologies and in forming recommendations for their use in the health system. [7]

Influence, Outcomes, and Effects

Each of the five CRPHT meetings generated two types of outputs to inform MAS-OHTAC deliberations regarding the evaluation of selected technologies:

  1. Qualitative reflections on a set of pre-circulated questions designed to elicit societal and ethical values related to the technology under review
  2. Pre- and post-meeting survey results of panel member rankings of the societal and ethical values central to the technology under review

A thematic summary of panel discussions was prepared following each panel meeting and shared with OHTAC members in either report or presentation format for discussion at monthly OHTAC meetings.

While tangible policy impacts are rarely observed in time-limited studies of this kind, there were some impacts observed over the course of this two-year project. Since it used social values and ethics data to inform its work, the CRPHT provided new insights in these areas to MAS and OHTAC. Panel input was also used explicitly in OHTAC deliberations for three of the five technologies reviewed. [1]

Based on direct reports back to the CRPHT about how participant input was used, most panel members felt that their input was valued by OHTAC. Attendance of senior MAS/OHTAC officials at two CRPHT meetings, combined with the open invitation for CRPHT members to attend an OHTAC meeting so that they could provide their input on the value of the panel model, reinforced confidence among participants about the panel's perceived value to the two organizations. Other panel members believed the CRPHT's work had minimal influence but felt it was a crucial step to increasing the awareness of MAS and OHTAC about the importance of incorporating public values into these types of decisions. [1]

In a news article in 2009, the following was said about the Citizen’s Reference Panel:
“The Citizens’ Reference Panel on Health Technologies (CRPHT) has already made an impact. At its inaugural meeting in February, the panel discussed patient information and choice in screening technologies for the early detection of colorectal cancer. Their views on how the public should be informed and choose whether to take part in colorectal cancer screening tests informed OHTAC’s final recommendations to the Ontario health care system regarding the use of these technologies.” [9]

Analysis and Lessons Learned

Several concerns exist with contemporary deliberative events such as the Citizen’s Reference Panel. For instance, consumer accountability now seems to drown out more traditional concerns with effectiveness and efficiency. “Operationalizing public involvement raises a host of questions about the differences between citizen values and consumer preferences; (5)  the extent to which selected individuals or groups can legitimately or accurately represent the public; how to deal with the heterogeneity of stakeholders that include large, professionally run organizations as well as smaller, volunteer-led groups; and how to address the potential biases associated with groups that may be funded by industry and/or professional organizations [6].”   

An even larger gap within current frameworks is the discussion of how the broader public will be apprised of these decisions and the manner and extent to which public values contributed to them.

The Canadian Institutes of Health Research assessed and summarized the strengths and challenges of the Citizens’ Reference Panel:


  • Integration of input from a wide range of perspectives (citizen included) to contribute to provincial policy making
  • Meeting its objective to provide citizen input on societal and ethical issues
  • Facilitation of meetings kept the group on track and encouraged participation from all members
  • Use of small groups to provide a platform for all members' input
  • Arm's length relationship between CRPHT and MAS/OHTAC
  • Sponsor's attendance at CRPHT meetings (re-affirmed panel role and contributions)
  • Raising awareness of the importance of incorporating public input into the HTA process
  • Creating a model for public involvement in HTA that can be improved upon in the future


  • Informational requirements: "going into it cold" and the large quantity of information to take in during the session
  • Discussions "meandered" or were "monopolized" at times
  • Challenging to feel part of the panel again at each meeting given the time between them
  • Size and representativeness of panel
  • Role confusion
  • Communication between citizen's panel and sponsor

(source: CRPHT members [1])


Secondary Sources

[1]. Ableson, Julia et. al. “Case 12: Consulting Ontario citizens to inform the evaluation of health technologies: The citizens’ reference panel on health technologies.” Canadian Institutes of Health Research.

[2]. Devidas Menon, Tania Stafinski, Health Technology Assessment in Canada: 20 Years Strong?. Value in Health, Volume 12, Supplement 2, June 2009, Pages S14-S19. <>

3]. Maxwell, J.S., Rosell, S., and Forest, P.-G. (2003) ʻGiving Citizens a Voice in Health care Policy in Canadaʼ, British Medical Journal, May, 326: 1031–1033.

[4]. Ableson, Julia, Gauvin, François-Pierre, & Martin, Élisabeth. (2010). “Practicing the Theory of Public Deliberation: Case studies from the Health Sector in Ontario and Quebec.” 2010 Canadian Political Science Association Conference. Ontario. <>

[5]. Shiell A, Hawe P, Seymour J. Values and preferences are not necessarily the same. Health Economics 1997; 6:515–8.

[6]. Coulter A. Perspectives on health technology assessment: response from the patient’s perspective. International Journal of Technology Assessment in Health Care 2004; 20(1): 92–6.

[7]. Yvonne Bombard, Julia Abelson, Dorina Simeonov, Francois-Pierre Gauvin, Eliciting ethical and social values in health technology assessment: A participatory approach, Social Science & Medicine, Volume 73, Issue 1, July 2011, Pages 135-144, ISSN 0277-9536, 10.1016/j.socscimed.2011.04.017. <>

[8]. Ableson, Giacomini, Lehoux, and Gauvin. Bringing “the public” into health technology assessment and coverage policy decisions: From principles to practice. Health Policy 82 (2007) 37-50.

External Links





Case Data


General Issue(s): 
Specific Topic(s): 


McMaster University Hamilton , ON
Ontario CA


What was the intended purpose?: 


Start Date: 
Sunday, November 30, 2008
End Date: 
Monday, May 31, 2010
Number of Meeting Days: 


Total Number of Participants: 
Targeted Participants (Demographics): 
Targeted Participants (Public Roles): 
Method of Recruitment: 
Other: Method of Recruitment: 
Civic Lottery


If yes, were they ...: 
Facetoface, Online or Both: 
Type of Interaction among Participants: 
If voting...: 
[no data entered]
Other: Targeted Audience: 
Medical Advisory Secretariat
Ontario Health Technology Advisory Committee
Method of Communication with Audience: 


Who paid for the project or initiative?: 
[no data entered]
Who was primarily responsible for organizing the initiative?: 
[no data entered]
Who else supported the initiative? : 
[no data entered]


Total Budget: 
[no data entered]
Average Annual Budget: 
[no data entered]
Number of Full-Time Staff: 
[no data entered]
Number of Part-Time Staff: 
[no data entered]
Staff Type: 
[no data entered]
Number of Volunteers: 
[no data entered]


No discussions have been started yet.