I just returned from Baltimore where I participated in the PCORI Cycle II Merit Review process. I learned a lot about PCORI, remembered a lot of my undergrad research (which I thought I’d blocked out), and brought the patient voice to what has been traditionally a scientists only club.
PCORI is the Patient Centered Outcomes Research Institute a non-profit funded by the Patience Centered Outcomes Research Trust Fund. Established under the Affordable Care Act, PCORI is charged with identifying research priorities and a research agenda to address these priorities in order to
Assist patients, clinicians, purchasers, and policy-makers in making informed health decisions by advancing the quality and relevance of evidence concerning the manner in which diseases, disorders, and other health conditions can effectively and appropriately be prevented, diagnosed, treated, monitored, and managed through research and evidence synthesis that considers variations in patient subpopulations, and the dissemination of research findings with respect to the relative health outcomes, clinical effectiveness, and appropriateness of the medical treatments, services,…”
As such, PCORI reviews research proposals requesting funding. And uniquely this process includes patients as reviewers alongside scientists and other stakeholders (including clinicians, purchasers, payers, organizational providers, industry, researchers, industry, providers, policymakers, and caregiver organizations) and requires research to be patient-centered. In December 2012, I was invited to become a patient reviewer for PCORI’s second cycle of their review process. Patient being defined as any individual with a specific health condition, patient surrogates, or anyone at risk for a specific health condition.
The PCORI Merit Review Process:
This process was rather confusing, A LOT of work, and honestly & ironically not very patient-centered. I don’t remember how I was invited but once I accepted the invitation I was assigned to the panel for Improving Healthcare Systems. The others are Addressing Disparities; Assessment of Prevention, Diagnosis, and Treatment Options; Communication and Dissemination; and Accelerating Patient-Centered Outcomes Research and Methodological Research. These 5 comprise PCORI’s national priorities for research and encompass the patient-centered comparative clinical effectiveness research that the Institute will support.
After going through brief training presentation (found here), and told how to use the PCORI Online system – which is a MESS – (instructions here), I was assigned 7 proposals and asked to ensure I had no conflicts of interest (a PCORI requirement under the ACA). I was introduced to my PSRO Roycelynn Mentor-Marcel and my mentor Kim Bailey (who had been part of the Cycle I review process). I was given login information to yammer.com where all reviewers could post questions much like facebook (note: I HATE facebook). We were invited to “townhalls” which were never really clear to me. I attempted to attend one, but technical issues ruined the session.
Then I delved into the proposals. Not having done research in a long time, and knowing that many patients have never encountered these types of proposals, I was struck by how hard they were to read. Patients and stakeholders were only asked to evaluate 3 of 8 criterion including
- Innovation/Potential for Improvement (Refers to the potential that the proposed research may lead to meaningful improvement in patient health, well-being, or quality of care.)
- Patient-Centeredness (Is the proposed research focused on questions and outcomes of specific interest to patients and their caregivers?)
- Team and Environment (specifically – Are relevant patients and other key users of the study information (eg, caregivers, clinicians, health system, community, or policy makers) appropriately included on the team?)
The other criterion that we could comment on but were not required to include:
- Impact of the condition
- Impact on healthcare performance
- Rigorous Research Methods
- Inclusiveness of Different Populations
- Efficient Use of Resources
As well as the budget and human subjects protections.
While the training gave some examples of how to score on a scale of 1 to 9 (9 being the worst) and write out 3 strengths and 3 weaknesses as well as an overall narrative for each criterion, it was not helpful (none of the training was). I was particularly concerned when I read the example for scoring on the patient-centeredness criterion. This is what I sent PCORI:
I wanted to make one more comment. When discussing the fourth review criteria on patient-centeredness the low scoring example is a study measuring the effect of a new medication on Hemoglobin A1c and says that that this level is not important to many who have diabetes. I have Type 1 diabetes and this statement patently false. Hemoglobin A1c levels are the primary focus of all the care I’ve ever received for this autoimmune disease.
But I laboured on and I spent HOURS reviewing these applications and developing comments. Then as reviewers started submitting our comments and scores, we realized how awful the PCORI Online system is to use. The system would kick us off before we could push submit – so we had to make comments in a word document and cut and paste into the system or we would lose all our work. Though I pushed submit, sometimes the system just didn’t accept it. And of course, Roycelynn and Kim inevitably came back with each one saying my narrative was not sufficient enough (me the sesquipedalian…) and I had them all reset and had to go through the process again. At this point I was beyond frustrated. I was sick and stressed and trying to keep up with work.
Finally everything was submitted and then days before having to take off to Baltimore (which I didn’t realize I’d be doing until I got a random email about setting up travel arrangements) we were asked to go through 45 other applications for conflicts of interest (other review groups had different numbers of proposals). THEN we were asked to review another 18 reviews that received the highest scores from the patient, 2 scientists, and a stakeholder. Two of mine were in that top list so I didn’t have to read those (I was so relieved that some of mine didn’t make the list!), but we were asked to read the rest for the upcoming discussion. I couldn’t get through them all in time. I didn’t realize that they gave us the scores and comments of the reviewers on other applications so I didn’t review those either – I wouldn’t have time even had I known they were there.
At this point I was overwhelmed. Law school taught me to read a lot and quickly, but for me sixteen 120+ page proposals was too much. I was ready to quit because the commitment was interfering with my life. I was volunteering my time to do this because I am interested and dedicated to improving research and giving patients a voice.
The final part of the process was Baltimore. PCORI spent a lot to get the reviewers to Maryland for the panel discussions (not to be confused with advisory panels explained below) for 2 days. They spent Wednesday night “training” patients in an optional session and then “training” the panels (mine being the Improving Healthcare Systems panel B). Neither training – like the presentation – were helpful. There was a video that mostly spoke to the National Institutes of Health (NIH) process. There was a mock review. But ultimately, these didn’t prepare me for the next day.
Thursday was a very very long day. Starting at 9am and working straight through until 4:30, our panel discussed the 18 assigned to us. I was impressed by the discussions for each proposal. I thought that given the rapid fire input many salient points were made so that everyone in the panel could make an informed decision as to the final score they would submit. The discussions went as followed – (time is approximate) a few minutes for the first scientific reviewer, a few for the patient, a minute for the second scientific reviewer, a minute or so for the stakeholder, an opportunity to give input on the budget or human subject protections and then a few minutes for a general discussion. In the general discussion, mostly scientists were asking other scientists questions or talking in general. I did not find the patient voice (or stakeholders) truly respected for all discussions (again on my panel at least).
Then at the end of this exhaustive day I got home, slept 14 hours, and wrote my comments to PCORI. I’d like to share a few here.
As I said above, ironically this process was not patient-centered. The expectations were not clear from the beginning. I had no idea what I was in for – the time commitment alone was shocking.
I felt that PCORI did not know me as a patient though they invited me as a patient. They didn’t know that spending hours reading these reviews in addition to just getting by with my health and trying to work would be so stressful for me (in fact, during this time I developed pneumonia which I think was linked to this stress). They didn’t know that a full day without breaks in intensive discussions is too hard on my body. I need rest breaks to reset and do my own self-care. They didn’t know that I couldn’t pay for costs of travel upfront including transportation to and from the airport to the hotel and meals. I told my mentor and PSRO that I wasn’t sure I could afford to go (both paying the upfront costs and missing work for a very small honorarium) but they didn’t know what to do. Only in posting rather embarrassingly that I needed help and asking other reviewers if they could lend a hand did PCORI contact me with a solution (though once they found out, they were very helpful in arranging everything for me).
On top of this was the frustration of their attempts to silence my patient voice through social media. I was open with PCORI that I’d be live tweeting through the discussions. I explained I would not be attributing comments to anyone or talking about the proposals we were reviewing but that I would tell people about the process in general as it was happening. They asked me not to.
The ACA charges PCORI with ensuring transparency, credibility, and access. It specifically says that it needs to conduct forums to increase public awareness and obtain and incorporate input and feedback through media…on research priorities, research findings… and other activities or processes…” I read this to include the review process because the public has a right to know how the process works, particularly how patients are engaged in the process. Here are a few of my tweets (you can find all by following #PCORI):
And as a result of these tweets, I was able to engage advocate @ElinSilveous and @felixgreaves a London doctor. Through tweeting, the world was able to engage. But PCORI did not understand this and really didn’t engage until the very end through @PCORI – a real loss of opportunity. And their input would have been very important – especially as some interesting articles were then being published about the advisory panels (discussed below). There was confusion as to what we were doing, how the process works, who was involved, what this all meant. Honestly I was too and that was why I wanted to explain my experience through live tweeting and through a blog post.
I think PCORI is really open to patient feedback. I like to give them the benefit of the doubt as this is only the second time they’ve done this and no other organization has attempted to do so. But this process still clearly favours scientists. The discussions should have been divided between the two days. Patients should have a stronger voice (maybe it was different for other panels). Patients should have been understood on an individual level and patients needs to attend a conference should have been incorporated into the process and meeting. And social media should have been utilized to reach out to the public and to ensure transparency of the process. I hope these things will change in the future.
PCORI Advisory Panels:
Last week, PCORI announced the names of those who will serve on Advisory Panels. This news has really confused many about the roles patients play throughout the Institute – in other words who are the patients and what is the difference between the patients that serve in the merit reviews and on the panels.
(from the website) The Advisory Panels are standing panels that are required to include representatives of practicing and research clinicians, patients, and experts in scientific and health services research, health services delivery, and evidence-based medicine who have experience in the relevant topic and, as appropriate, experts in integrative health and primary prevention strategies. They are not official decision makers, but their recommendations and advice will be taken into consideration by our staff, Board, and Methodology Committee in:
- Modeling robust patient and stakeholder engagement efforts.
- Refining and prioritizing specific research questions.
- Providing other scientific or technical expertise.
- Providing input on other questions that may arise relevant to the Institute’s mission and work.
The Panels mirror the review panels:
- Advisory Panel on Addressing Disparities.
- Advisory Panel on Assessment of Prevention, Diagnosis, and Treatment Options.
- Advisory Panel on Improving Healthcare Systems.
- Advisory Panel on Patient Engagement.
(In the future, PCORI expects to create additional advisory panels.)
Panel members represent specific stakeholder groups mandated by Congress and are appointed for one year, but they can re-up for another term.
The problem is, that PCORI did not provide any relevant information on who the panelists are – individuals who will have a huge impact on the future of research in America. As Millenson points out:
PCORI isn’t a church, where all are created equal in the eyes of God, but a politically created, politically governed, controversial dispenser of a very large amount of money that a host of interest groups would like to control. PCORI staff chose the panel members in part by looking at their affiliations…
Only after this posting made waves throughout the patient advocate community, did PCORI respond. Their website now reads:
We’ll be providing affiliation information and short biographies on all confirmed panel members shortly.
A long way to go:
As I said, PCORI is still very much a mess (to be blunt) as it starts to establish itself, as we would expect of any organization establishing itself with such a large responsibility. They are open to feedback and are learning what patient-centeredness means beyond their assumptions. I am glad I could be part of the process – to understand what PCORI is doing and bring that information back to the public so that the Institute doesn’t just say it’s working on patient centered outcomes research but that it truly is involving patients in the review process, in the organization, and in the research that matters and will meaningfully impact patients. PCORI wants more patients involved but they really don’t know how to and need more input from the public. They have a long way to go to realize their mission but I think they are working very hard to do so.