My Experience as a PCORI Patient Merit Reviewer

I just returned from Baltimore where I participated in the PCORI Cycle II Merit Review process.  I learned a lot about PCORI, remembered a lot of my undergrad research (which I thought I’d blocked out), and brought the patient voice to what has been traditionally a scientists only club.


PCORI is the Patient Centered Outcomes Research Institute a non-profit funded by the Patience Centered Outcomes Research Trust Fund.  Established under the Affordable Care Act, PCORI is charged with identifying research priorities and a research agenda to address these priorities in order to

Assist patients, clinicians, purchasers, and policy-makers in making informed health  decisions by advancing the quality and relevance of evidence concerning the manner in which diseases, disorders, and other health  conditions can effectively and appropriately be prevented,  diagnosed, treated, monitored, and managed through research and evidence synthesis that considers variations in patient subpopulations, and the dissemination of research findings with respect to the relative health outcomes, clinical effectiveness, and appropriateness of the medical treatments, services,…”

As such, PCORI reviews research proposals requesting funding. And uniquely this process includes patients as reviewers alongside scientists and other stakeholders (including clinicians, purchasers, payers, organizational providers, industry, researchers, industry, providers, policymakers, and caregiver organizations) and requires research to be patient-centered. In December 2012, I was invited to become a patient reviewer for PCORI’s second cycle of their review process. Patient being defined as any individual with a specific health condition, patient surrogates, or anyone at risk for a specific health condition.

The PCORI Merit Review Process:

This process was rather confusing, A LOT of work, and honestly & ironically not very patient-centered.  I don’t remember how I was invited but once I accepted the invitation I was assigned to the panel for Improving Healthcare Systems.  The others are Addressing Disparities; Assessment of Prevention, Diagnosis, and Treatment Options; Communication and Dissemination; and Accelerating Patient-Centered Outcomes Research and Methodological Research.  These 5 comprise PCORI’s national priorities for research and encompass the patient-centered comparative clinical effectiveness research that the Institute will support.

After going through brief training presentation (found here), and told how to use the PCORI Online system – which is a MESS – (instructions here), I was assigned 7 proposals and asked to ensure I had no conflicts of interest (a PCORI requirement under the ACA).  I was introduced to my PSRO Roycelynn Mentor-Marcel and my mentor Kim Bailey (who had been part of the Cycle I review process).  I was given login information to where all reviewers could post questions much like facebook (note: I HATE facebook).  We were invited to “townhalls” which were never really clear to me.  I attempted to attend one, but technical issues ruined the session.

Then I delved into the proposals.  Not having done research in a long time, and knowing that many patients have never encountered these types of proposals, I was struck by how hard they were to read.  Patients and stakeholders were only asked to evaluate 3 of 8 criterion including

  • Innovation/Potential for Improvement (Refers to the potential that the proposed research may lead to meaningful improvement in patient health, well-being, or quality of care.)
  • Patient-Centeredness (Is the proposed research focused on questions and outcomes of specific interest to patients and their caregivers?)
  • Team and Environment (specifically – Are relevant patients and other key users of the study information (eg, caregivers, clinicians, health system, community, or policy makers) appropriately included on the team?)

The other criterion that we could comment on but were not required to include:

  • Impact of the condition
  • Impact on healthcare performance
  • Rigorous Research Methods
  • Inclusiveness of Different Populations
  • Efficient Use of Resources

As well as the budget and human subjects protections.

While the training gave some examples of how to score on a scale of 1 to 9 (9 being the worst) and write out 3 strengths and 3 weaknesses as well as an overall narrative for each criterion, it was not helpful (none of the training was).  I was particularly concerned when I read the example for scoring on the patient-centeredness criterion.  This is what I sent PCORI:

I wanted to make one more comment.  When discussing the fourth review criteria on patient-centeredness the low scoring example is a study measuring the effect of a new medication on Hemoglobin A1c and says that that this level is not important to many who have diabetes.  I have Type 1 diabetes and this statement patently false. Hemoglobin A1c levels are the primary focus of all the care I’ve ever received for this autoimmune disease.

But I laboured on and I spent HOURS reviewing these applications and developing comments.  Then as reviewers started submitting our comments and scores, we realized how awful the PCORI Online system is to use.  The system would kick us off before we could push submit – so we had to make comments in a word document and cut and paste into the system or we would lose all our work.  Though I pushed submit, sometimes the system just didn’t accept it.  And of course, Roycelynn and Kim inevitably came back with each one saying my narrative was not sufficient enough (me the sesquipedalian…) and I had them all reset and had to go through the process again.  At this point I was beyond frustrated.  I was sick and stressed and trying to keep up with work.

Finally everything was submitted and then days before having to take off to Baltimore (which I didn’t realize I’d be doing until I got a random email about setting up travel arrangements) we were asked to go through 45 other applications for conflicts of interest (other review groups had different numbers of proposals).  THEN we were asked to review another 18 reviews that received the highest scores from the patient, 2 scientists, and a stakeholder.  Two of mine were in that top list so I didn’t have to read those (I was so relieved that some of mine didn’t make the list!), but we were asked to read the rest for the upcoming discussion.  I couldn’t get through them all in time.  I didn’t realize that they gave us the scores and comments of the reviewers on other applications so I didn’t review those either – I wouldn’t have time even had I known they were there.

At this point I was overwhelmed.  Law school taught me to read a lot and quickly, but for me sixteen 120+ page proposals was too much.  I was ready to quit because the commitment was interfering with my life. I was volunteering my time to do this because I am interested and dedicated to improving research and giving patients a voice.

The final part of the process was Baltimore.  PCORI spent a lot to get the reviewers to Maryland for the panel discussions (not to be confused with advisory panels explained below) for 2 days.  They spent Wednesday night “training” patients in an optional session and then “training” the panels (mine being the Improving Healthcare Systems panel B).  Neither training – like the presentation – were helpful.  There was a video that mostly spoke to the National Institutes of Health (NIH) process.  There was a mock review.  But ultimately, these didn’t prepare me for the next day.

Thursday was a very very long day.  Starting at 9am and working straight through until 4:30, our panel discussed the 18 assigned to us.  I was impressed by the discussions for each proposal.  I thought that given the rapid fire input many salient points were made so that everyone in the panel could make an informed decision as to the final score they would submit.  The discussions went as followed – (time is approximate) a few minutes for the first scientific reviewer, a few for the patient, a minute for the second scientific reviewer, a minute or so for the stakeholder, an opportunity to give input on the budget or human subject protections and then a few minutes for a general discussion. In the general discussion, mostly scientists were asking other scientists questions or talking in general.  I did not find the patient voice (or stakeholders) truly respected for all discussions (again on my panel at least).

Return Home:

Then at the end of this exhaustive day I got home, slept 14 hours, and wrote my comments to PCORI.  I’d like to share a few here.

As I said above, ironically this process was not patient-centered.  The expectations were not clear from the beginning.  I had no idea what I was in for – the time commitment alone was shocking.

I felt that PCORI did not know me as a patient though they invited me as a patient.  They didn’t know that spending hours reading these reviews in addition to just getting by with my health and trying to work would be so stressful for me (in fact, during this time I developed pneumonia which I think was linked to this stress).  They didn’t know that a full day without breaks in intensive discussions is too hard on my body.  I need rest breaks to reset and do my own self-care.  They didn’t know that I couldn’t pay for costs of travel upfront including transportation to and from the airport to the hotel and meals.  I told my mentor and PSRO that I wasn’t sure I could afford to go (both paying the upfront costs and missing work for a very small honorarium) but they didn’t know what to do.  Only in posting rather embarrassingly that I needed help and asking other reviewers if they could lend a hand did PCORI contact me with a solution (though once they found out, they were very helpful in arranging everything for me).

On top of this was the frustration of their attempts to silence my patient voice through social media.  I was open with PCORI that I’d be live tweeting through the discussions.  I explained I would not be attributing comments to anyone or talking about the proposals we were reviewing but that I would tell people about the process in general as it was happening.  They asked me not to.

The ACA charges PCORI with ensuring transparency, credibility, and access.  It specifically says that it needs to conduct forums to increase public awareness and obtain and incorporate input and feedback through media…on research priorities, research findings… and other activities or processes…”  I read this to include the review process because the public has a right to know how the process works, particularly how patients are engaged in the process.  Here are a few of my tweets (you can find all by following #PCORI):

Erin Gilmer@GilmerHealthLaw 4 Apr : Definitely learning a lot in these reviews. Love the different perspectives. #PCORI

Erin Gilmer@GilmerHealthLaw 3 Apr: #PCORI panel orientation wasting time on issues not relevant to whole panel. not sure point of this. no new info to learn. @PCORI

Erin Gilmer@GilmerHealthLaw 3 Apr: glad to meet @chrisfriese_rn and @msbrowntatum @PCORI review! #PCORI

And as a result of these tweets, I was able to engage advocate @ElinSilveous and @felixgreaves a London doctor.  Through tweeting, the world was able to engage.  But PCORI did not understand this and really didn’t engage until the very end through @PCORI – a real loss of opportunity.  And their input would have been very important – especially as some interesting articles were then being published about the advisory panels (discussed below).  There was confusion as to what we were doing, how the process works, who was involved, what this all meant.  Honestly I was too and that was why I wanted to explain my experience through live tweeting and through a blog post.

I think PCORI is really open to patient feedback.  I like to give them the benefit of the doubt as this is only the second time they’ve done this and no other organization has attempted to do so.  But this process still clearly favours scientists.  The discussions should have been divided between the two days.  Patients should have a stronger voice (maybe it was different for other panels). Patients should have been understood on an individual level and patients needs to attend a conference should have been incorporated into the process and meeting. And social media should have been utilized to reach out to the public and to ensure transparency of the process. I hope these things will change in the future.

PCORI Advisory Panels:

Last week, PCORI announced the names of those who will serve on Advisory Panels.  This news has really confused many about the roles patients play throughout the Institute – in other words who are the patients and what is the difference between the patients that serve in the merit reviews and on the panels.

(from the website) The Advisory Panels are standing panels that are required to include representatives of practicing and research clinicians, patients, and experts in scientific and health services research, health services delivery, and evidence-based medicine who have experience in the relevant topic and, as appropriate, experts in integrative health and primary prevention strategies. They are not official decision makers, but their recommendations and advice will be taken into consideration by our staff, Board, and Methodology Committee in:

  • Modeling robust patient and stakeholder engagement efforts.
  • Refining and prioritizing specific research questions.
  • Providing other scientific or technical expertise.
  • Providing input on other questions that may arise relevant to the Institute’s mission and work.

The Panels mirror the review panels:

(In the future, PCORI expects to create additional advisory panels.)

Panel members represent specific stakeholder groups mandated by Congress and are appointed for one year, but they can re-up for another term.

As I mentioned, with the release of these names, the by the public started wondering who they are.  Michael Millenson wrote on The Health Care Blog a piece on why the PCORI picks matter.

He explains:

The problem is, that PCORI did not provide any relevant information on who the panelists are – individuals who will have a huge impact on the future of research in America.  As Millenson points out:

PCORI isn’t a church, where all are created equal in the eyes of God, but a politically created, politically governed, controversial dispenser of a very large amount of money that a host of interest groups would like to control. PCORI staff chose the panel members in part by looking at their affiliations…

Only after this posting made waves throughout the patient advocate community, did PCORI respond.  Their website now reads:

We’ll be providing affiliation information and short biographies on all confirmed panel members shortly.

Find out more about how we selected our advisory panels here. Additional information is in our list of frequently asked questions.

A long way to go:

As I said, PCORI is still very much a mess (to be blunt) as it starts to establish itself, as we would expect of any organization establishing itself with such a large responsibility.  They are open to feedback and are learning what patient-centeredness means beyond their assumptions.  I am glad I could be part of the process – to understand what PCORI is doing and bring that information back to the public so that the Institute doesn’t just say it’s working on patient centered outcomes research but that it truly is involving patients in the review process, in the organization, and in the research that matters and will meaningfully impact patients.  PCORI wants more patients involved but they really don’t know how to and need more input from the public.  They have a long way to go to realize their mission but I think they are working very hard to do so.


24 Responses to My Experience as a PCORI Patient Merit Reviewer

  1. Melissa says:

    That is somewhat disappointing to hear – that the process was both confusing and relatively un-patient-centered. While disappointing, I can’t say that it is surprising. The PCORI website, documentation, and process I found quite confusing and I’ve practiced law for 15 years, healthcare law at that, and been involved as a patient advocate for the last few. In some ways, I almost see it as an “emperor wears no clothes” scenario where many are confused, frustrated, and the like but few will stand up and admit what you have in your post. Kudos to you. And kudos to PCORI if they take in these thoughts and recommendations and act on them.

  2. Precisely, even as a lawyer with 3 years of intense scientific review experience, this was quite tough for me. I do think that PCORI will make progress, have to be patient, but I want to be a voice to help them along the way!

  3. I have to post a twitter comment by Susannah Fox of the Pew Research Center who will soon be teaching a master class at Stanford on participatory medicine (go here to see how she’s starting the discussion to include in the course –

    Susannah Fox @SusannahFox
    “Lab rat talking back” came to mind while reading about @GilmerHealthLaw’s experience as a PCORI reviewer:

  4. e-Patient Dave says:

    Erin, this is an *extraordinary* post. Thank you so much for putting in the time on this!

    My own experience with PCORI resonates pretty well with yours. It seems clear, as you say, that they’re working hard at it, but they just don’t quite get it yet. Here’s to solving it – here’s hoping their efforts improve things dramatically – because if this organization gets itself firmed up without seriously changing things, the whole thing will be a squandered opportunity.

    I know they want to change. But my impression is that they aren’t very accomplished yet at listening to US, vs listening to the forces of the establishment.

    And really, PCORI folks – that’s all it would take – listen to the patients who say “No, you’re on the right track but you’re just not getting it yet.” Otherwise it’s exactly like all the hospitals who THINK they’re being patient-centered, even though their customers don’t agree.

    C’mon. We can change this thing.

    Thanks again, Erin. This is a terrific, valuable post.

  5. Received today! In an email to the reviewers PCORI sent out this morning they included the following that I think is important to share:

    “As we conclude this cycle of PCORI merit review, we want to extend our deepest gratitude to you for participating as a reviewer. We are extremely proud of the work you accomplished this past week in Baltimore—and of the many hours of careful review and preparation you spent prior to arriving in Baltimore.

    During Cycle II merit review, 413 applications were reviewed, by 223 reviewers, who submitted 1644 critiques. PCORI could not have done this without you!”

  6. Yesterday – PCORI finally released the bios of those on their Advisory Panels. They can be found at

  7. Thank you Laura! I thought I was the only one who was overwhelmed by the workload. I am also an attorney by training and spent at least 5 hours reviewing all the proposals I was assigned. I got ill just before I was going to start reviewing all the additional proposals we were supposed to read before going to Baltimore for the group review but did not think I could have devoted enough time to have done as thorough of a job as I would have liked.

    It does seem pretty unreasonable to only allow the patient to have perhaps a few minutes and the advocate to have many a minute for each of the reviews after all the hours and hours that were spent preparing.

    If patients (many of us may have limited stamina, health limitations, and other commitments) are to have meaningful participation, it does not seem reasonable for the review process to be a stamina contest that sounds MUCH more formidable than anything most of us regularly expose ourselves to.

    I am glad I am not the only one who does not feel there is inadequate preparation on how to gauge these proposals and no good examples of what might move a proposal from one number (say an 8) to another, say a 6 or a 5.

    I was unable to attend the group review because I was ill and relapsed before I was supposed to have attended the group review. It sounds like the grueling schedule would have guaranteed me a further and worse relapse.

    If future group reviews will be run similarly to this one and the workload will be similar, it does not sound like something that will work for my health and well-being. Getting a huge number of approximately100+ page proposals shortly before attending a group review meeting is wholly unreasonable to me. It also makes me wonder how much PATIENTS are really being considered, considering this is the PATIENT Centered Outcome Research Institute.

  8. Susan says:


    Thank you for this wonderful post. I was also a reviewer for cycle II, but was not able to attend the in-person review sessions due to illness (some of which I also contribute to stress caused by trying to complete the reviews).

    I was a reviewer as a patient. I have astrocytoma garde II/III (brain cancer). I was only able to complete 3 out of 6 study reviews. Like you, I had no idea the insane time committment that would be required to participate or that payment would need to be made up front (not a realistic expectation for people like me who are paying almost $500/month for co-pays for needed medications).

    I became very sick during the review process and therefore could not complete my reviews. I will say that Rocelynn and Kim were wonderful and understanding.

    My hope is that PCORI takes your feedback to heart. I also think it is fabulous that they are working hard to include patients in the process, but that they probably should sit down with patients and/or advocates with chronic conditions to learn more about what are realistic expectations for feedback from the patient communities.

    Thanks for speaking up!

  9. Great post Erin. I am sorry it seems they have so far to go. Let’s work for that change. I hope they invite you in and back to help design it better.

    It’s funny because we designed a judging process for a CrowdsCure contest the other day that does much of the same as the reviewer panel does by experts including patients and while complex, it was designed to be online and automated and take 20 minutes per review, a “crowd judging” as it were by patients, doctors, partners, and leaders in that medical care space. That client decided not to go live with that portion of the contest, but we look forward to showing it to the world soon. Looks like PCORI could use it.

    Keep up the fight for patient rights! You are a true Patient Hero!

  10. After hearing that other reviewers from Cycle II had been asked back to engage in Cycle III, I realized that I may have been cut off from the process all together because of my blog post here and because of my personal comments to them via email. So today, I sent this email to and my mentor and PSRO:

    I just wanted to follow up on my comments from my experience as a PCORI II cycle reviewer. I feel like I never really received any feedback and am worried I am now shut out of the organization because of my comments to you all through email and through my blog posting or tweets. When we came to PCORI on Wednesday night, we were told that we’d always be part of the PCORI family, but I don’t feel that way at all. I know cycle III is about to start, but I wasn’t invited. I also haven’t heard how you all are making improvements on the process. I have so many people asking me about PCORI – how we can improve it and make it work as its meant to under the ACA. I have received comments and emails from Yammer and on my blog from people near and far, those involved and those who didn’t know about this endeavor. People want to know about PCORI, are interested in it’s success but what do I say if I feel that anything negatively said about PCORI means that all engagement is cut off.

    So I would like to follow up, hear what other comments you received, how you are improving the process to make it more patient-centered for reviewers and recognizing them for their hard work, how you are engaging more patients from across the country, and how you will continue to include those who have served PCORI. I would also like to hear more about PCORI intends to engage in social media – I have begged and yet no response and I find that off putting. The law MANDATES that you be open and transparent, but I don’t feel that is happening.

    I look forward to hearing from you.

  11. cindy says:

    Not suprised –any time politics, govt, is involved–a complete mess and bias/agendas/pork projects prevails. IF so ‘patient-“centered”–why TWO “scientific” reviewers–enough of the egg head research and more real and practical evaluations to help the terrible mess this country is in.

  12. Tammi says:

    more of nothing. These “grant” “programs” are nothing more than political pork, crony fest and back slapping.

    puffery, self justification, and denial of the obvious…the american way.

  13. Sally Crowe says:

    Hi there Erin

    E Patient Dave directed me to your blog (we met a few weeks ago at a UK meeting) and I am really impressed with the detail of your post and how useful your comments will be to those organisations wanting people to review/appraise research material, they need to understand what they are asking people to do and what might be needed to support them in this task and then keep the feedback loop open and check in on how it is going. I am certainly going to direct some people to your blog! Best Sally

    • Thank you, Sally for listening/reading. I really hope my experiences can make a difference for the PCORI and other similar undertakings. Unfortunately, the feedback loop since with PCORI has been closed to me. But I will try to remain optimistic that they will continue to improve their work to make a difference for all patients.

  14. Sara says:

    The whole PCORI process and this tax payer extortion back schlap organization is nothing more than smoke, mirrors, and the illusion of american “health care” “innovation.” all to give the perception of change, or something “unique.” it’s mostly geared for those in denial, delusional, and willfully blind–they can feel exceptional that they are rewriting or “chaning” american health “care.” good luck on the titanic.

  15. Taby A. says:

    It is a dog and pony show. If the US govt was truly concerned about creating a real health care system like practically every other big boy country–they would have already do so. They wont. the country is predominately about money and money and illusions. The US is the only 1st world country whereby “health” “care” is FOR profit–even the warm/fuzzy “non profits’ act like FOR profits. It is principally about making money off of the sick and the infirm. So, perhaps, PCORI et al can give the illusion of “change” or “patients at the helm” in reality, it is a dog and pony show, and for bureaucrats to look important. Sad.

  16. Sunny Guttormsen says:

    went as a reviewer to. all they did was talk, talk talk. all these academic professor types, talking alot. sounded good, however like salesmen. where were the patients and caregivers in all of this. they sure talk about them, however, where’s the beef? where’s the real world real change?

  17. […] of who I am and throughout my life that attribute has tended to lead to negative consequences.  Being critical of PCORI meant that they have excluded me from the community.  Being critical of hospitals that made severe […]

  18. […] initiatives that tout their “inclusivity.” Three and a half years ago, I wrote how the Patient Centered Outcomes Research Institute (PCORI) was not very patient-centered at all and all the barriers that existed to get to the event and […]

  19. Sara says:

    Curious if the pcori bureaucrats have vastly improved the failed ” merit ” review process, or more of nothing? also, do they even pay you guys to review? 40+pgs of hard to read stuff, plus write ups, etc…huge time committment for what really–a paid trip to be used and abused? Any improvements from these folks–or just more deaf ears and pc proclamating?

    • I have not personally heard any more regarding the process as they effectively ex-communicated me for writing this. Though I tried to stay in touch and work with the administrators of the program, they did not take kindly to criticism. We did get a small honorarium. It was not enough to even begin to cover the amount of time spent on application review or the hours spent traveling to and from and sitting in the meeting rooms. I haven’t heard of many patients wanting to get involved with the process and even a few who wholeheartedly agreed with my assessment, so I’m guessing changes have not been made. It’s truly unfortunate.

  20. Sara says:

    Thank you for your kind note Ms. Gilmer. Appreciate your passion, wisdom, and authenticity for real change for patients and their caregivers. Not surprising there was no communication from pcori regarding your experiences and critical input. Bureaucrats do not like anything except happy talk and smiles–master BS artists who are dependent on american grant money (taxes) to fund their infantile mainly lives with artificial promises, sales pitches, and positive pop psychology, Recycled PPTs, and redundant meetings are their steady diet.

    It’s the US–pcori appears to be more of nothing–lots of promises, hand waving, and somedays from the top down. The affiliations of the pcori folks is a well connected bunch all clamoring for grant monies. Hustlers and opportunists with an appetite for more, and more, and more. Insatiable. American greed.

    The fact that pcori made you persona non grata for stating your realities speaks volumes on that farcical “patient-centric” org.

    What I do not understand is that if this organization wants patients and caregivers to truly and authetically participate (NOT window dressing) then how can these PHDs expect patients who are sick to review 50pgs+ of hard to read, fly in/out, hotels, read through tons of abstract PHD level studies, using computer systems that are arcane, dozens of hours and travel spent on these academic studies for a tiny honorarium and to be marginalised by academic elites living in a fantasy world?!.

    What then is pcori exactly!? another us govt bloated pork project, or something that was used to get the illusion of change for political opportunism? Sickening–so sorry Ms. Gilmer–keep doing what you’re doing–it is NOT unnoticed. Call em to task, and name names–we pay their salaries! They work for the populace.

Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: