Summary statement II.

[rivka]

I got the summary statement back for my grant re-submission. This time, in addition to two long reviewers' critiques, I got a "resume and summary of discussion" which addresses what was said about my project at the study section meeting.

Here's the money quote: "This is an innovative application junior investigator with a promising research track record. The applicant has been moderately responsive to the converns raised in the prior review of this application and the present study is much improved. [...] However, this continues to be an ambitious project and the committee was concerned about the feasibility of the project. This concern and other concerns reflected in the individual critiques served to limit the committee's enthusiasm for the proposed study."

So that's that.

Reviewer 2 liked me a lot more than Reviewer 1 did. Here's my favorite part of Critique 2: "The investigator appears well-qualified to conduct this project; the size and scope of the proposed study is commensurate with her experience to date. She has developed a very solid early career publication record and is very definitely a rising investigator. This study would represent an important next step for her career development." And Reviewer 2's overall evaluation: "This is a very strong exploratory/developmental project proposed by a junior investigator with a strong research track record. Some concerns are raised about feasibility and practical implications, but these do not detract substantively from what is viewed as a very strong re-submission."

Reviewer 2's biggest concern: "It would be unfortunate if a project of this import collapsed due to it not being truly feasible."

Reviewer 1 wants even yet still more theoretical conceptualization and integration. He or she also has some problems with my data analysis plan and suggests that I consult with a statistician. Reviewer 1 does say: "This research is highly significant and has clear public health relevance." So that's nice. Reviewer 1's overall evaluation: "This is an application for an interesting and potentially important study [...] Lack of a clear specification of core schemas hypothesized to exist in the study population and an explication of how those core schemas influence irrational beliefs contribute to difficulties in evaluating the likely outcome of the study."

I'm pretty much positive that I won't be funded this go-round. And it's going to take some serious thought, and a consultation with the Program Officer, to figure out what I should do with the next resubmission. Addressing "concerns about feasibility" usually means making the study smaller and less ambitious, but that is likely to compromise the scientific quality. (For example, enrolling fewer subjects results in less statistical power to detect effects.) It's going to be touchy.

Comments

[bosssio.livejournal.com]

Very interesting feedback. And take this as a very good indication of things to come - you are pushing the envelope, you have identified important topics to be addressed, and they think you can do it.

how about some language about how you would be able to use the results if the study were to "collapse" -i.e. not go as far as you expect it too? That even a partial study is more valuable than no study at all.

I think going forward with a "pilot" study - to test the feasibility, to run through the logistics, get the stats sorted, test whether your abstract approach works in a real life scenario, etc. You know that fewer participants impacts scientific quality, but what you are really doing is setting up the system now - testing it, honing it - so you will eventually get the study that you want to perform. And that way, you can think bigger too - set up the parameters of the study for this particular question, but think about other questions you want to ask of this same population - nuances and variations. Once you have set up your system - and proven that it gets you scientifically valid data - you can potentially reuse it to expand the population, expand the question set, etc.

Just some thoughts...

[ailbhe]

I don't have the background to address this practically or usefully, but I want to say that I think it's obvious you're working on really important things here (and that the reviewers think so) and I hope you get to actually do the study somehow.

[eeyorerin]

Well damn. "Feasibility" is such a hard thing to address. It'd be a lot easier if they just hated your data analysis or theoretical approach.

[therealjae.livejournal.com]

Man, your whole process has been such a deja vu thing for me. I still get "ambitious project that she may not be able to finish" and "more unifying theoretical framework, please," though it's no longer enough to get my grants rejected.

Do they spell out at any point exactly what they think isn't feasible? I mean, if it's just the number of participants, that's easy to fix (I see your point about that, but I had to do that with my first grant, and it worked out all right). But if it's more like that they think your project has too many facets, that will require a whole different kind of tweaking.

It sounds like you're learning from the process, though, as I have too. That's good!

-J

[rivka.livejournal.com]

It's encouraging to hear that I'm following the same path you did, because your career has been so successful. And I'm sure it was your lack of theoretical framework and overambitious planning that got you there! ;-)

After talking it over with my co-worker/assistant/friend Steve, I'm trhinking that I might keep the sample size the same (or maybe slightly smaller), and cut out one of the follow-up points. I had been planning to do 6- and 12-month follow-ups, and upon further reflection that may indeed be too much for a grant of this length. And a study that's shorter than it could be is probably a lot better than a study that's too small to detect moderate effects.

I'll need to run it by the Project Officer and see what she thinks, though. She was actually there when my proposal was discussed at the study section, so she may have more information about what exactly people thought was infeasible.

I just wish they'd raised feasibility concerns in the first review, because I could have fixed it on the last go-round. Argh.

[therealjae.livejournal.com]

I'm really irritated on your behalf that they didn't spell out what they thought wasn't feasible. That's supremely lazy and unhelpful. And you're right to be annoyed that they didn't raise feasibility concerns in the first review, but in their defense, I suspect it's because the feasibility concerns weren't visible because of the lack of a full-fledged theoretical framework in the first version. That happened to me once, too, in exactly that order.

What I've learned over the years is that the best way to avoid the kinds of errors that mask other errors is to show successive drafts of your grant proposal to everybody and their brother. Scholars in your field, scholars not in your field, your dog, your mom...stop at nothing. And beg them to be BRUTAL, or at least to point out anything they have even the slightest concerns about. I'm utterly convinced that this is the only way my last grant application got funded on the first try. I no longer have any shame about this!

I'm amused that you think my career has been "so successful." I'd personally rate myself as a decent, but unspectacular academic. What I truly excel at, though, is balancing work with...well, life. And I'd much rather be successful in that way than in the conventional one. :-)

-J

[jenett]

On the feasability thing:

I like the idea of a tiered proposal: "We hope to enroll X number of participants, because with that number, we can accomplish things A, B and C. However, if only Y participants are feasible (for reasons Q, Z, and W), we would be able to address these aspects of thing A and portions of C."

Another possibility would maybe to do the "We believe this project is feasible, because we already see X number of people fitting the requirements every week/month/quarter, and the requirements of this project would add Y time and Q other resources which we have done in similar form." (In other words, whatever other relevant data that demonstrates you have manpower, space, ability to track, an existing population you know is there, or whatever else they had questions about.)

[wcg.livejournal.com]

It's enough to induce tail-biting in rats, isn't it? Getting turned down because your proposal is *too* good.

Ah well, better luck next time around.

[rivka.livejournal.com]

Getting turned down because your proposal is *too* good.

Where on earth are you getting that?

[wcg.livejournal.com]

I got the impression they think you're promising more than you can deliver.

[rivka.livejournal.com]

Well, yeah. I guess I just don't make the jump from there to "too good." Now, if I had already done it, and they didn't believe me, I'd call that being rejected for being "too good." But the jury's still out on whether my proposed project is actually feasible.

[tassie-gal.livejournal.com]

GAH! This is what I am giving a presentation about on Wednesday - but geared towards ethics committees and guidelines for epidemiologists.
I strongly suspect that around the world reviewers have a check list. That check list includes feasability totally unrelated to statitical significance and sample size. Despite the fact that generally the three of them go hand in hand - and especially in your type of longitudinal cohort studies.
Good luck with it anyway. Sometimes proposing a small "tester" study and getting funding through a small NGO or something THEN apply to the big guys with a "we did this and saw this but the numbers arent big enough to draw nice conclusions can we please have some money to make this bigger" works.