UPDATE: I think I’ve found the method I’m going to go with. I’ve got a sample survey with corresponding data table here. I’d appreciate any feedback.
I really like the philosophy journal wiki, but it would be nice to have more automated system that presents the data in an easier to digest format. I’ve been planning on experimenting with the best way to do this for a while. Now is as good a time as any.
I’m adding a page to this blog called “Journal Surveys”. Any journal that I create a survey post for will automatically show up on that page. Survey posts for a journal ask people to provide four pieces of information on a single paper that they submitted to that journal: Initial Verdict, Review Time, Comment Quality, and Verdict after a Revise and Resubmit (if an R&R was given in the initial verdict).
There is no restriction on the number of times you may take the survey, since philosophers (of course) submit to the same journal more than once. To take the surveys for a journal again simply refresh your browser.
I’m starting with just a few journals this week. My hope is that by next week or the week after, I’ll be able to launch with some 90+ journals. However, before I commit to putting in the time to generate survey pages for some 90+ journals, I’d like to figure out the answers to (a)-(c).
(a) What is the best method to administer/display the surveys?
(b) What are the most useful questions to ask?
(c) For each question, what are the best/most useful answer options to include?
Launching with a few journals will enable me to see how this is going to work, and it will enable other philosophers to give me feedback. Once I work the kinks out, I have a fairly automated format to generate the other survey posts, and populate the list on the Journal Survey page.
To take a survey (or see the results) click on the Journal Survey page. Then click on one of the journals.
Take a look at the way the surveys are set up. Are there any questions/answers you think should be there? Are there any questions/answers you think shouldn’t be there? Are there any questions/answers that should be clarified? I want to get these kinds of details hammered out now. Once I do that, I can quickly generate a bunch of survey posts.
I’d appreciate any feedback. Hopefully, the philosophical community will find this to be useful.
I, for one, like the idea and hope you run with it.
As to (a), it might be best to give the overview information graphically–perhaps as percentages of respondants who gave each of a number of multiple choice answers for each question for each journal, with the total number for each journal very prominant. Then, perhaps a link to a sub-page with all the qualitative feedback (e.g., further comments).
Here are some non-exhaustive questions for (b):
~comments: none, limited, fairly helpful, very helpful
~time for initial verdict: 12 months
~maybe also questions on how responsive the editor was to questions and queries, initial verdict, final verdict, etc…
For (c), I guess I’ve already done this a bit for some of the questions in (b).
Let me know if you want any help on the project.
Kevin,
Thanks for the input and the offer to help.