Report to Governing Board of PSA on PSA2004 Program

Sandra Mitchell, PSA2002 Program Chair, passed on to me copies of many documents used in planning the program, and her report to the Governing Board. She was also available for questions about procedures. I would be happy to do the same for the PSA2006 Program Chair.

Doing the work of Program Chair was mostly interesting and worthwhile; please read the comments below while bearing in mind that I am grateful for the opportunity and would have agreed to do it even if I had known how much work it was going to be! Because I care about PSA even more now than I did two years ago, I want to pass on some experiences and ideas.

Infrastructure

PSA2002 successfully used, for the first time, web-based interactive software for planning conferences. Sandy Mitchell was given some teaching release and a research assistant, David Miller, who adapted the software Confman to PSA’s needs.

Temple University could not give me much support (no research assistants or teaching reductions), but the Dean of CLA, Susan Herbst, gave a one-time grant of $2000 to pay for technical assistance in setting up the interactive conference website. A philosophy graduate student, Micah Sparacio, worked with me on finding suitable conference software; Confman was not suitable for technical reasons. We settled on OpenConf (see important features below), and Micah hosted a dedicated website (http://www.psa2004.org [broken]) throughout 2003 and 2004.

I produced and managed the conference website at http://www.temple.edu/psa2004 [broken] and, ultimately, linked with the Phil-Sci Archive at the University of Pittsburgh in making available the accepted papers.

Important Features of OpenConf

OpenConf uses rankings along several dimensions. Rankings are (1) overall recommendation (from 1 to 6, with 6 being the best) (2) balance of conceptual and empirical (no judgment about which is better!) (3) new contribution to the field and (4) reviewer familiarity with the subject matter. Reviewers can, but do not have to, put in comments to the program committee and/or comments to the author(s). The system calculates the mean reviewer score for overall recommendation.

After the first stage of reviewing is complete, the program chair can unlock the system so that everyone can read everyone else’s evaluations, and change their opinion if desired. (Speaking as a social epistemologist, this method of group decision making, which anchors first at individual’s independent opinions, is one of the better ones, empirically.)

OpenConf processes submissions for blind review. Blind review of symposia and workshops is not possible (names are mentioned in the content of the proposal), but all papers were reviewed blind. All decisions about acceptance and rejection were made before the authors of papers, or any demographic facts about them, were known.

Review Procedure for Symposia and Workshops

Each proposal was read and evaluated by every program committee member (except that I did not make evaluations). After everyone had completed their evaluations, I unlocked the system so that everyone could read everyone else’s evaluations and vote again. We then accepted all symposia and workshops with scores exceeding 4.4. There were 13 such proposals (11 symposia and 2 workshops). Other symposia and workshops with high scores (all exceeding 4.0) were accepted based on the need to create balance in the program. Some symposia with scores between 4.0 and 4.4 were accepted as workshops rather than as symposia.

Results for Symposia and Workshops

Submitted Accepted Accepted as Workshop

USA, symposia 24 10 5 (4)*

International, symposia 13 4 3

(including Canada)

TOTAL, symposia 37 14 8 (7)*

USA, workshops 4 2

International, workshops 4 2

TOTAL, workshops 8 4

TOTAL

Symposia + workshops 45 19 (18)*

*One symposium was accepted as a workshop, but the participants declined the invitation to present as a workshop.

Comparison with PSA 2002: PSA 2002 had 33 submitted proposals (I do not have figures for the number of symposia and the number of workshops submitted), 14 were accepted as symposia (3 international) and 7 as workshops (1 international). So the number of submissions is up (from 33 in 2002 to 45 in 2004), and the number of international acceptances is higher (from 4 in 2002 to 6 in 2004). The overall acceptance rate is lower (from 64% in 2002 to 42% in 2004).

Review Procedure for Papers

Each paper was read and evaluated by two committee members, who were typically experts on the topic of the paper. Papers with an average score of 5.0 or above were automatically accepted. There were 24 such papers. Papers with an average score below 3.5 were automatically rejected, with two exceptions for papers with 3.0 rankings (based on the need for balance in the program). Papers with scores between 3.5 and 5.0 were sometimes accepted—the higher the score, the more likely. (In only one case did we reject a paper with a 4.5 ranking, and this was because it was judged to be not really philosophy of science.) I made a recommendation about which of the borderline papers to accept, based on (a) my assessment of reviewer disagreement, which was frequent and (b) the need for balance in the program. I then unlocked the system so that reviewers’ comments were available to the whole program committee, and asked for the program committee’s thoughts on my recommendations. After some negotiations, the committee settled on a final list of 59 paper acceptances.

Results for Papers

There were 176 submissions and 59 acceptances, making an overall acceptance rate of 33.5%. The breakdown into demographic groups:

Submitted Accepted Acceptance Rate

Male 147 48 33%

Female 29 11 38%

Post-PhD 121 37 31%

Graduate Student 55 22 40%

USA 105 36 34%

International (including Canada) 71 23 32%

None of the differences in acceptance rate is statistically significant (by Chi-square test at the .05 level of significance). It is notable that graduate students are performing at least as well as faculty, and that so many graduate students submitted papers. Also notable that only 20% of the submissions are from women. (I do not have figures for the demographics of PSA as a whole.).

Comparison with PSA 2002: PSA 2002 had 147 total submissions, and 58 were accepted (overall acceptance rate of 39%) of which 18 were international. So the number of submissions is up, the acceptance rate is down, and the number of international acceptances is up.

Observations

These comments have been compiled from the comments of program committee members, as well as my own observations.

1. The number of submissions is up from PSA 2002 (the only year from which I have statistics) from 33 symposia and workshops in 2002 to 45 symposia and workshops in 2004, and from 147 papers in 2002 to 176 papers in 2004. This speaks to the health of the society. But it also means that (a) the program committee has even more work and (b) acceptance rates are down, to keep the meeting to the same size and the two associated issues of Philosophy of Science within budget.

2. The reviewing work was onerous, especially for papers. Each Program Committee member had to review 44 papers in about 6 weeks, in the middle of the academic year. There is no way that they could do this with the kind of care that usually goes into reviewing papers for publication in Philosophy of Science. A few program committee members managed to provide extensive comments, probably at cost to their health and sanity, but most only made cursory remarks, which made it difficult for me to adjudicate in cases of disagreement between reviewers. The Program Committee did a terrific job, given the constraints of their workload. The blind refereeing and decision making at least contributed to the impartiality of the process. But it was not the kind of careful process that goes into regular journal refereeing and editing. Program Committee members said that the work for PSA far exceeds work they have done for other program committees.

3. The PSA office (at HSS) did not provide much support. There was neither secretarial nor computer support for the work of putting together the program and communicating with authors. Since Temple University had few resources to help me (neither a graduate assistant nor secretarial support could be provided), I ended up spending a good deal of time on work that could easily have been delegated. The HSS office helped only with the local arrangements and with registration, and even then, only when prodded repeatedly by George Gale and myself.

4. The distinction between symposia and workshops, started in 2002, is not clear, either in the minds of the proposers or in the minds of the Program Committee. (We muddied it further by accepting some symposia as workshops; in some cases, workshops were symposia that didn’t make the cut.)

Recommendations

• PSA needs to have an office that can provide the secretarial and computer support that is now needed for an international academic society. It should not be left to each program chair to recreate the necessary infrastructure and try to persuade their institution to provide a graduate assistant/secretarial support.

PSA is decentralized in a way that makes it inefficient and difficult to work with: the newsletter and association webpage are at the University of Wisconsin under Malcolm Forster; George Gale (the secretary treasurer) is at University of Missouri; the PhilSci archive is at University of Pittsburgh and has a separate staff; the membership is run by the University of Chicago Press; the membership list is inextricably tangled with that of HSS so that communications are difficult; the Journal is at the University of South Carolina; the office is at HSS in South Florida (where HSS always comes first); and this year’s program is run through Temple University’s website and an offsite website managed by Micah Sparacio (in New Jersey). If it worked, the decentralization wouldn’t matter—it might even be a strength—but it hasn’t worked well for me.

• The workload for committee members is too great. Some of us make the following suggestions (we do not all agree that they are good suggestions, but they are each worthy of mention):
1. appoint a larger program committee, or add reviewers from e.g. the journal’s Editorial Board.
2. no more automatic/large scale publication of papers; they can always be submitted to the journal independently (then reviewers aren’t as concerned about letting a not-so-good paper through; a paper that is ready for presentation may not yet be ready for publication)
3. annual rather than biennial meetings, given the volume of acceptable papers. This would also address the loss of momentum that occurs in the off-meeting years, and acknowledge the growth of the society.
4. no more submission of papers; submit only abstracts; and (optional) charge the program committee with selecting the 20 best papers for a journal issue or (another idea) let authors figure out themselves how they are going to submit their papers, or groups of papers, to various journals (not only Philosophy of Science)
5. no more (automatic) publication of symposia (this might help encourage more real workshops, also).
6. publication only of top ranked symposia and papers; the rest of the papers could be presented as “works in progress”. That way, the committee wouldn’t agonize as much about accepting interesting papers that aren’t really ready for publication.

• If the two categories, symposium and workshop, are kept, the difference between the two should be clarified.

• Enter serious negotiations with HSS (and 4S, when we meet with them) to plan joint sessions. These could also take the form of mini-conferences before or after the meetings. We are so busy with our own fields that we often overlook the importance of these interdisciplinary events.

• A local arrangements committee to do the local arrangements in consultation with the PSA office. (I was surprised to find out that I was responsible for this. I could not devote as much time to it as I would have liked, because of my main responsibilities as Program Chair. Also, I’ve never been to Austin, Texas. It’s a good idea to have local arrangements done by someone who knows the city or at least visits beforehand.) Someone has already made an offer to do the local arrangements for PSA2006 in Vancouver!

Again, my sincere thanks for the privilege of working together with an excellent committee: Christina Bicchieri, David Hilbert, Carl Hoefer, Harold Kincaid, Fred Kronz, Roberta Millstein, Alan Richardson, and Andrea Woody, to put together PSA 2004.

Miriam Solomon

October 16, 2004