Week-at-a-Glance
Ethics of the Papers Review Process for Visualization Conferences
-
Protect Ideas.
As a reviewer for Visualization, you have the responsibility to protect the confidentiality of the ideas represented in the papers you review. Visualization submissions are by their very nature not published documents. The work is considered new or proprietary by the authors; otherwise they would not have submitted it.
Of course, their intent is to ultimately publish to the world, but most of the submitted papers will not appear in the proceedings of the conference. Thus, it is likely that the paper you have in your hands will be refined further and submitted to some other journal or conference, or even to the same conference next year. Sometimes the work is still considered confidential by the author's employers. These organizations do not consider sending a paper for review to constitute a public disclosure. Protection of the ideas in the papers you receive means:
- Do not show the paper to anyone else, including colleagues or students, unless you have asked them to write a review, or to help with your review.
- Do not show movies or other supplemental material to non-reviewers.
- Do not use ideas from papers you review to develop new ones.
After the review process, destroy all copies of papers and supplemental material that are not returned to the senior reviewer and erase any implementations you have written to evaluate the ideas in the papers, as well as any results of those implementations.
- Avoid Conflict of Interest.
As a reviewer of a Visualization Conference paper, you have a certain power over the reviewing process. It is important for you to avoid any conflict of interest. Even though you would, of course, act impartially on any paper, there should be absolutely no question about the impartiality of review. Thus, if you are assigned a paper where your review would create a possible conflict of interest, you should return the paper and not submit a review. Conflicts of interest include (but are not limited to) situations in which:
- You are a co-author of the work.
- You have a strong affiliation with the same institution as one of the authors. This includes, but not limited to your current employment as a professor, adjunct professor, visiting professor, or similar position, in the role of a consulting or advisory arrangement, previous employment with the institution within the last 12 months, being considered for employment at the institution, any role as an officer, governing board membership, or relevant committee, or the current enrollment as a student.
- You have been directly involved in the work and will be receiving credit in some way. If you're a member of the author's thesis committee, and the paper is about his or her thesis work, then you were involved.
- You suspect that others might see a conflict of interest in your involvement. For example, even though Microsoft Research in Seattle and Beijing are in some ways more distant than Berkeley and MIT, there is likely to be a perception that they are "both Microsoft" and folks from one should not review papers from the other.
- You have collaborated with one of the authors in the past three years (more or less). Collaboration is usually defined as having written a paper, book or grant proposal together, although you should use your judgement.
- You were the MS/PhD advisor of one of the authors or the MS/PhD advisee of one of the authors. Funding agencies typically consider advisees to represent a lifetime conflict of interest.
- You are related to one of the co-authors. This includes, but not limited to spouse, child, sibling, or parent, as well as any affiliation or relationship of your spouse, of your minor child, of a relative living in your immediate household or of anyone who is legally your partner that you are aware of.
- Other relationship, such as close personal friendship or significant animosity between you, that you think might tend to affect your judgment or be seen as doing so by a reasonable person familiar with the relationship.
The blind reviewing process for some of our conferences will help hide the authorship of many papers, and senior reviewers will try hard to avoid conflicts. But if you recognize the work or the author and feel it could present a conflict of interest, send the paper back to the senior reviewer as soon as possible so he or she can find someone else to review it. If you are in doubt about any conflict, you should discuss it with the editor/paper chairs or the person that assigned the paper to you. You should never contact the authors directly.
- Be Specific.
The publishing of scholarly work is essential for the academic community specifically and the research community in general. Therefore careers and reputations hinge on publishing in the proceedings, academic tenure decisions are based on the proceedings, and patent infringement cases discuss whether something is considered novel enough to publish in the proceedings.
Therefore, it is our duty to do a careful, objective and scholarly review. The emphasis of our reviews should be to help the authors on how to improve their work and therefore to improve the overall quality of the work in our research community. In general, it will not be helpful to anyone - neither the program chair, nor the authors
- to do a quick or superficial review, to say that the work is good or bad without giving clear reasons,
- to state that the work has been done before without giving clear citations of previous work
- to complain about the structure of the paper without making suggestions on how to improve it
- to dismiss the evaluation method without being specific about the flaws
- etc.
A casual or flippant review of a paper that the author has seriously submitted is not appropriate and may be rescinded from the reviewing process. In the long run, casual reviewing is a most damaging attack on our conference. There is no dishonor in being too busy to do a good review, or to realize that you have over-committed yourself and cannot review all the papers you agreed to review. But it is a big mistake to take on too much, and then not back out early enough to allow recovery. If you cannot do a decent job, give the paper back and say so. But please, do it early so that the senior reviewer has time to select another reviewer before the deadline.
-
Be Helpful.
Have an open mind, or at least reveal your biases.
If you are a die-hard algorithm-driven researcher, and you are assigned a user study paper, and the call for papers welcomes both types of papers, don't bash the paper simply for its methodology. It's not fair and it benefits no one. Either assess the paper according to the appropriate user-study standards, or admit that you are not capable of doing so. At most, you might discuss why the algorithmic approach does not provide a suitable answer to the research question. But don't force the author to ask a different question that can be answered using algorithmic methods. Many of us have been frustrated by reviews in which the judge basically told the researcher to do a different study. Work within the author's goals.
Look for the good in a paper.
No matter how much you dislike the paper, try to find the good aspects. Perhaps there is a different approach to a problem, a novel application, a promising evaluation methodology, or even a potential twist which could be studied further during the revision of a paper. Be helpful to the authors and point out the potential positive aspects. Let the authors know, if you think their work would be better suited for a different conference, journal or venue. Be encouraging.
- Be Tactful.
Belittling or sarcastic comments may help display one's wit, but they are unnecessary in the reviewing process. The most valuable comments in a review are those that help the authors understand the shortcomings of their work and how they might improve it. If you intensely dislike a paper, give it a low score. That makes a sufficient statement.
While we will assure the anonymity of the reviewing process, read over your review and ask yourself whether you would be able to recite your critique in front of the authors? If the answer is 'no' you might want to work on some of the wording of the criticism. A scathing review is usually not only unhelpful to the authors and paper chairs, but it also tends to alienate typically new researchers that would like to enter into our community. Put out a helping hand.
- In Summary.
Adherence to ethics sometimes requires a more careful analysis of the work as well as a well-thought out response. However, while it appears that we might be losing some time, this will be made up by improving the culture in our community overall, by helping each other to put out better results and overall increasing the quality of our conferences and journals. That is what we are striving for in our visualization community and it is well worth the effort.
Adapted in February 2009 from
the SIGGRAPH Ethics Guidelines at
http://www.siggraph.org/s2009/submissions/technical_papers/ethics.php,
incorporating ideas by Lance Fortnow
at
http://weblog.fortnow.com/2006/04/reviewer-ethics.html,
NSF
http://www.nsf.gov/publications/pub_summ.jsp?ods_key=form1230p,
and Karen
Markin
http://www.aejmc.org/_officers/officer_resources/paper_reviewtips.php
