ECIS 2027 Guidelines for Responsible Use of Artificial Intelligence in Peer Review
Given the expected surge in submissions driven by GenAI, ECIS will endeavour to offset some of the community reviewing burden by using a dedicated AI reviewing service as part of the peer review process. The reviews produced by this system will be shown to AEs as a “Reviewer 3”. The AI reviewer will not remove human judgement from the review process, will not automate outcomes, will not provide a recommendation for the paper to be accepted or rejected, and will have human oversight.
As such, we maintain the role of peer review as an exercise of human scholarly judgment. At ECIS2027, reviewers are selected for their expertise, experience, and ability to evaluate research within the intellectual traditions of the information systems discipline. The responsibility entrusted to an ECIS2027 reviewer selected to evaluate submissions to ECIS2027 therefore cannot be delegated to an AI system. AI may support limited administrative or editorial aspects of peer review, but AI should not substitute for the substantive intellectual work of a human reviewer or editor. Human reviewers, Associate Editors, Track Chairs, and Program Chairs remain responsible for the evaluations and decisions made during the review process. Substantive aspects of reviewing and decision making may thus not be delegated to AI.
Because we already have an AI tool designed specifically for reviewing ECIS papers, and because we must protect the intellectual property of submitting authors, the below rules apply to reviewing for ECIS2027. Should you have questions or concerns, you can reach out to the ECIS 2027 Program Chairs at ecis2027programchairs@gmail.com
Sincerely,
Leona Chandra Kruse, Jan Recker, and Amber Young – the ECIS 2027 Program Chairs
Guidelines for ECIS2027 Reviewers and Associate Editors
1) Reviewers and AEs SHOULD
- Read the assigned manuscript themselves and independently assess its contribution, theory, methods, evidence, and implications.
- Develop their own criticisms, recommendations, and evaluation of the manuscript.
- Validate the correctness of any references included in a manuscript
- Verify factual claims when doing so is relevant to their evaluation.
- Provide constructive, specific, and professionally expressed feedback.
- Protect the confidentiality of the manuscript and all information obtained through the review process.
- Inform the AE or appropriate track chair if they encounter indications that authors violated submission policies (e.g., concerning the use of AI in manuscripts or in the case of hallucinated references).
- Inform the AE or appropriate track chair if they encounter circumstances that prevent them from completing an independent and competent review.
2) Reviewers MAY NOT use AI in substantive ways during the reviewing task.
ECIS reviewers and AEs may not enter any part of an ECIS submission into an AI tool. Reviewers or AEs may not upload, paste, transmit, or otherwise provide confidential submission materials to external AI systems. This prohibition includes excerpts, tables, figures, abstracts, reviewer comments, and sufficiently detailed descriptions of unpublished work. Reviewers or AEs may not ask an AI system to evaluate a paper, identify its strengths or weaknesses, assess its theoretical contribution, evaluate its methodology or analysis, determine its novelty, formulate substantive criticisms, generate reviewer recommendations, assign ratings, or recommend acceptance or rejection.
Prohibited substantive uses include asking AI to:
- Read, summarize, or evaluate the submitted manuscript;
- Identify the manuscript’s strengths or weaknesses;
- Identify theoretical or methodological problems;
- Evaluate the appropriateness of a theory, research method, statistical analysis, or interpretation;
- Assess novelty, contribution, rigor, significance, or publishability;
- Compare the manuscript with the literature in order to determine its contribution;
- Generate criticisms or suggestions for improving the manuscript;
- Generate all or substantial portions of a review;
- Recommend a numerical rating, ranking, acceptance, rejection, or other disposition;
- Simulate an additional reviewer or provide a “second opinion” on the paper; or
- Substantially rewrite a review in ways that introduce new intellectual content.
Reviewers may also not circumvent these restrictions by providing an AI system with summaries, extensive notes, extracted arguments, tables, results, or other representations of confidential manuscript content.
3) Limited Non-Substantive Uses
AI-assisted tools may be used for limited, non-substantive purposes when no confidential manuscript information is provided to the system and when permitted by the conference.
Examples may include:
- Checking spelling and grammar in text independently written by the reviewer;
- Improving the readability of reviewer-authored comments without changing their substantive content; and
- General-purpose searches concerning publicly available concepts or literature that do not reveal information about the submitted manuscript.
Such assistance must not introduce substantive judgments or arguments into the review.
When there is uncertainty about whether a proposed AI use is substantive, reviewers should err on the side of not using the AI system or consult AEs, track chairs, or the ECIS 2027 Program Chairs
Additional Guidelines for ECIS2027 Associate Editors
Associate Editors occupy an important position in safeguarding the integrity of the ECIS peer review process. AEs are responsible not merely for aggregating reviewer recommendations but for exercising independent scholarly judgment over the papers assigned to them.
1) Independent Assessment
AEs should personally engage with each assigned manuscript sufficiently to understand its central argument, contribution, methodology, and major strengths and weaknesses. AI should not be used to substitute for this assessment.
2) Prohibitions
AEs may not ask an AI system to:
- Evaluate or summarize an ECIS submission;
- Compare or adjudicate competing reviewer assessments;
- Determine which reviewer arguments are correct;
- Generate the AE’s assessment of the manuscript;
- Recommend a decision-making;
- Rank manuscripts;
- Generate substantive discussion points for the review team; or
- Determine whether a paper should be accepted or rejected
3) Handling Reviewer Compliance
If an AE has reasonable grounds to believe that a review was generated substantially by AI or that a reviewer otherwise used AI contrary to this guidance, the AE should not attempt to determine misconduct solely through AI-detection software. Instead, the AE should raise their concern with the appropriate Track Chair or Program Chair. Depending upon the circumstances, conference leadership may request clarification from the reviewer, disregard the review, obtain an additional review, or take other appropriate action. AEs should avoid making accusations about inappropriate AI use based solely on writing style or automated AI-detection scores.
