FAQ: AIS AI Reviewer
What is this service?
ECIS 2027 is offering an optional AI-generated review as an opt-out feedback service for authors who submit Completed or Short Research Papes. The AI Review Service was developed by the AIS and piloted at ICIS in 2026 as an opt-in service where it was well received by authors.
Our goal in using the AIS AI review service is to provide authors with additional developmental feedback and to manage the reviewing burden on the IS community in light of the growing number of submissions conferences receive. In particular, ECIS anticipates a substantial increase in submission volumes, and we do not want the resulting additional reviewing burden simply to be transferred to the IS community. We hope to lighten the load for conference reviewers while maintaining the quality and integrity of the review process.
The AI review does not replace human judgement but provides a third developmental perspective alongside the two human reviews. Importantly, the AI reviewer does not provide an acceptance or rejection recommendation; its role is to provide authors with constructive feedback that can help them further develop their manuscripts, including following the conference in preparation for a possible journal submission. Given that this tool can serve as Reviewer 3, only two human reviewers will be recruited for each submission, and each human reviewer will be asked to ensure their review is not colored by any AI input.
The AIS AI reviewer is specially trained on ECIS submissions to adhere to AIS and ECIS standards. Unlike a generic LLM, it is therefore customized to the ECIS context, drawing on prior ECIS submissions and ECIS-specific expectations to provide more tailored feedback. Thus, what we seek from human reviewers is purely human insight and evaluation rather than output from a non-AIS AI tool. It is our hope that freeing reviewers to focus more closely on fewer papers will enhance the ECIS review process while helping us accommodate increasing submission volumes without transferring that burden to the community. As with any innovation, we expect there to be a learning curve as AI is implemented in the review process. We ask for collaborative and constructive feedback throughout the process, with the goal of maintaining the standard of excellence ECIS is known for.
Will the AI reviewer make a decision on my paper?
No. The AI review is trained to provide supplemental feedback only. It does not provide a recommendation as to whether a submission should be accepted or rejected. Acceptance decisions remain the responsibility of the human decision makers in the ECIS review process. AEs will be instructed to weigh the AI feedback less heavily than human-generated reviews in paper acceptance decisions. The AI review may be used by track chairs and associate editors as an additional source of developmental feedback to make decisions, but the AI tool will not give a recommendation about whether a submission should be accepted or rejected.
Will my paper be used to train OpenAI models?
No. AIS uses OpenAI through its API, and OpenAI states that data submitted through the API is not used to train or improve OpenAI models. OpenAI’s business data policy states the same for API customers.
Will my paper be stored by OpenAI?
AIS configures its OpenAI API calls with store=false, which disables storage of response objects for later retrieval. However, OpenAI states that API data may still be retained for abuse monitoring for up to 30 days, after which it is not retained unless longer retention is legally required. For that reason, the most accurate statement is that your paper is not used for model training and is not stored, but temporary retention may still occur for security and abuse-monitoring purposes.
How does the system generate the AI review?
The system retrieves consenting papers from the regular submission system and uses an AI review pipeline built on OpenAI’s API together with retrieval-augmented generation (RAG). Unlike a generic LLM reviewing a manuscript without conference-specific context, the AIS AI Review Service is customized for ECIS and uses prior ECIS submissions as part of the knowledge base informing its feedback. The AIS AI review service is trained to assess five dimensions of scholarly contribution: theory, method, framing, phenomenon, and composition. To shape the style and content of the review, AIS developed and tested numerous prompts and review models using prior publications and selected the prompt that best aligned with desirable conference review traits, such as being informative, concise, fair, and constructive.
How reliable is the AI review?
The system is designed to generate feedback using a state-of-the-art OpenAI model, data from prior ECIS papers, and a prompt informed by an editorial on what reviewers should look for in papers. This ECIS-specific customization is intended to provide more relevant and contextually appropriate feedback than would be obtained from a generic LLM. Even so, AI-generated feedback may sometimes be incomplete, inaccurate, or less nuanced than expert human review. Decision makers and authors will therefore treat the AI review as supplemental feedback, not as an authoritative judgment on paper quality or as an acceptance or rejection recommendation.
Do I have to opt-in?
No. While the AIS AI review service is provided as a default setting, authors can opt-out during submissions. Authors who opt out will receive 2 human peer reviews. Authors who do not opt out will receive the two human reviews together with the supplemental AI-generated review.
Who can I contact if I have additional questions?
About the AIS AI review service, including technical questions, you can contact:
Dorothy E. Leidner, PhD
President, The Association for Information Systems (AIS)
Leslie H. Goldberg Jefferson Scholars Foundation Eminent Professor, University of Virginia
Reza Mousavi, PhD
Associate Vice President of AI, The Association for Information Systems (AIS)
Associate Professor, University of Virginia
About the ECIS 2027 submission and review process you can contact:
Leona Chandra Kruse, Jan Recker, Amber Grace Young
ECIS 2027 Program Co-Chairs
About AIS AI Reviewer Policies, you can contact:
Amber Grace Young, PhD
Vice President of C.O.R.E., The Association for Information Systems (AIS)
Associate Professor, University of Oklahoma
