IRRJ reviewer guidelines

As an IRRJ editorial board member, your main task is to conduct objective and thorough reviews in a timely manner. The goals of your review is to give feedback to the authors and to identify significant contributions to information retrieval research. 

General guidelines

  • Be positive and generous: Review works of others as you would like your works  to be reviewed;
  • Be precise and concrete. (include references to back up your claims);
  • Provide constructive feedback;
  • Explicitly state what you do not fully understand;
  • Do not reject a paper just because you don't find it "interesting".  IRRJ emphasizes technical correctness over subjective significance, to facilitate scientific discourse on topics that may not yet be accepted in mainstream venues but may be important in the future.

When reading the paper, consider:

  • The objective of the work;
  • The proposal and the evidence given;
  • Strong points;
  • Weak points;
  • Be mindful of (your) biases; be open minded.

When writing the review:

  • Summarize the paper's main claim(s). Be positive and generous;
  • List strong and weak points;
  • Overall evaluation (accept or reject) with key reasons;
    •  What is the specific question and/or problem tackled by the paper?
    • Is the approach well motivated and well-placed in the literature?
    • Does the paper support the claims?
    • Are methods reproducible?
  • Supporting arguments of your evaluation;
  • Questions to the authors;
  • Feedback for the authors: How to improve the paper?

Your recommendation

  • Choose one of “Accept submission”, “Revisions Required” (Accept with revisions), “Decline submission” (Reject). Please, only choose one of the other options in exceptional occasions.

Confidentiality

  • Do not share the manuscript with anyone else: If you need an additional expert opinion, please send a request to the associate editor via the discussion option.
  • Do not use AI or AI-assisted tools (such as ChatGPT) to review submissions or to generate peer review reports. Reviewers are solely responsible for the content of their reports and the use of AI technologies for this purpose constitutes a breach of peer review confidentiality.

Further considerations

  • The authors do not know your identity. Keep your review anonymous.
  • Not all researchers have access to GPU clusters:  Do not ask for additional evaluations that require substantial processing, such as a BERT baselines.
  • In your review, do not suggest your own papers to be cited.