To reviewers: do you upload your review to an AI? (NOT the paper itself) [D]

[u/StingMeleoron]

The way I review a paper has stayed substantially the same, except for an added step. Trying to get a sense of how common it is here.

Nowadays I do the review, and I upload my review to an AI to ask for second thoughts. I don't ever upload a paper I'm reviewing, though. The authors can do it themselves (if they want to, it's their call).

So in the end, it's even more work for me. Not fun. But it has produced higher quality reviews, most of the time. And less typos.

[u/StingMeleoron]

Just stumbled upon this post here which made me ask this:

https://www.reddit.com/r/MachineLearning/s/ajDV3JC0QC

I usually review a paper from the standpoint of "how can I make the authors make it better, and is it good enough to be accepted or should they first revise it?" - but that's just how I work, I guess.

It has been a humbling experience for me because, even though I try to do my very best to understand and judge a paper, the AI model (usually Claude) does sometimes spot inconsistencies in my own review, which makes me return to the paper and read it again. In the end this does produce a better quality review, most of the time.

I have colleagues that do the opposite (start by uploading the paper and then do the review), but I believe that can leads to bias. Also, that's not how I read a paper anyway (most of the time).

What about y'all? Looking forward to hear from your side now!

[u/Spacekat405 → u/StingMeleoron]

Uploading the paper is forbidden by most of the reviewer guidelines I’ve seen, so never that. I don’t upload my review either, since without the paper whatever the AI model says will be missing all the context

[u/StingMeleoron → u/Spacekat405]

Right? I was baffled when I saw that most people seem to do it now (at least those who responded in that linked thread). Some are even saying that the ToC allow it, which surprised me. I didn't expect that at all. Thanks for the comment!

[u/pastor_pilao]

Sincerely, no one cares about the typos Ina  review. AI makes everything more verbose, and if you do use it not only to improve grammar but also to check the paper, it will overly focus on small irrelevant details instead of evaluating the actual contribution of the paper, which is your main work as reviewer.

The main overarching question you have to answer in your review is: would it be really worth of someone's time to read that paper in it's entirety?

Don't worry about some typos or small mistakes, I guarantee the AC will appreciate more and it will be more useful to have a genuine review than whatever thr AI spits out or focuses on

[u/Smallpaul → u/pastor_pilao]

The poster said they ask the AI for “second thoughts”, or to rewrite and replace the review.

[u/soft_abyss]

Yes I do it. Not to give feedback on my review more so just to help make it easier to read/formatting. A lot faster than editing it manually from scratch.

[u/hihey54]

I am writing this as a reviewer, as an AC/Editor, and as an author of papers.

Don't do it.

You say "higher quality reviews", but this is (most likely) your perception.

If an expert (as reviewers should be) got something wrong from the paper, they shouldn't turn to an LLM to "ask for second thoughts". Thoughts about what, exactly?

If the review has issues, that's fine: this is what the rebuttal is for. You should approach the rebuttal with the same mindset as the one you have when you receive the response of the LLM: you made a mistake? Then adjust your review, and re-adjust your view on the paper.

We are writing papers so that they're read by humans, not LLMs.

As an author, I would like to receive an assessment that comes from the intended audience of the paper (humans, not LLMs).

As an AC, it is quite easy to infer reviews for which LLMs have a substantial influence. They tend to overly-focus on very small details that are trivial to fix with minor rewriting, but the review makes them sound like they're serious mistakes. When this happens, if the reviewer does not change their mind after being corrected by the authors, I overrule that---but doing so is, frankly, painful.

(my2c)

[u/StingMeleoron → u/hihey54]

That's interesting! Thank you for sharing.

The "second thoughts" I ask for are basically an assessment of anything that could be improved on my own review. Perhaps I missed an important paper that could be cited to help the authors (especially when I am out of my domain), or maybe something in the end of my review contradicts what I said at some other section. It has been helpful.

But of course I don't let it guide me more than offering a useful input. Reading from you that this is not appreciated though is quite reassuring of my own reviewing habits.

(Usually what I ask is something of the sort: "is a weak reject here well justified or am I being too harsh?", or something)

I guess in the end, I am prompting the LLM not to improve the review of the paper, but improving my reviewing skill itself. (I just started my first postdoc, so that might be the strongest factor.)

Thank you again for sharing! This post got few comments, but they were exactly I was looking for. Cheers!