Peer review is an essential step to ensure the quality of papers accepted to large conferences. In our work, we will examine the different behaviors of reviewers in the bidding stage, where our main focuses will be on how reviewer’s behavior is affected by paper’s display order, reviewers preference, and ‘points’ associated with each paper that ostensibly reflects his demand; the higher the ‘points’, the higher the chance to get the paper. Our empirical methodology consisted of a bidding platform, where reviewers bid on a set of papers from a large list of submitted papers.
We implemented a paper review bidding system where we conducted a series of controlled lab experiments. In each experiment, we isolated one variable and examined how it affects reviewers’ behavior in three key parameters – order bias, price bias, and private preference bias. We found that presenting a score for each paper significantly changes the behavior of reviewers and incentivized them to choose papers with a higher score, i.e. papers with a higher chance of getting. Also, we found that there is a bias in selecting papers that are displayed at the top of the list, and that reviewers choose papers that match their declared preferences. Finally, we fitted a regression model for the results of each experiment which presents the factors influencing the behavior of the reviewers and enables prediction about future bids.