MRRs for thumb-based and comparison-
based ranking for
At m = 1000, for the best
γ, MRR is 0.17 for comparison-based approach and
0.06 for thumbs.
Figure 10: Convergence of MRR on real data from shoutvelocity.
scale preferences. For an excellent exposition on pairwise comparisons, the reader is referred to . Thomas Saaty motivates the need for pairwise comparison in order to rank entities that have no tangible properties with known scales of measurement .
Hacker and Von Ahn  propose a 2-person online game, where users express preferences on photos, and these prefer- ences are used for ranking. They compare the performance of their scoring function with other well-known functions such as ELO  and TrueSkill . The Elo rating sys- tem  is used to compute the relative skill of players in two-player games such as chess. Unlike , our focus is to design a comparison-based mechanism taking into account feedback bandwidth consideration, where each person sub- mitting an item reviews only one more pair of items. We borrow the score update function from the Elo system.
Ajtai et al  study the problems of selection and ranking with imprecise comparisons. Again, the idea behind their work is similar in spirit to what we suggest in this work: for every user there exists a value δ > 0 which differentiates between a just noticeable di erence and otherwise, i.e., if the values of the two elements being compared is less than δ, the the result of the comparison could go either way. When the value of the difference is more then δ, then the comparison is correct.
Figure 11: Latency of shouts plotted against fraction of shouts.
Figure 12: Cumulative distribution of scores from shoutvelocity.
This paper addressed the problem of designing ranking mechanisms for forums. Broadly, we studied independent thumb-based and comparison-based reviewing of items in the forum. We theoretically showed the benefits of compari- son based ranking mechanisms based on desirable properties including accuracy and rapid convergence with minimal user feedback. We presented shoutvelocity, an online forum that fully implements the comparison-based ranking mechanism from this paper. We experimented with synthetically gener- ated data as well as real data from shoutvelocity, and showed that shoutvelocity’s comparison-based ranking significantly outperforms thumb-based ranking on the desired properties.
Star-ratings are a clear generalization of thumb-based rat- ings, and our theoretical analysis can be easily extended to star-ratings: Intuitively, a rating of 3 stars out of 5 is roughly like getting 3 thumbs-ups and 2 thumbs-downs. Analo- gously, pairwise comparisons can be generalized to n-way comparisons, but the theory doesn’t directly carry over. Al- though the practicality of n-way comparisons is questionable as they may impose a more significant reviewing burden on users, their theoretical study would be interesting.
 Digg. http://digg.com.  Facebook. http://www.facebook.com.