Figure 4: Architecture for Shoutvelocity s Ranking System based on Pairwise Comparisons.
date process lowers the score of the losing item, and increases the score of the winning item based on Section 4.2.
5.3.2 Theoretical Guarantee
We are unable to analyze the above tournament algo- rithms theoretically. But we prove that a simplified version of the algorithm is able to estimate rank r within error r with bounded number of comparisons per item for the spe- cific distributions we study. The formal proof the following
theorem appears in Section 7. heorem 5.3. With pairwise comparisons,
erf( x) and g is the normal distribution there is an algo- rithm to estimate the rank within multiplicative error 1 ± with bounded feedback bandwidth per item.
The algorithm that achieves the result above compares items with other items with successively increasing scores till it consistently loses: Consider items with geometrically decreasing percentile ranks 1/2, ρ/2, ρ2/2..., 1/n. We com- pare a particular item to these items successively till we find two consecutive ranks between which the item lies. The number of comparisons with each of the above items is large enough to conclude whether we should proceed to the next item with high confidence. We clearly approximate the rank
within a factor of 1 + if ρ =
We elaborate on the comparison-based reviewing module adopted by our shoutvelocity system. Figure 4 shows the various components of our system, essentially obtained by expanding the reviewing module from Figure 1. In our sys- tem, when a user submits an item, he gets shown a pair of items for review. Therefore, for every item he submits, we get feedback on two items. Obviously, even without submit- ting an item, the user is given the option of reviewing any number of pairs of items.
In general, if we wish to obtain better rank estimates for the top items, items with higher score estimates should be
Figure 5: Screenshot of top shouts from shoutveloc- ity.com.
given more number of reviews. One method is to pick items randomly biased by the current score or rank estimates of the items. From a practical standpoint, another factor that needs to be taken into account is the age of the item. Clearly newer items should be preferred over older items. A simple way to achieve this is to time discount the score estimates.
Screenshots and Statistics
The shoutvelocity website was launched in June 2008. In the interest of confined testing of various UI and ranking approaches, we only made a limited release of the site. While the website remains open to public, we explicitly invited only a small number of people, and we’ve attracted 196 users since our release. Our website has seen around 1245 distinct shouts posted, and together these shouts have received 4853 reviews in all.
We give screenshots to illustrate two of the core features of shoutvelocity. Figure 5 shows the shouts that were rated among the best by users. We give a score for each shout, as well as links to see the“shouter”, comments left by users, and history of other shouts against which it won/lost. On top of Figure 5 we see the various ranking options supported by shoutvelocity, which include ranking purely based on score, filtered by recency, or weighted based on score decay with age.
Figure 6 shows an example of a review screen, where two shouts are shown. The user is asked to pick which shout she prefers, and may additionally leave comments/emoticons on either shout.
We reiterate that the above figures only show a small sampling of the various features supported by shoutveloc- ity. Since the screenshots, and even this paper covers only the core technical issues with shoutvelocity, we encourage the readers to visit http://shoutvelocity.com to enjoy the capabilities of our complete system.
Proof of Theorem 5.1: In order to ensure that every item is rated till it loses, we show that the average number
of ratings µf
required tends to infinity as n → ∞, for