Surprisingly popular

The surprisingly popular answer is a wisdom of the crowd technique that taps into the expert minority opinion within a crowd. For a given question, a group is asked two questions:

What is the probability that this answer is correct?
What is the average probability others will give to this answer?

The answer that maximizes the average difference between the "right" and "popular" answers is the "surprisingly popular" answer. The term "surprisingly popular" was coined in a 2017 paper published in Nature entitled "A solution to the single-question crowd wisdom problem", which outlined the technique.

Algorithm

Suppose we'd like to determine the answer to the question "Is Philadelphia the capital of Pennsylvania?" The two questions asked of the group, and the average responses, are:
The difference between the answers to the right question and the popular question:

Yes: 65% − 75% = −10%
No: 35% − 25% = 10%

Thus, the No answer is surprisingly popular.

Explanation

The technique avoids the "double-counting" of prior probabilities across participants, a major issue for belief aggregation rules under the naive assumption that participants' answers are independent. Say a crowd has two groups:

Experts, who have some valuable piece of evidence which is not common knowledge. They combine this evidence with their prior probability to get an improved posterior probability.
Non-experts only have common knowledge to go off of, and therefore provide only the prior probability.

When asked to answer a question, non-experts will tend to give equal answers to both questions. This is because they have no reason to expect they are wrong in either direction—their answer is just as likely to be an overestimate as it is an underestimate.
However, the experts have access to both the prior probability and the posterior probability, which allows them to make a better estimate of the group's opinion. Because they know the group contains both experts and non-experts, they will expect the average probability to be in between the prior and the posterior. This means that, unlike the non-experts, their answers will not tend to cancel out when the prior probability is subtracted out.
Looking again at the capital example, say there are two groups, experts and non-experts:

Experts – "Philadelphia is/is not the capital, but most others won't know that."
* This group thinks they have unknown information about whether Philadelphia is likely to be the capital.
* This group thinks the probability that Philadelphia is the capital is low, but that not everybody will realize this.
* Therefore, the group will tend to assume others assign a "bad" probability to Philadelphia being the capital.
Non-experts – "Philadelphia is/is not the capital, and others will agree."
* This group is answering based on common knowledge.
* This group has no reason to think the average probability that Philadelphia is the capital will be different from their own estimate.
** Thus, their estimate for the popularity of Philadelphia is roughly equal to their estimate for the probability that Philadelphia is the capital.
** This means that when subtracting the two probabilities, the group's contributions to the overall probability cancel out.

The strength of the method is that it causes the two non-expert groups to cancel out, thus identifying the opinions of the expert group.

For rankings

For m>2 candidates, the Surprisingly Popular Algorithm requires votes from an infinite number of voters on all possible ranked permutations of the alternatives to recover the ground-truth ranking with complete certainty, as discussed in the Nature article. However, the algorithm can be extended to recover rankings using various elicitation formats.