Quality rater and algorithmic evaluation systems: Are major changes coming?

November 22, 2023

Crowd-sourced human quality raters have been the mainstay of the algorithmic evaluation process for search engines for decades. The purpose of this relevance data collection is to mathematically roll it up and adjust overall proposed systems. Each has its advantages and disadvantages, largely about relevance labeling quality (compared with gold standard or ground truth) and how scalable the approach is. Human quality raters do not judge websites or webpages in isolationEvaluation is a measurement of systems, not web pages – with “systems” meaning the algorithms generating the proposed changes. Some problems with crowd-sourced human quality ratersIn addition to the lack of scale, other issues come with using the crowd.

The source of this news is from Search Engine Land