Title: GPT is Not an Annotator: The Necessity of Human Annotation in Fairness Benchmark Construction
By: Jonathan May, K. Felkner, Jennifer A. Thompson
Abstract: Social biases in LLMs are usually measured via bias benchmark datasets. Current benchmarks have limitations in scope, grounding, quality, and human effort required.