Do large language model outputs exhibit algorithm aversion? A case study using GPT-3.5

Existing studies suggest that LLMs tend to exhibit a bias in favor of content generated by AI over those generated by humans. Conversely, human decision-makers tend to prefer human experts over algorithmic tools or AI in specific contexts—a tendency commonly referred to as algorithm aversion. Prior studies have characterized various biases in LLMs, yet little work has jointly examined biases toward human in LLMs and algorithm aversion observed in humans within a unified analytical framework. Adopting a human-AI collaboration perspective, this study moves beyond a sole focus on human attitudes toward algorithmic tools by examining preference patterns (human experts vs algorithmic tools) operationalized through fairness, trust, liking, and usefulness ratings in GPT-3.5-generated outputs across different decision objective (human-centered vs machine-centered), triangulating the phenomenon through analyses of AI-generated data and human evaluations and perceptions. The results indicated that GPT-3.5 produced lower trust ratings in algorithmic tools across six decision-making tasks when the decision objective was human-centered, in contrast to the neutral trust ratings; however, it exhibited increased trust when the decision objectives shifted to machine-centered. In terms of human participants, they demonstrated a similar pattern in decision-making method preferences and evaluated AI more favorably when its expressed preferences aligned with those of humans. The findings contribute to the emerging field of the social science of AI by providing new insights into the alignment and divergence of human and AI biases in decision-making processes.