How to provide information sources and comments in conversational agent response content: The roles of specialists and internet users

Conversational agents (CAs) have been widely applied, and user experience with CAs is receiving increasing attention. However, few studies have focused on the contribution of the design of CA responses to the user experience. We conducted two studies examining participants' user experience with CAs' response content design on information sources (Study 1) and comment information (Study 2). The influence of specialists and internet users was investigated, and the factor of the application scenario was also examined. A mixed-method approach, including user experience (UX) scoring and paired comparison tasks, was adopted, and twenty-one people participated in the formal study. The results showed that information sources and comments from specialists can be added to the response content of CAs to improve the user experience. Participants liked the CAs' responses with comments from internet users rather than the responses with no comments. Application scenarios could affect the user experience with CAs’ response content designs. Participants preferred the responses with contextual information from specialists in professional application scenarios (e.g., news query and health query scenarios), whereas they liked the responses with contextual information from internet users in entertaining application scenarios (e.g., music query, travel query, and restaurant query scenarios). Compared with comment information design, information source design may provide a better user experience. This study proposes guidelines for the future generation of CAs and helps voice interaction designers to create CAs that meet the needs of users.