Assessing ChatGPT's Comprehension of Perturbed Text through Text Linguistic Features

This paper presents an evaluation method for assessing ChatGPT's ability to understand perturbed text, offering a novel approach to evaluating the robustness of large language models from a text understanding perspective. The proposed method analyzes ChatGPT's comprehension of perturbed text across three dimensions: static sensitivity, dynamic sensitivity, and comprehensive ability. This is achieved through the utilization of traditional metrics and a difference norm based on textual linguistic features. The integration of linguistic features into the assessment of text perturbation degree provides a fast, efficient, and accurate evaluation method that can be easily applied to other large language models. Analysing of results identifies perturbations with high static and dynamic sensitivity for ChatGPT, and assesses its comprehensive comprehension both qualitatively and quantitatively.