Text Data Augmentation: Improving Classification Accuracy at the Expense of Calibration?
Primary Investigator:
Julia (Taylor) Rayz
Geetanjali Bihani
Abstract
Text data augmentation (TDA) has been shown to improve generalization in neural NLP pipelines by having a 'regularizing effect'. Training on additional augmented examples provides more space for the model to learn class decision boundaries. Although TDA creates higher data diversity and reduces model overfitting, it remains unclear whether it enhances the model's confidence in correct decisions. To address this gap, we study the impact of TDA-induced 'generalization' on classification decisions and associated confidence levels. We assess the resulting calibration error and focus on two particular subsets of predictions, 1) incorrect but overconfident classifications and 2) correct but underconfident classifications. Our results show that TDA improves accuracy at the cost of model reliability. As we apply more TDA, the model's confidence in all decisions increases, regardless of their correctness. This calls for improved methods of TDA that also account for miscalibration and reduce calibration error in NLP tasks.