To address the inadequate accuracy in chatter detection during milling operations, this study proposes a novel milling chatter detection methodology based on an optimized iTransformer-BiGRU-Random Forest (iTBU-RF) hybrid model. Initially, sensitivity analysis of time-frequency domain features is conducted employing Pearson correlation coefficients and significance levels to identify the features most sensitive to chatter detection. Subsequently, a chatter detection model integrating iTBU and RF is constructed. The hyperparameters of the ensemble model are optimized through the Ivy optimization algorithm. Following hyperparameter optimization, the model’s accuracy is substantially enhanced, achieving a maximum improvement of 2.40% compared to the pre-optimized configuration. Upon feature optimization, the model maintains superior classification performance while simultaneously reducing training time from 153.83 seconds to 116.74 seconds, thereby improving computational efficiency by approximately 24.11%. In comparison with benchmark methodologies, the proposed approach demonstrates optimal performance across all evaluation metrics, including accuracy. This investigation provides a novel technological framework for enhancing the precision of chatter detection in milling operations.