【Objective】Rapid and accurate estimation of soil organic matter (SOM) is crucial for assessing soil fertility, guiding sustainable agricultural management, and supporting carbon accounting at regional and global scales. SOM is a key indicator of soil quality, influencing nutrient cycling, microbial activity, crop productivity, and soil carbon sequestration potential. While reliable, traditional chemical analysis methods are costly, time-consuming, and destructive, making them unsuitable for large-scale or repeated monitoring. Visible and near-infrared (Vis-NIR) spectroscopy provides a rapid, non-destructive, and environmentally friendly alternative for SOM assessment. However, the effectiveness of Vis-NIR spectroscopy is often limited by spectral noise, baseline drift, and low sensitivity to absorption features associated with organic components. Therefore, developing advanced spectral transformation and modeling strategies that can enhance weak spectral signals and extract effective features related to SOM is essential. This study aims to construct a collaborative modeling framework combining fractional-order derivative (FOD) transformation and spectral indices to improve the interpretability and predictive accuracy of Vis-NIR spectral data from typical black soil regions in Northeast China.【Methods】A total of 227 soil samples were collected from representative farmland in the black soil region, an important grain-producing area in Northeast China. The reflectance spectra and SOM content were obtained in the laboratory. The spectral data were processed with FOD ranging from 0 to 2.0 (increment by 0.1 at each step). Two-dimensional (2D) and three-dimensional (3D) spectral indices were calculated to explore the interaction information between different wavelength combinations. The correlation between each spectral index and SOM content was analyzed to determine the most sensitive index. Two machine learning algorithms—random forest (RF) and Cubist—were used to construct prediction models. The input datasets were divided into two categories: (1) FOD-transformed reflectance (FOD dataset); and (2) spectral indices that were most strongly correlated with SOM (index dataset). Four models were thus constructed: FOD-RF, Index-RF, FOD-Cubist, and Index-Cubist. Ten-fold cross-validation was used to evaluate the performance of the model, and the determination coefficient (R²) and root mean square error (RMSE) were used as evaluation indexes. In addition, this study also analyzes the importance of model characteristics to determine the key wavelength or exponential combination that is helpful for SOM prediction.【Results】FOD transform significantly improved the spectral interpretability and enhanced the detection of weak organic absorption characteristics in the Vis-NIR band. The Cubist model with the 0.3-order derivative spectrum exhibited the best performance and provided a validation R² of 0.74. The RF model performs best at higher derivatives (1.6-1.9), and the R² value remains between 0.63 and 0.65. In addition, the correlation between 3D spectral index and SOM is stronger than that of 2D index, and 3D spectral index improves the interpretability of features. The characteristic importance analysis showed that the most sensitive spectral regions predicted by SOM were located within 1410-1880 nm and 2200-2350 nm, corresponding to the overtone and combined vibration of C-H, N-H and O-H functional groups.【Conclusion】The combination of FOD preprocessing and spectral index provides a robust, flexible and scalable framework for estimating SOM using Vis-NIR spectroscopy. This study emphasizes a promising direction in the field of intelligent soil remote sensing monitoring and information technology, and provides methodological progress for digital soil mapping, precise nutrient management and sustainable land management. Its application potential is not only limited to the prediction of SOM, but also extended to a wider range of soil property assessment, which provides a theoretical and technical basis for the construction of intelligent soil information system to support the sustainable development of agriculture in major grain producing areas such as northeast China.