The proposed approach is divided into three steps, shown schematically in Fig. 1: data preprocessing, domain adaptation, and the information extraction approach. Domain adaptation (a technique where a model trained on data from one domain is adapted to perform well in another domain) improves the feature engineering capacity of neural networks21, but is, to the best knowledge of the authors, never been implemented in spectroscopy. The challenge of a small and sparse dataset is adressed by employing domain adaptation based on the network MobileNetv222 (154 layers and ca. 3.5 million parameters) that has been pretrained on the ImageNet dataset23 (containing more than 14 million images and 20000 classes) and is known for its feature extraction capabilities. Domain adaptation is composed of two phases: transfer learning and fine tuning (Fig. 1b). This last step addresses the deficiency that the model has never seen any EEMs in its training data. Finally, to investigate the oxidation process, we introduce the Information Elimination Approach (IEA) (Fig. 1c). The IEA is the key to extracting information from the experimental observations. The method consists in removing part of the spectral information fed to the network as input and evaluating the model performance drop, thus assessing the importance of the removed information by its impact on the predictions.Fig. 1Overview of the phases of the machine learning approach. (a) The data preprocessing phase consists of splitting the dataset for the LOO approach, normalisation of pixel values, and preparation for the MobileNetv2 network input layer by reshaping and creating the three necessary layers. (b) The domain adaptation phase consists of transfer-learning and fine-tuning of the network using the training dataset. The trained network is then evaluated on the test dataset and its performance assessed through the Mean Absolute Error (MAE). (c) Information Elimination Approach (IEA) process diagram. doi references indicate the papers that describe some of the used components.Fluorescence excitation-emission matrices and UV absorption datasetThe dataset comprises fluorescence spectroscopy EEMs and UV absorption spectroscopy data for 24 commercial EVOO, fresh, and in 9 oxidation stages. The oils were chosen to be as heterogeneous as possible in both the origin of production and the price to identify general features of the oxidation process. The characteristics of the oils are described in24. The oils underwent accelerated oxidation at 60 °C to investigate the impact of long-term storage. The effects of thermal degradation were evaluated at ten intervals over a time of 53 days. Fluorescence emission and UV absorption measurements at each stage provided insights into the oxidation process, which is responsible for changes in the oil’s chemical properties. The raw data for the fluorescence spectroscopy EEMs, acquired with an Agilent Cary Eclipse Fluorescence Spectrometer, are stored in CSV files, detailing fluorescence intensities across various excitation and emission wavelengths. Each file contains the fluorescence emission from 300 nm to 800 nm in 2 nm increments (251 values), under excitation at 35 wavelengths ranging from 300 nm to 640 nm in 10 nm increments. All data were acquired under identical conditions and at a controlled temperature of 22 °C within a thermalised sample holder, so the intensities are directly comparable. For UV spectroscopy measurements, the olive oil samples were diluted in isooctane and prepared according to EU regulations18,19. The analysis was carried out in sealed quartz cuvettes under identical conditions with an Agilent Cary 300 UV-Vis spectrophotometer and at a controlled temperature of 22 °C within a thermalised sample holder, collecting the extinction coefficients at four specific wavelengths: 232 nm, 264 nm, 268 nm and 272 nm.The complete dataset is available for download and is described in detail in24 where the link to download the data can be found.Domain adaptationDomain adaptation, in the context of machine learning, is a technique that involves adapting a model trained on data from one domain to perform well on data from another domain. This kind of approach is widely used for improving the model’s generalisation ability and performance on new, unseen data that differ from the model’s training data. To apply this approach in the case described in this work, it is necessary to prepare the data according to a set of steps. Firstly, the dataset was split for leave-one-out (LOO) cross-validation to ensure that each sample was validated exactly once. EEM’s pixel values (fluorescence intensities) ranged from 0 to 1000 counts, therefore, the intensity values were normalised in the dataset by dividing pixel values by a fixed value of 1000 to ensure no data leakage. Furthermore, data were reshaped to meet the input requirements of the MobileNetv2 network22, involving adjustments to \(160 \times 160\) pixels, reformatting to three channels and conversion to an unsigned 8-bit integer format. The three channels were synthetically generated by triplicating the measured fluorescence intensity (so all three channels are identical), as depicted in Fig. 1a. Intensities from Rayleigh scattering, where the emission wavelength matches the excitation wavelength, were retained in the raw data. These intensities are typically considered not relevant, but we wanted to check if the network ignored them or not (the network correctly ignored the Rayleigh scattering). We also tested the results by removing the Rayleigh scattering (either by setting pixel values to 0 around it or by interpolating spectra around it) and we observed no difference in the results. These preprocessing steps are the key to adequately preparing the data for effective feature extraction and subsequent analysis using the pretrained MobileNetv2 network.The model training is then performed according to the following phases, depicted in Fig. 1b.
1.
Phase I – Transfer learning: in a first phase a network is built and trained according to the following recipe:
Start with the MobileNetv222 network with the weights obtained by training it with the ImageNet dataset.
Remove all dense layers from the MobileNetv2 after the backbone.
Freeze the backbone of MobileNetv2. The backbone will not be trained during this phase.
Add the following sequence of layers to the MobileNetv2 backbone: (i) global averaging, (ii) a dropout layer with a factor of 0.2, (iii) a dense layer with 32 neurons with a ReLU activation function, (iv) a dense layer with 16 neurons with a ReLU activation function, (v) a dense layer with 8 neurons with a ReLU activation function, (vi) a dense layer with 1 neuron with an identity activation function.
Train the network with the Adam optimiser. For all parameters \(K_{232}\) and \(K_{268}\) we used the following parameters: learning rate \(\gamma =10^{-4}\), mean squared error (MSE) as loss function, mini-batch size \(b=230\), and 1000 epochs.
2.
Phase II – Fine-tuning: in a second phase the training proceeds according to the following steps.
Unfreeze the last 54 layers of the MobileNetv2 backbone. During the training process, the initial 100 layers of the network remain frozen, with subsequent layers actively trained. The decision on the number of layers to unfreeze was based on multiple tests, which indicated minimal variation in performance when the number of frozen layers ranged from 100 to 120.
Train the network with the Adam optimiser. For the parameters \(K_{232}\) we used the following parameters: learning rate \(\gamma =10^{-6}\), mean squared error (MSE) as loss function, mini-batch size \(b=230\), for 500 epochs. For the parameters \(K_{268}\) we used the following parameters: learning rate \(\gamma =10^{-5}\), MSE as loss function, mini-batch size \(b=32\), and 500 epochs.
The dataset comprises 24 oils and thus has a small size. To perform cross-validation, we chose the leave-one-out (LOO) approach. LOO cross-validation works by using all observations from the original sample except one (23 oils at their oxidation stages) as training data and the left-out single observation (one oil at its 10 oxidation stages) as validation data. This process is repeated so that each oil in the sample is used once as validation data. This method is particularly useful for our small dataset because it maximally uses the data for training while still ensuring that each data point is validated. The dataset is partitioned on the basis of individual oils to prevent data leakage, ensuring that all oxidation stages for a specific oil are grouped together in the same split.Information elimination algorithmThe Information Elimination Algorithm (IEA), presented in this paper, takes advantage of the representation of internal features of large NNs to explain and understand natural processes. It is crucial to emphasise that our aim is not to comprehend the thought process of the model, but rather to utilise the acquired knowledge to gain insights into the oxidation process. This algorithm systematically removes specific regions from the EEMs and observes the impact on the model’s performance, particularly focussing on the prediction errors. Specifically, the algorithm works as follows. Consider the trained model that has been validated on oil j. We proceed by first removing a region of \(5\times 5\) pixels from the EEM of oil \(j\) and assessing the impact on the prediction of \(K_{232}\) and \(K_{268}\) by recording the absolute error (AE). The process begins at the top left corner and proceeds horizontally towards the right edge. Upon reaching the end of a row, it continues from the leftmost side of the row immediately below, repeating this pattern until the entire image have been covered. This process is repeated iteratively until the importance of all portions of the image to the model’s decision-making process is assessed. The result is a heatmap that identifies the spectral bands relevant to the prediction and indicative of the physico-chemical oxidation process. For clarity, the heatmap is smoothed with a Gaussian filter (\(\sigma = 3\) pixels), and contour lines are added to illustrate how AE increases are distributed. This technique leverages the internal feature representation of the network to study natural processes without retraining the model.This method allows us to identify and visualise which parts of the EEMs are most critical for accurate predictions, and thus, as a proxy, to the physico-chemical process of oil oxidation, enhancing the interpretability of the DL model. This approach essentially maps the sensitivity of the model’s predictions to variations in different spectral areas of the data. This model has been inspired by existing methods, such as the backward feature elimination and Y-randomisation approach (see, for example,25), although it uses a fundamentally different approach. To understand the physico-chemical processes underlying the observations, we selectively remove bands associated with different substances and processes, such as chlorophyll (which has a fluorescence emission in the 600 nm – 750 nm range, for example) or primary and secondary oxidation products (which have a fluorescence emission in the 375 nm – 550 nm range), from the EEMs to evaluate their importance for the predictions. Assuming that the NN has developed an internal representation that models these physico-chemical processes, we can deduce which substances or processes are most influential in the oxidation of the oils.The size of the removed region (5 \(\times\) 5 pixels) corresponds to the increment in the excitation wavelengths of the original data (before reshaping). In the original EEMs (35 \(\times\) 251 pixels), the sampling step for the excitation wavelength is 10 nm. 5 pixels in the reshaped matrix (160 \(\times\) 160 pixels), corresponds to 11 nm (1 pixel corresponds to 2.2 nm). In the other dimension (emission wavelengths) one could also take a smaller size. However, because of the broad fluorescence features in the EEMs, this will not lead to better interpretability. Thus, we decided to choose a square shape for the region to be removed.