In recent years Machine Learning (ML) and deep learning (DL) methods have become more frequently applied in solar physics. Nevertheless, usage of data-driven approaches for tracking of CME-related phenomena is currently limited due to insufficiency of training sets, and having a tool that would make preparation of such training sets a more easy task was one of the intermediate stages of the current research. Semantic image segmentation is a method which allows to label each pixel of an image with a corresponding class of objects. In the case of predicting for every pixel in the image, this task is commonly referred to as dense prediction.
The typical use of convolutional networks is on classification tasks, where the output to an image is a single class label. However, the desired output should include localization, i.e., a class label is supposed to be assigned to each pixel. Moreover, obtaining tens of thousands of verified training images was recently out of reach and could be problematic even with an automated algorithmic segmentation tool which works in pipelines, such as Wavetrack. Therefore, we have turned to approaches that reduce the necessary training dataset sizes, in order to be applied to problems in heliophysics.
We have adapted the U-Net DL model for Solar Eruptive Feature Extraction and Characterization. U-Net is a convolutional neural network that was developed for biomedical image segmentation. This network and training strategy relies on the extended use of data augmentation, in order to incorporate the available annotated samples more efficiently. The architecture consists of a contracting path to capture context, and a symmetric expanding path that enables precise localization. Such a network can be trained end-to-end from a very limited set of images, but still outperforms the previous best method (a sliding-window convolutional network) on the ISBI challenge for segmentation of structures.
The main idea behind the U-Net concept is to supplement a usual contracting network by successive layers, where pooling operators are replaced by upsampling operators. Hence, these layers increase the resolution of the output. In order to localize, high resolution features from the contracting path are combined with the upsampled output. A successive convolution layer can then learn to assemble a more precise output based on this information. The upsampling part also has a large number of feature channels, which allow the network to propagate context information to higher resolution layers. As a consequence, the expansive path is more or less symmetric to the contracting path, and yields a u-shaped architecture. To predict the pixels in the border region of the image, the missing context is extrapolated by mirroring the input image.
One of the most important tasks in machine learning is the preparation of datasets for teaching the supervised models. The typical training set preparation routine goes as follows: after Level-1 solar images (from SDO/AIA in this case) are processed with the Wavetrack software and masks are obtained (here we selected EUV coronal bright fronts), a feature engineering routine is applied in order to enrich the training dataset and improve the learning of the model. Masks and base difference images are rotated and shifted to increase the number of images in the training set. The U-net model was then trained from scratch on the resulting final training set using the 4-GPU MOSAIICS machine at IANAO.
The figure below shows results from application of the model on a test dataset — a comparison between resulting masks obtained with a trained U-net model and control data, where the feature masks were algorithmically obtained with Wavetrack. We find very good agreement between the algorithmic “ground truth” of Wavetrack and the DL model’s output. This work is being written up and will be submitted shortly to a peer-reviewed journal. Generalized pre-trained models for solar features segmentation, alongside with the code will be released in Fall 2022, upon acceptance of the article.
Demonstration of the performance of the U-net model trained for instance segmentation of coronal bright fronts, using Wavetrack output. (Left column) AIA base difference images showing the CBFs for three different snapshots in time. (Middle column) The U-net model output masks for the corresponding images. (Right column) The Wavetrack-defined masks, shown for comparison.