WSM-Net: a weakly supervised multimodal network based on attention mechanism and fusion strategy.

Soft tissue sarcomas are sarcomas that often occur in the soft tissues such as striated muscle and adipose tissue with a high mortality rate. To effectively devise an optimized surgical strategy, it is crucial to comprehensively assess the sarcoma region, leveraging the patient’s MRI images as a critical resource. However, inconsistencies in MRI imaging machine, image parameter settings, etc. often result in differences in the quality of soft tissue sarcoma images, which not only increases the time and effort of manual diagnosis, but also easily leads to misdiagnosis. To investigate ways of developing improved automatic segmentation methods for soft tissue sarcoma and utilizing multi-modal MRI images to delineate the sarcoma region, we have gathered 21,433 horizontal MRI images and 11,741 coronal MRI images from 50 patients diagnosed with soft tissue sarcoma. After conducting a series of preprocessing steps, multiple clinicians annotate the soft tissue sarcoma areas. Two multi-modal image datasets of soft tissue sarcomas are thus obtained. Thereafter, a multi-encoder and single-decoder network is developed to adapt to different modalities of input to segment the sarcoma region. At the same time, a feature fusion strategy is implemented at the bottleneck, and an attention mechanism is integrated into the skip connection to enable the network to capture key semantic features across different modalities. Moreover, the effects of different methods of corrupting input images on network performance for weakly-supervised training are explored. Further experiments demonstrate the availability of our datasets, and our network outperforms the current state-of-the-art methods and produces robust results when one or more modalities are missing.