Cross-modality sub-image retrieval using contrastive multimodal image representations

Müller, H., Michoux, N., Bandon, D. & Geissbuhler, A. A review of content-based image retrieval systems in medical applications-clinical benefits and future directions. Int. J. Med. Inform. 73, 1–23 (2004).Article 
PubMed 

Google Scholar 
Kapoor, R., Sharma, D. & Gulati, T. State of the art content based image retrieval techniques using deep learning: A survey. Multimed. Tools Appl. 80, 29561–29583. https://doi.org/10.1007/s11042-021-11045-1 (2021).Article 

Google Scholar 
Qayyum, A., Anwar, S. M., Awais, M. & Majid, M. Medical image retrieval using deep convolutional neural network. Neurocomputing 266, 8–20. https://doi.org/10.1016/j.neucom.2017.05.025 (2017).Article 

Google Scholar 
Mbilinyi, A. & Schuldt, H. Cross-modality medical image retrieval with deep features. In International Conference on Bioinformatics and Biomedicine (BIBM), 2632–2639. https://doi.org/10.1109/BIBM49941.2020.9313211 (2020).Putzu, L., Loddo, A. & Ruberto, C. D. Invariant moments, textural and deep features for diagnostic MR and CT image retrieval. In Computer Analysis of Images and Patterns, 287–297 (Springer, 2021).Kong, B., Supancic, J. S., Ramanan, D. & Fowlkes, C. C. Cross-domain forensic shoeprint matching. In British Machine Vision Conference (BMVC) (2017).Sivic, J. & Zisserman, A. Efficient visual search of videos cast as text retrieval. IEEE Trans. Pattern Anal. Mach. Intell. 31, 591–606 (2009).Article 
PubMed 

Google Scholar 
Philbin, J., Chum, O., Isard, M., Sivic, J. & Zisserman, A. Object retrieval with large vocabularies and fast spatial matching. In 2007 IEEE Conference on Computer Vision and Pattern Recognition, 1–8 (2007).Caicedo, J. C., Cruz, A. & Gonzalez, F. A. Histopathology image classification using bag of features and kernel functions. In Artificial Intelligence in Medicine, 126–135 (Springer, 2009).Cao, B., Araujo, A. & Sim, J. Unifying deep local and global features for image search. In Computer Vision—ECCV 2020 (eds Vedaldi, A. et al.) 726–743 (Springer, 2020).Chapter 

Google Scholar 
Hedge, N., Hipp, J. & Liu, Y. Similar image search for histology: SMILY. npj Digit. Med. 1, 2. https://doi.org/10.1038/s41746-019-0131-z (2019).Article 

Google Scholar 
Komura, D. et al. Luigi: Large-scale histopathological image retrieval system using deep texture representations. bioRxivhttps://doi.org/10.1101/345785 (2018).Article 

Google Scholar 
Otálora, S., Schaer, R., Jimenez-del Toro, O., Atzori, M. & Müller, H. Deep learning based retrieval system for gigapixel histopathology cases and open access literature. bioRxivhttps://doi.org/10.1101/408237 (2018).Article 

Google Scholar 
Chen, P. et al. Interactive thyroid whole slide image diagnostic system using deep representation. Comput. Methods Programs Biomed. 195, 105630. https://doi.org/10.1016/j.cmpb.2020.105630 (2020).Article 
PubMed 
PubMed Central 

Google Scholar 
Barisoni, L., Lafata, K. J., Hewitt, S. M., Madabhushi, A. & Balis, U. G. J. Digital pathology and computational image analysis in nephropathology. Nat. Rev. Nephrol. 16, 669–685. https://doi.org/10.1038/s41581-020-0321-6 (2020).Article 
PubMed 
PubMed Central 

Google Scholar 
Hristu, R. et al. Influence of hematoxylin and eosin staining on the quantitative analysis of second harmonic generation imaging of fixed tissue sections. Biomed. Opt. Express 12, 5829–5843. https://doi.org/10.1364/BOE.428701 (2021).Article 
CAS 
PubMed 
PubMed Central 

Google Scholar 
Keikhosravi, A., Bredfeldt, J. S., Sagar, A. K. & Eliceiri, K. W. Chapter 28—Second-harmonic generation imaging of cancer. In Quantitative Imaging in Cell Biology, Methods in Cell Biology, Vol. 123, 531–546 (Academic Press, 2014).Zhou, X., Han, X., Li, H., Wang, J. & Liang, X. Cross-domain image retrieval: Methods and applications. J. Multimed. Inf. Retr. 11, 199–218. https://doi.org/10.1007/s13735-022-00244-7 (2022).Article 
CAS 

Google Scholar 
Liu, F. et al. Infrared and visible cross-modal image retrieval through shared features. IEEE Trans. Circuits Syst. Video Technol. 31, 4485–4496. https://doi.org/10.1109/TCSVT.2020.3048945 (2021).Article 

Google Scholar 
Wu, A., Zheng, W.-S., Yu, H.-X., Gong, S. & Lai, J. RGB-infrared cross-modality person re-identification. In International Conference on Computer Vision (ICCV), 5390–5399. https://doi.org/10.1109/ICCV.2017.575 (2017).Xiong, W., Xiong, Z., Cui, Y. & Lv, Y. A discriminative distillation network for cross-source remote sensing image retrieval. IEEE J. Sel. Top. Appl. Earth Observ. Remote Sens. 13, 1234–1247. https://doi.org/10.1109/JSTARS.2020.2980870 (2020).Article 
ADS 

Google Scholar 
Xiong, W., Lv, Y., Zhang, X. & Cui, Y. Learning to translate for cross-source remote sensing image retrieval. IEEE Trans. Geosci. Remote Sens. 58, 4860–4874. https://doi.org/10.1109/TGRS.2020.2968096 (2020).Article 
ADS 

Google Scholar 
Li, Y., Zhang, Y., Huang, X. & Ma, J. Learning source-invariant deep hashing convolutional neural networks for cross-source remote sensing image retrieval. IEEE Trans. Geosci. Remote Sens. 56, 6521–6536. https://doi.org/10.1109/TGRS.2018.2839705 (2018).Article 
ADS 

Google Scholar 
Lin, H. et al. TC-Net for ISBIR: Triplet classification network for instance-level sketch based image retrieval. In Proceedings of ACM International Conference on Multimedia, 1676–1684. https://doi.org/10.1145/3343031.3350900 (ACM, 2019).Zhang, J. et al. Generative domain-migration hashing for sketch-to-image retrieval. In Computer Vision—ECCV 2018 (eds Ferrari, V. et al.) 304–321 (Springer, Cham, 2018).Chapter 

Google Scholar 
Bai, C., Chen, J., Ma, Q., Hao, P. & Chen, S. Cross-domain representation learning by domain-migration generative adversarial network for sketch based image retrieval. J. Vis. Commun. Image Represent. 71, 102835. https://doi.org/10.1016/j.jvcir.2020.102835 (2020).Article 

Google Scholar 
Bui, T., Ribeiro, L., Ponti, M. & Collomosse, J. Compact descriptors for sketch-based image retrieval using a triplet loss convolutional neural network. Comput. Vis. Image Underst. 164, 27–37 (2017).Article 

Google Scholar 
Lei, H. et al. A new algorithm for sketch-based fashion image retrieval based on cross-domain transformation. Wirel. Commun. Mob. Comput.https://doi.org/10.1155/2021/5577735 (2021).Article 

Google Scholar 
Yang, E. et al. Deep Bayesian hashing with center prior for multi-modal neuroimage retrieval. IEEE Trans. Med. Imaging 40, 503–513. https://doi.org/10.1109/TMI.2020.3030752 (2021).Article 
PubMed 
PubMed Central 

Google Scholar 
Fang, J., Fu, H. & Liu, J. Deep triplet hashing network for case-based medical image retrieval. Med. Image Anal. 69, 101981. https://doi.org/10.1016/j.media.2021.101981 (2021).Article 
PubMed 

Google Scholar 
Pielawski, N. et al. CoMIR: Contrastive multimodal image representation for registration. In Advances in Neural Information Processing Systems, Vol. 33, 18433–18444 (Curran Associates, Inc., 2020).Bay, H., Ess, A., Tuytelaars, T. & Van Gool, L. SURF: Speeded up robust features. Comput. Vis. Image Underst. (CVIU) 110, 346–359 (2008).Article 

Google Scholar 
Wetzer, E., Breznik, E., Lindblad, J. & Sladoje, N. Re-ranking strategies in cross-modality microscopy retrieval. In IEEE ISBI 2022 International Symposium on Biomedical Imaging, 28–31 March, 2022, Kolkata, India (Institute of Electrical and Electronics Engineers (IEEE), 2022).Eliceiri, K., Li, B. & Keikhosravi, A. Multimodal biomedical dataset for evaluating registration methods (patches from TMA cores). zenodohttps://zenodo.org/record/3874362 (2020).Conklin, M. W. et al. Aligned collagen is a prognostic signature for survival in human breast carcinoma. Am. J. Pathol. 3, 1221–1232. https://doi.org/10.1016/j.ajpath.2010.11.076 (2011).Article 

Google Scholar 
Jégou, S., Drozdzal, M., Vazquez, D., Romero, A. & Bengio, Y. The one hundred layers tiramisu: Fully convolutional densenets for semantic segmentation. In Proceedings of CVPR Workshops, 11–19 (2017).Isola, P., Zhu, J.-Y., Zhou, T. & Efros, A. A. Image-to-image translation with conditional adversarial networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017).Zhu, J.-Y., Park, T., Isola, P. & Efros, A. A. Unpaired image-to-image translation using cycle-consistent adversarial networks. In International Conference on Computer Vision (ICCV) (2017).Lowe, D. G. Object recognition from local scale-invariant features. In Proceedings of International Conference on Computer Vision (ICCV), Vol. 2, 1150–1157 (1999).He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770–778. https://doi.org/10.1109/CVPR.2016.90 (2016).DeVille, J. S., Kihara, D. & Sit, A. 2DKD: A toolkit for content-based local image search. Source Code Biol. Med.https://doi.org/10.1186/s13029-020-0077-1 (2020).Article 
PubMed 
PubMed Central 

Google Scholar 
Babenko, A. & Lempitsky, V. Aggregating local deep features for image retireval. In International Conference on Computer Vision (ICCV) (2015).Bhandi, V. & Sumithra Devi, K. A. Image retrieval by fusion of features from pre-trained deep convolution neural networks. In International Conference on Advanced Technologies in Intelligent Control, Environment, Computing Communication Engineering (ICATIECE), 35–40. https://doi.org/10.1109/ICATIECE45860.2019.9063814 (2019).Jun, H., Ko, B., Kim, Y., Kim, I. & Kim, J. Combination of multiple global descriptors for image retrieval. CoRR (2019).Sit, A. & Kihara, D. Comparison of image patches using local moment invariants. IEEE Trans. Image Process. 23, 2369–2379. https://doi.org/10.1109/TIP.2014.2315923 (2014).Article 
ADS 
MathSciNet 
PubMed 

Google Scholar 
Song, J., Yu, Q., Song, Y.-Z., Xiang, T. & Hospedales, T. M. Deep spatial-semantic attention for fine-grained sketch-based image retrieval. In International Conference on Computer Vision (ICCV), 5552–5561. https://doi.org/10.1109/ICCV.2017.592 (2017).