Radiomics in immuno-oncology

  • Author Footnotes
    † These authors contributed equally to this work.
    Z. Bodalal
    Footnotes
    † These authors contributed equally to this work.
    Affiliations
    Department of Radiology, Netherlands Cancer Institute, Amsterdam, The Netherlands

    GROW School for Oncology and Developmental Biology, Maastricht University, Maastricht, The Netherlands
    Search for articles by this author
  • Author Footnotes
    † These authors contributed equally to this work.
    I. Wamelink
    Footnotes
    † These authors contributed equally to this work.
    Affiliations
    Department of Radiology, Netherlands Cancer Institute, Amsterdam, The Netherlands

    Technical Medicine, University of Twente, Enschede, The Netherlands
    Search for articles by this author
  • S. Trebeschi
    Affiliations
    Department of Radiology, Netherlands Cancer Institute, Amsterdam, The Netherlands

    GROW School for Oncology and Developmental Biology, Maastricht University, Maastricht, The Netherlands
    Search for articles by this author
  • R.G.H. Beets-Tan
    Correspondence
    Correspondence to: Professor Regina G. H. Beets-Tan, Chairwoman of the Department of Radiology, Netherlands Cancer Institute, Plesmanlaan 121, 1066CX Amsterdam, The Netherlands. Tel.: +31-20-512-1728
    Affiliations
    Department of Radiology, Netherlands Cancer Institute, Amsterdam, The Netherlands

    GROW School for Oncology and Developmental Biology, Maastricht University, Maastricht, The Netherlands
    Search for articles by this author
  • Author Footnotes
    † These authors contributed equally to this work.
Open AccessPublished:April 16, 2021DOI:https://doi.org/10.1016/j.iotech.2021.100028

      Highlights

      • A radiological image contains data that encodes the morphological profile.
      • Radiomic features can be linked to clinical endpoints and tumour biology.
      • Combining radiomics with artificial intelligence creates algorithms to support clinical decisions.
      • Radiomics can be used to predict response or adverse events to immunotherapy.
      • Radio(geno)mics carries great promise for precision medicine and immunotherapy.
      With the ongoing advances in imaging techniques, increasing volumes of anatomical and functional data are being generated as part of the routine clinical workflow. This surge of available imaging data coincides with increasing research in quantitative imaging, particularly in the domain of imaging features. An important and novel approach is radiomics, where high-dimensional image properties are extracted from routine medical images. The fundamental principle of radiomics is the hypothesis that biomedical images contain predictive information, not discernible to the human eye, that can be mined through quantitative image analysis. In this review, a general outline of radiomics and artificial intelligence (AI) will be provided, along with prominent use cases in immunotherapy (e.g. response and adverse event prediction) and targeted therapy (i.e. radiogenomics). While the increased use and development of radiomics and AI in immuno-oncology is highly promising, the technology is still in its early stages, and different challenges still need to be overcome. Nevertheless, novel AI algorithms are being constructed with an ever-increasing scope of applications.

      Key words

      Introduction

      Radiological imaging plays a vital role in the detection of cancer,
      • Fass L.
      Imaging and cancer: a review.
      ,
      • Kisilev P.
      • Walach E.
      • Hashoul S.
      • Barkan E.
      • Ophir B.
      • Alpert S.
      Semantic description of medical image findings: structured learning approach.
      disease staging

      Bergan, RC, Raza A, Mundle SD, et al. Imaging in Oncology. (Blake MA, Kalra MK, eds.); 2008.

      Altun E, Jewells VS, Fielding JR. Imaging in Oncology. 2nd ed. (Husband J, Reznek RH, eds.). Taylor and Francis Group; 2015.

      • Reilly R.M.
      Medical Imaging for Health Professionals: Technologies and Clinical Applications.
      and monitoring treatment response.
      • Thoeny H.C.
      • Ross B.D.
      Predicting and monitoring cancer treatment response with DW-MRI.
      ,
      • Eisenhauer E.A.
      • Therasse P.
      • Bogaerts J.
      • et al.
      New response evaluation criteria in solid tumours: revised RECIST guideline (version 1.1).
      With emergence of the RECIST criteria in 2000,
      • Therasse P.
      • Arbuck S.G.
      • Eisenhauer E.A.
      • et al.
      New guidelines to evaluate the response to treatment in solid tumors.
      a standardized set of rules was constructed that enabled clinicians to better categorize tumour response during treatment. The field of oncology, however, is changing rapidly, primarily driven by increasing understanding of the underlying tumour biology. The emergence of novel immunotherapy using immune checkpoint inhibitors and genetically driven targeted therapy has improved the prognosis of several tumour types dramatically.
      • Esfahani K.
      • Roudaia L.
      • Buhlaiga N.
      • Del Rincon S.V.
      • Papneja N.
      • Miller W.H.
      A review of cancer immunotherapy: from the past, to the present, to the future.
      ,
      • Sambi M.
      • Bagheri L.
      • Szewczuk M.R.
      Current challenges in cancer immunotherapy: multimodal approaches to improve efficacy and patient response rates.
      While the clinical benefits provided to patients by these new biological treatments is undeniable, medical imaging has faced a new set of challenges in attempts to characterize the morphological patterns of response, progression and adverse events to targeted therapy and immunotherapy.
      • Dromain C.
      • Beigelman C.
      • Pozzessere C.
      • Duran R.
      • Digklia A.
      Imaging of tumour response to immunotherapy.
      • Erinjeri J.P.
      • Fine G.C.
      • Adema G.J.
      • et al.
      Immunotherapy and the interventional oncologist: challenges and opportunities-a society of interventional oncology white paper.
      • Galldiks N.
      • Kocher M.
      • Ceccon G.
      • et al.
      Imaging challenges of immunotherapy and targeted therapy in patients with brain metastases: response, progression, and pseudoprogression.
      Classical RECIST criteria, for example, proved suboptimal in the context of immunotherapy as patients treated with checkpoint inhibitors responded very differently from patients who received conventional chemo(radio)therapy. This shortcoming led to the creation of the iRECIST guidelines as a possible solution.
      • Seymour L.
      • Bogaerts J.
      • Perrone A.
      • et al.
      iRECIST: guidelines for response criteria for use in trials testing immunotherapeutics.
      The iRECIST criteria were undoubtedly steps in the right direction, especially for patients who would be classified as progressive disease under previous RECIST criteria.
      • Park H.J.
      • Kim G.H.
      • Kim K.W.
      • et al.
      Comparison of RECIST 1.1 and irecist in patients treated with immune checkpoint inhibitors: a systematic review and meta-analysis.
      However, considering that only 20%-40% of patients respond to immunotherapy, there is a pressing need for reliable predictive biomarkers.
      • Sharma P.
      • Hu-Lieskovan S.
      • Wargo J.A.
      • Ribas A.
      Primary, adaptive, and acquired resistance to cancer immunotherapy.
      In comparison with other diagnostic modalities (e.g. pathological biopsies), imaging offers longitudinal insight into the patient's condition. This feature has brought radiology to the centre of therapeutic response and adverse event monitoring. Alongside the abovementioned iRECIST criteria, a number of studies have examined the use of positron emission technology (PET) imaging to assess response (e.g. PECRIT/PERCIMT criteria) and detect immune-related side effects.
      • Cho S.Y.
      • Lipson E.J.
      • Im H.-J.
      • et al.
      Prediction of response to immune checkpoint inhibitor therapy using early-time-point 18F-FDG PET/CT imaging in patients with advanced melanoma.
      • Anwar H.
      • Sachpekidis C.
      • Winkler J.
      • et al.
      Absolute number of new lesions on 18F-FDG PET/CT is more predictive of clinical response than SUV changes in metastatic melanoma patients receiving ipilimumab.
      • Sachpekidis C.
      • Anwar H.
      • Winkler J.
      • et al.
      The role of interim 18F-FDG PET/CT in prediction of response to ipilimumab treatment in metastatic melanoma.
      • Unterrainer M.
      • Ruzicka M.
      • Fabritius M.P.
      • et al.
      PET/CT imaging for tumour response assessment to immunotherapy: current status and future directions.
      Outside the scope of this review, it is believed that molecular imaging will prove undoubtedly helpful as the field of oncology moves towards precision medicine.
      Significant advances in oncologic imaging have been witnessed in the quantitative aspects of radiology, particularly the domain of imaging features. Radiomic research is currently enjoying a rapid boom to identify non-invasive imaging markers/phenotypes capable of being linked with clinical/biological outcomes. The aim of this review is to provide a broad overview of imaging features, primarily radiomics, and highlight applications relevant to immunotherapy (response prediction) and targeted therapy (radiogenomics).

      Imaging features

      Imaging features, obtained from either anatomical or functional images, can be broadly divided into qualitative (or semantic) features and quantitative features. Semantic features are acquired by an experienced reader (i.e. radiologist) who scores different tumour characteristics (lesion shape, number of lesions, lesion intensity, etc.) during the evaluation of medical images. Quantitative features are derived by applying advanced mathematical algorithms to the images (as in the case of radiomics).
      • Kisilev P.
      • Walach E.
      • Hashoul S.
      • Barkan E.
      • Ophir B.
      • Alpert S.
      Semantic description of medical image findings: structured learning approach.
      ,
      • Yip S.S.F.
      • Liu Y.
      • Parmar C.
      • et al.
      Associations between radiologist-defined semantic and automatically computed radiomic features in non-small cell lung cancer.
      ,
      • Bodalal Z.
      • Trebeschi S.
      • Nguyen-Kim T.D.L.
      • Schats W.
      • Beets-Tan R.
      Radiogenomics: bridging imaging and genomics.
      Semantic and quantitative features can be used in statistical or artificial intelligence (AI) models to predict specific clinical endpoints. Individual radiomic features can be combined to form imaging ‘signatures’ or ‘phenotypes’. Table 1 provides an overview of the three major types of imaging feature approaches: semantics, handcrafted radiomics and deep learning radiomics.
      Table 1A key summary of the advantages and disadvantages of the different features
      AdvantagesDisadvantages
      Semantics
      • -
        More easily understandable to humans
      • -
        Many semantic features are already included in the radiological workup of specific tumour types
      • -
        Requires an experienced reader to generate and interpret
      • -
        Vulnerable to inter- and intra-observer variability
      • -
        Limited (dimensionality)
      Handcrafted radiomics
      • -
        Many features represent intuitive morphological features
      • -
        Features encode morphological information beyond the limits of the human eye
      • -
        Clear process pipeline
      • -
        When the feature extraction is performed expertly, artificial intelligence trained on handcrafted radiomic features can perform just as well as deep learning, especially in smaller datasets
      • -
        Requires less data than deep learning
      • -
        Algorithms contain human bias
      • -
        Delineation is required
      • -
        Influenced by different parameters (scanning equipment, pre-processing, scanning protocol)
      Deep learning radiomics
      • -
        Order of magnitude more features
      • -
        No pre-engineered algorithms
      • -
        Often no expert delineation required
      • -
        Can create automatic segmentation
      • -
        Fully automated
      • -
        Greater accuracy in specific tasks compared with traditional computer vision techniques
      • -
        Requires a significant number of samples for training
      • -
        Publically available high-quality well-annotated data in medicine is scarce
      • -
        Black box

       Semantic features

      Semantic features reflect immediately intuitive and often clinically reported image properties such as lesion shape, boundary type, density characteristics, intensity and others (see Figure 1).
      • Kisilev P.
      • Walach E.
      • Hashoul S.
      • Barkan E.
      • Ophir B.
      • Alpert S.
      Semantic description of medical image findings: structured learning approach.
      ,
      • Yip S.S.F.
      • Liu Y.
      • Parmar C.
      • et al.
      Associations between radiologist-defined semantic and automatically computed radiomic features in non-small cell lung cancer.
      ,
      • Rahul P.
      • Schabath M.
      • Balagurunathan Y.
      • et al.
      Explaining deep features using radiologist-defined semantic features and traditional quantitative features.
      • Rogers W.
      • Thulasi Seetha S.
      • Refaee T.A.G.
      • et al.
      Radiomics: from qualitative to quantitative imaging.
      • Opulencia P.
      • Channin D.S.
      • Raicu D.S.
      • Furst J.D.
      Mapping LIDC, RadLexTM, and lung nodule image features.
      • Langlotz C.P.
      RadLex: a new method for indexing online educational materials.
      These features, or image properties, provide additional information to the radiologist during the detection and diagnosis of disease as well as during the monitoring of response to treatment.
      • Rahul P.
      • Schabath M.
      • Balagurunathan Y.
      • et al.
      Explaining deep features using radiologist-defined semantic features and traditional quantitative features.
      Semantic traits were proven to be useful in non-invasive discrimination between malignant and benign computed tomography (CT) lung nodules.
      • Liu Y.
      • Balagurunathan Y.
      • Atwater T.
      • et al.
      Radiological image traits predictive of cancer status in pulmonary nodules.
      In non-small cell lung cancer (NSCLC), semantic features could differentiate between patients with epidermal growth factor receptor (EGFR), anaplastic lymphoma kinase (ALK) and Kirsten rat sarcoma viral oncogene homologue (KRAS) mutations.
      • Rizzo S.
      • Petrella F.
      • Buscarino V.
      • et al.
      CT radiogenomic characterization of EGFR, K-RAS, and ALK mutations in non-small cell lung cancer.
      ,
      • Lv J.
      • Zhang H.
      • Ma J.
      • et al.
      Comparison of CT radiogenomic and clinical characteristics between EGFR and KRAS mutations in lung adenocarcinomas.
      EGFR-mutated nodules tend to be smaller,
      • Rizzo S.
      • Petrella F.
      • Buscarino V.
      • et al.
      CT radiogenomic characterization of EGFR, K-RAS, and ALK mutations in non-small cell lung cancer.
      ,
      • Lv J.
      • Zhang H.
      • Ma J.
      • et al.
      Comparison of CT radiogenomic and clinical characteristics between EGFR and KRAS mutations in lung adenocarcinomas.
      and display pleural retraction
      • Rizzo S.
      • Petrella F.
      • Buscarino V.
      • et al.
      CT radiogenomic characterization of EGFR, K-RAS, and ALK mutations in non-small cell lung cancer.
      and speculation.
      • Lv J.
      • Zhang H.
      • Ma J.
      • et al.
      Comparison of CT radiogenomic and clinical characteristics between EGFR and KRAS mutations in lung adenocarcinomas.
      KRAS-mutated nodules are associated with a rounder shape,
      • Rizzo S.
      • Petrella F.
      • Buscarino V.
      • et al.
      CT radiogenomic characterization of EGFR, K-RAS, and ALK mutations in non-small cell lung cancer.
      the presence of nodules in non-tumour lobes
      • Rizzo S.
      • Petrella F.
      • Buscarino V.
      • et al.
      CT radiogenomic characterization of EGFR, K-RAS, and ALK mutations in non-small cell lung cancer.
      and the presence of multiple small nodules.
      • Lv J.
      • Zhang H.
      • Ma J.
      • et al.
      Comparison of CT radiogenomic and clinical characteristics between EGFR and KRAS mutations in lung adenocarcinomas.
      Larger volume multifocal thoracic lymphadenopathies on CT imaging were found to be indicative of ALK mutations.
      • Halpenny D.F.
      • Riely G.J.
      • Hayes S.
      • et al.
      Are there imaging characteristics associated with lung adenocarcinomas harboring ALK rearrangements?.
      Figure 1
      Figure 1A schematic representation of different semantic features that can be extracted from medical images.
      The different features can be denoted in a feature vector. Feature vectors between the two lesions can differ due to changes in tumour properties.
      Semantic features, however, come with their own set of drawbacks and caveats, most chiefly standardization. Different readers can score the same lesion very differently, resulting in a high degree of variability for semantic features. Features derived using such a subjective method may lead the multidisciplinary team to draw incorrect conclusions regarding the diagnosis or treatment response.
      • Kisilev P.
      • Walach E.
      • Hashoul S.
      • Barkan E.
      • Ophir B.
      • Alpert S.
      Semantic description of medical image findings: structured learning approach.
      Moreover, there is a learning curve for readers to be able to generate reliable semantic features, highlighting the risk of inter- and intra-observer variability.
      • Li Q.
      • Balagurunathan Y.
      • Liu Y.
      • et al.
      Comparison between radiological semantic features and lung-RADS in predicting malignancy of screen-detected lung nodules in the National Lung Screening Trial.
      Inter- and intra-observer variability is often cited as a point of concern in such radiological studies.
      • Wu W.
      • Pierce L.A.
      • Zhang Y.
      • et al.
      Comparison of prediction models with radiological semantic features and radiomics in lung cancer diagnosis of the pulmonary nodules: a case-control study.
      • Charbonnier J.P.
      • Chung K.
      • Scholten E.T.
      • et al.
      Automatic segmentation of the solid core and enclosed vessels in subsolid pulmonary nodules.
      • Wetzel S.G.
      • Cha S.
      • Johnson G.
      • et al.
      Relative cerebral blood volume measurements in intracranial mass lesions: interobserver and intraobserver reproducibility study.
      • Van Riel S.J.
      • Sánchez C.I.
      • Bankier A.A.
      • et al.
      Observer variability for classification of pulmonary nodules on low-dose CT images and its effect on nodule management.
      • Chen S.
      • Qin J.
      • Ji X.
      • et al.
      Automatic scoring of multiple semantic attributes with multi-task feature leverage: a study on pulmonary nodules in CT images.
      • Ridge C.A.
      • Yildirim A.
      • Boiselle P.M.
      • et al.
      Differentiating between subsolid and solid pulmonary nodules at CT: inter- and intraobserver agreement between experienced thoracic radiologists.
      Another limitation of semantic features is the need for visual assessment by human readers. The tumour description is inherently limited to what is immediately discernable to human eyes, meaning that high-dimensional, and possibly valuable, imaging features will not be taken into account.
      • Wu W.
      • Pierce L.A.
      • Zhang Y.
      • et al.
      Comparison of prediction models with radiological semantic features and radiomics in lung cancer diagnosis of the pulmonary nodules: a case-control study.
      ,
      • Sun K.Y.
      • Hu H.T.
      • Chen S.L.
      • et al.
      CT-based radiomics scores predict response to neoadjuvant chemotherapy and survival in patients with gastric cancer.
      • Aerts H.J.W.L.
      • Velazquez E.R.
      • Leijenaar R.T.H.
      • et al.
      Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach.
      • Cook G.J.R.
      • Siddique M.
      • Taylor B.P.
      • Yip C.
      • Chicklore S.
      • Goh V.
      Radiomics in PET: principles and applications.
      • Chicklore S.
      • Goh V.
      • Siddique M.
      • Roy A.
      • Marsden P.K.
      • Cook G.J.R.
      Quantifying tumour heterogeneity in 18F-FDG PET/CT imaging by texture analysis.
      Early attempts were made to automatically predict qualitative features directly from the image; however, this approach has not seen wide adoption, especially with the rise of fully-automated quantitative features.
      • Yip S.S.F.
      • Liu Y.
      • Parmar C.
      • et al.
      Associations between radiologist-defined semantic and automatically computed radiomic features in non-small cell lung cancer.
      ,
      • Depeursinge A.
      • Kurtz C.
      • Beaulieu C.
      • Napel S.
      • Rubin D.
      Predicting visual semantic descriptive terms from radiological image data: preliminary results with liver lesions in CT.
      Although humans excel at pattern recognition, the execution of complex quantitative assessments is often inadequate, making the semantic observations an ideal ‘stepping stone’ for further research to obtain a more quantitative, objective and automatic analysis.
      • Miller G.A.
      The magical number two, plus or minus one: some limits on our capacity for processing musical information.
      ,
      • Wang Y.-X.J.
      • Ng C.K.
      The impact of quantitative imaging in medicine and surgery: charting our course for the future.
      The quantitative field of radiomics thus presents itself as an automatic and objective method to extract these features from medical images, overcoming many of the shortcomings that plagued semantic features.

       Radiomic features

      The fundamental concept driving the field of radiomics is that biomedical images contain useful predictive information, beyond what is visible to the naked eye, that can be decoded employing quantitative image analysis.
      • Rogers W.
      • Thulasi Seetha S.
      • Refaee T.A.G.
      • et al.
      Radiomics: from qualitative to quantitative imaging.
      ,
      • Hectors S.J.
      • Lewis S.
      • Besa C.
      • et al.
      MRI radiomics features predict immuno-oncological characteristics of hepatocellular carcinoma.
      Routine medical images are converted into high-dimensional mineable data, in the form of radiomic features, using mathematical formulae that describe image properties quantitatively.
      • Sun K.Y.
      • Hu H.T.
      • Chen S.L.
      • et al.
      CT-based radiomics scores predict response to neoadjuvant chemotherapy and survival in patients with gastric cancer.
      ,
      • van Griethuysen J.J.
      • Fedorov A.
      • Parmar C.
      • et al.
      Computational radiomics system to decode the radiographic phenotype.
      • Shaver M.M.
      • Kohanteb P.A.
      • Chiou C.
      • et al.
      Optimizing neuro-oncology imaging: a review of deep learning approaches for glioma imaging.
      • Rizzo S.
      • Botta F.
      • Raimondi S.
      • et al.
      Radiomics: the facts and the challenges of image analysis.
      • Schweitzer A.D.
      • Chiang G.C.
      • Ivanidze J.
      • Baradaran H.
      • Young R.J.
      • Zimmerman R.D.
      Computer-extracted texture features to distinguish cerebral radionecrosis from recurrent brain tumors on multiparametric MRI: a feasibility study.
      • Peng L.
      • Parekh V.
      • Huang P.
      • et al.
      Distinguishing true progression from radionecrosis after stereotactic radiation therapy for brain metastases with machine learning and radiomics.
      • Zwanenburg A.
      • Vallières M.
      • Abdalah M.A.
      • et al.
      The image biomarker standardization initiative: standardized quantitative radiomics for high-throughput image-based phenotyping.
      Radiomic feature extraction methods yield a far greater number of features compared with human-derived semantic features, with only a fraction of the effort.
      As almost all patients undergo scans for diagnosis or treatment response assessment, a potentially vast database of sources exists for both retrospective and prospective research. From these digital images, a large number of quantitative features can be extracted and analysed for hypothesis generation or experimentation.
      • Gillies R.J.
      • Kinahan P.E.
      • Hricak H.
      Radiomics: images are more than pictures, they are data.
      Radiomic data can be used alone or in combination with semantic features to improve the understanding of underlying biological, therapeutic and post-therapeutic changes, and treatment-related adverse events.
      • Cook G.J.R.
      • Siddique M.
      • Taylor B.P.
      • Yip C.
      • Chicklore S.
      • Goh V.
      Radiomics in PET: principles and applications.
      The introduction of radiomics into the imaging analysis workflow adds a promising new tool to better sub-stratify patients and customize treatment plans. Quantitative analysis of images can help to differentiate different types of cancer, predict disease outcomes and determine response to treatment.
      • Song S.H.
      • Park H.
      • Lee G.
      • et al.
      Imaging phenotyping using radiomics to predict micropapillary pattern within lung adenocarcinoma.
      • Coroller T.P.
      • Agrawal V.
      • Huynh E.
      • et al.
      Radiomic-based pathological response prediction from primary tumors and lymph nodes in NSCLC.
      • Coroller T.P.
      • Grossmann P.
      • Hou Y.
      • et al.
      CT-based radiomic signature predicts distant metastasis in lung adenocarcinoma.
      • Huang Y.
      • Liu Z.
      • He L.
      • et al.
      Radiomics signature: a potential biomarker for the prediction of disease-free survival in early-stage (I or II) non-small cell lung cancer.
      • Huang Y.Q.
      • Liang C.H.
      • He L.
      • et al.
      Development and validation of a radiomics nomogram for preoperative prediction of lymph node metastasis in colorectal cancer.
      • Wibmer A.
      • Hricak H.
      • Gondo T.
      • et al.
      Haralick texture analysis of prostate MRI: utility for differentiating non-cancerous prostate from prostate cancer and differentiating prostate cancers with different Gleason scores.
      Based on the method of extraction, radiomic features can be divided into two main approaches: handcrafted (or classical) radiomics and deep learning radiomics (see Figure 2).
      • Bodalal Z.
      • Trebeschi S.
      • Nguyen-Kim T.D.L.
      • Schats W.
      • Beets-Tan R.
      Radiogenomics: bridging imaging and genomics.
      ,
      • Rogers W.
      • Thulasi Seetha S.
      • Refaee T.A.G.
      • et al.
      Radiomics: from qualitative to quantitative imaging.
      ,
      • Afshar P.
      • Mohammadi A.
      • Plataniotis K.N.
      • Oikonomou A.
      • Benali H.
      From handcrafted to deep-learning-based cancer radiomics: challenges and opportunities.
      Figure 2
      Figure 2An overview of the two main radiomic pipelines.
      The top path shows classical radiomics, with tumour delineation and manual feature engineering and extraction. After feature extraction, statistical models or machine learning can be used to link the features to the endpoint. The lower path shows deep learning radiomics, where the deep learning network extracts features without human interference to link them to the desired endpoint.
      Handcrafted features are derived from pre-engineered formulas based on visualized aspects of the image.
      • Bodalal Z.
      • Trebeschi S.
      • Nguyen-Kim T.D.L.
      • Schats W.
      • Beets-Tan R.
      Radiogenomics: bridging imaging and genomics.
      ,
      • Rogers W.
      • Thulasi Seetha S.
      • Refaee T.A.G.
      • et al.
      Radiomics: from qualitative to quantitative imaging.
      ,
      • Afshar P.
      • Mohammadi A.
      • Plataniotis K.N.
      • Oikonomou A.
      • Benali H.
      From handcrafted to deep-learning-based cancer radiomics: challenges and opportunities.
      The handcrafted features are often based on intensity histograms, shape attributes and textures, encoding morphological, phenotypic characteristics.
      • Aerts H.J.W.L.
      • Velazquez E.R.
      • Leijenaar R.T.H.
      • et al.
      Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach.
      Before handcrafted features can be computed, regions of interest (ROIs) within the radiological image must first be delineated (segmented). The combination of delineation followed by handcrafted feature extraction is a defining characteristic of classical radiomics.
      • Rogers W.
      • Thulasi Seetha S.
      • Refaee T.A.G.
      • et al.
      Radiomics: from qualitative to quantitative imaging.
      The extracted features can then be used in either statistical or machine learning models. Delta radiomics, a specific subgroup within classical radiomics, compares (and ultimately subtracts) pre- and post-treatment radiomic features (i.e. feature changes) to generate a new set of ‘delta radiomic’ features meant to reflect treatment-induced morphological changes.
      • Mazzei M.A.
      • Nardone V.
      • Di Giacomo L.
      • et al.
      The role of delta radiomics in gastric cancer.
      ,
      • Fave X.
      • Zhang L.
      • Yang J.
      • et al.
      Delta-radiomics features for the prediction of patient outcomes in non-small cell lung cancer.
      As a human-dependent process, manual delineation/segmentation is vulnerable to inter- and intra-observer variability.
      • Traverso A.
      • Wee L.
      • Dekker A.
      • Gillies R.
      Repeatability and reproducibility of radiomic features: a systematic review.
      Furthermore, because handcrafted features are extracted mathematically by human-defined and curated quantitative formulas, bias can occur.
      • Chalkidou A.
      • O'Doherty M.J.
      • Marsden P.K.
      False discovery rates in PET and CT studies with texture features: a systematic review.
      ,
      • Hosny A.
      • Aerts H.J.
      • Mak R.
      Handcrafted versus deep learning radiomics for prediction of cancer therapy response.
      New AI-driven methodologies have emerged to mitigate many of the perceived pitfalls of classical radiomics.
      The second methodological approach to medical image analysis is deep learning, a novel technique that is considered a potential ‘game-changer’ for the field of radiology. Deep learning is a broad term used to describe a collection of neural networks which generate their own features and perform classification without human involvement.
      • Rogers W.
      • Thulasi Seetha S.
      • Refaee T.A.G.
      • et al.
      Radiomics: from qualitative to quantitative imaging.
      It has been shown that deep learning has greater accuracy for specific tasks compared with traditional computer vision techniques.
      • O'Mahony N.
      • Campbell S.
      • Carvalho A.
      • et al.
      Deep learning vs. traditional computer vision.
      While an in-depth explanation of deep neural networks is beyond the scope of this review, a small introduction can be given. Many different types of deep learning networks exist, but all are based on the same premise: an artificial neuron. Artificial neurons model our biological neurons and form the basic building block of neural networks. Each neural network contains multiple neurons arranged in different layers. With each neuron obtaining a different value, combinations of these neurons form features such as lines, shapes, patterns and anatomical structures. To obtain the underlying feature extraction and classification parameters, the network needs to learn; this task is commonly referred to as ‘training’. During training, backpropagation, a phenomenon where the neurons are tuned to learn the data, occurs. Backpropagation ensures that the neuron combinations form different features.
      In the field of medical image analysis, the convolutional neural network (CNN) is the class of deep learning networks employed most often, as their feature design and extraction are free from human interference. CNNs can generate quantitative features several orders of magnitude greater than the methods applied during semantic feature extraction.
      • Rahul P.
      • Schabath M.
      • Balagurunathan Y.
      • et al.
      Explaining deep features using radiologist-defined semantic features and traditional quantitative features.
      Additionally, many deep learning approaches do not require expert delineation/segmentation, further mitigating human bias. With deep learning, feature extraction, selection and classification all occur end-to-end automatically and impartially from human interference.
      Radiomic features aim to quantify the morphological phenotype, but they cannot be used for prediction on their own. Rather, the features are combined together, and subsequently, specific predictive radiomic signatures are identified within statistical models/machine learning algorithms. Statistical models, such as univariate or multivariate analysis, are more commonly used in early proof-of-concept studies, where the objective is merely to find mathematical associations between the radiomic features and the relevant clinical endpoint. Machine learning algorithms provide the ability to construct predictive systems that learn directly from the data. Machine learning and deep learning algorithms provide a set of advantages over classic statistical modelling, namely the fact that there are fewer mathematical assumptions and minimal human involvement is required. AI algorithms may be classifiers that try to predict binary/categorical outcomes (KRAS-mutated/wild-type), regression models that give outputs as a continuous number (i.e. body mass index = 15.2), or survival analysis models designed specifically to predict the probability of survival. When designing the AI aspect of a radiomic study, care should be taken when determining which type of machine learning/deep learning algorithm would be optimal for the data and endpoint in question.

      Applications of radiomics and AI

       Radiomics for prediction of response and adverse events in immunotherapy

      Immunotherapy, as an emerging pillar in cancer treatment, has been researched extensively at the basic, translational and clinical levels.
      • Ribas A.
      • Wolchok J.D.
      Cancer immunotherapy using checkpoint blockade.
      • Meiliana A.
      • Wijaya A.
      • Sriwijaya U.
      Cancer immunotherapy: a review.
      • Zhang H.
      • Chen J.
      Current status and future directions of cancer immunotherapy.
      To date, publications on deep learning/radiomics in the context of immunotherapy are limited. However, as the possibility for synergy between non-invasive imaging markers and immunotherapy gains further recognition, it is anticipated that this will become a major line of medical imaging research.
      Broadly classified radiomic research in immunotherapy has focused on either prediction of immune-related adverse events or response, using either RECIST or survival metrics as the endpoint.
      With the aim of predicting the immunotherapeutic response, Trebeschi et al. extracted radiomic features from CT images of patients with metastatic NSCLC and melanoma who had received anti-PD1 immune checkpoint blockade. Lesions that displayed a more heterogeneous morphologic imaging phenotype with non-uniform density patterns and compact borders were associated with a better response to immunotherapy.
      • Trebeschi S.
      • Drago S.G.
      • Birkbak N.J.
      • et al.
      Predicting response to cancer immunotherapy using noninvasive radiomic biomarkers.
      Imaging markers linked to vessel tortuosity were also used to differentiate between responders and non-responders among patients with NSCLC after nivolumab treatment [area under curve (AUC) = 0.79].
      • Velcheti V.
      • Alilou M.
      • Khunger M.
      • Thawani R.
      • Madabhushi A.
      Changes in computer extracted features of vessel tortuosity on CT scans post-treatment in responders compared to non-responders for non-small cell lung cancer on immunotherapy.
      Subsequent studies have shown similar associations between specific radiomic features and post-immune checkpoint inhibitor response.
      • Nardone V.
      • Tini P.
      • Pastina P.
      • et al.
      Radiomics predicts survival of patients with advanced non-small cell lung cancer undergoing PD-1 blockade using Nivolumab.
      • Alilou M.
      • Bera K.
      • Vaidya P.
      • et al.
      Quantitative vessel tortuosity radiomics on baseline non-contrast lung CT predict response to immunotherapy and are prognostic of overall survival.
      • Khorrami M.
      • Prasanna P.
      • Gupta A.
      • et al.
      Changes in CT radiomic features associated with lymphocyte distribution predict overall survival and response to immunotherapy in non-small cell lung cancer.
      • Mu W.
      • Tunali I.
      • Gray J.E.
      • Qi J.
      • Schabath M.B.
      • Gillies R.J.
      Radiomics of 18F-FDG PET/CT images predicts clinical benefit of advanced NSCLC patients to checkpoint blockade immunotherapy.
      • He B.
      • Dong D.
      • She Y.
      • et al.
      Predicting response to immunotherapy in advanced non-small-cell lung cancer using tumor mutational burden radiomic biomarker.
      • Rundo F.
      • Spampinato C.
      • Banna G.L.
      • Conoci S.
      Advanced deep learning embedded motion radiomics pipeline for predicting anti-PD-1/PD-L1 immunotherapy response in the treatment of bladder cancer: preliminary results.
      Tunali et al. demonstrated that combining radiomic features with clinical data enhanced the predictive power of the radiomic algorithm (AUC = 0.80-0.87 with radiomic features and AUC = 0.77-0.78 without radiomic features).
      • Tunali I.
      • Gray J.E.
      • Qi J.
      • et al.
      Novel clinical and radiomic predictors of rapid disease progression phenotypes among lung cancer patients treated with immunotherapy: an early report.
      A combination of RNA sequence data and CT-derived radiomic features was also able to identify immune infiltration in solid tumours.
      • Sun R.
      • Limkin E.J.
      • Vakalopoulou M.
      • et al.
      A radiomics approach to assess tumour-infiltrating CD8 cells and response to anti-PD-1 or anti-PD-L1 immunotherapy: an imaging biomarker, retrospective multicohort study.
      CT-derived radiomic features combined with delta radiomics was also predictive of an early response to immunotherapy (AUC = 0.857).
      • Wang Z.
      • Mao L.
      • Zhou Z.
      • Si L.
      • Zhu H.
      • Chen X.
      Pilot study of CT-based radiomics model for early evaluation of response to immunotherapy in patients with metastatic melanoma.
      Overall survival and progression-free survival were also linked to tumour skewness in pre-treatment CT scans of patients with metastatic melanoma treated with pembrolizumab.
      • Durot C.
      • Mulé S.
      • Soyer P.
      • Marchal A.
      • Grange F.
      • Hoeffel C.
      Metastatic melanoma: pretreatment contrast-enhanced CT texture parameters as predictive biomarkers of survival in patients treated with pembrolizumab.
      Magnetic resonance imaging (MRI) radiomic features showed an association with overall survival in patients with melanoma brain metastases receiving immune checkpoint inhibitors.
      • Bhatia A.
      • Birger M.
      • Veeraraghavan H.
      • et al.
      MRI radiomic features are associated with survival in melanoma brain metastases treated with immune checkpoint inhibitors.
      Some radiomic projects also focused on adverse events seen in cohorts treated with immunotherapy, such as pneumonitis,
      • Colen R.R.
      • Fujii T.
      • Bilen M.A.
      • et al.
      Radiomics to predict immunotherapy-induced pneumonitis: proof of concept.
      sarcoid-like granulomatous disease of the lung
      • Nguyen-Kim T.D.L.
      • Trebeschi S.
      • Pouw J.E.E.
      • et al.
      Deep learning radiomics distinguishes intrapulmonary disease from metastases in immunotherapy-treated melanoma patients.
      and, most notably, pseudoprogression.
      • Elshafeey N.
      • Kotrotsou A.
      • Hassan A.
      • et al.
      Multicenter study demonstrates radiomic features derived from magnetic resonance perfusion images identify pseudoprogression in glioblastoma.
      • Hu X.
      • Wong K.K.
      • Young G.S.
      • Guo L.
      Support Vector Machine (SVM) multi-parametric MRI identification of pseudoprogression from tumor recurrence in patients with resected glioblastoma.
      • Li M.
      • Tang H.
      • Chan M.D.
      • Zhou X.
      • Qian X.
      DC-AL GAN: pseudoprogression and true tumor progression of glioblastoma multiform image classification based on DCGAN and AlexNet.
      • Jang B.S.
      • Jeon S.H.
      • Kim I.H.
      • Kim I.A.
      Prediction of pseudoprogression versus progression using machine learning algorithm in glioblastoma.
      • Bacchi S.
      • Zerner T.
      • Dongas J.
      • et al.
      Deep learning in the detection of high-grade glioma recurrence using multiple MRI sequences: a pilot study.
      Distinguishing pseudo-progression from genuine progressive disease is an essential yet challenging task, often requiring serial imaging to make the distinction. Elshafeey et al. used radiomic features paired with a support vector machine to classify pseudo-progression from progressive disease in patients with glioblastoma (AUC = 0.89).
      • Elshafeey N.
      • Kotrotsou A.
      • Hassan A.
      • et al.
      Multicenter study demonstrates radiomic features derived from magnetic resonance perfusion images identify pseudoprogression in glioblastoma.
      Similar results were obtained by Hu et al., whose work used imaging features derived from eight different MRI sequences (AUC = 0.94).
      • Hu X.
      • Wong K.K.
      • Young G.S.
      • Guo L.
      Support Vector Machine (SVM) multi-parametric MRI identification of pseudoprogression from tumor recurrence in patients with resected glioblastoma.
      The combination of PET/CT radiomic features with blood parameters was associated with early differentiation of pseudo-progression in patients with metastatic melanoma (AUC = 0.82).
      • Basler L.
      • Gabryś H.S.
      • Hogan S.A.
      • et al.
      Radiomics, tumor volume, and blood biomarkers for early prediction of pseudoprogression in patients with metastatic melanoma treated with immune checkpoint inhibition.
      For deep learning, few papers that differentiate pseudo-progression from progressive disease have been published. An intriguing approach is that of Li et al., who combined two novel deep learning techniques (a generative adversarial network and AlexNet) (AUC = 0.92).
      • Li M.
      • Tang H.
      • Chan M.D.
      • Zhou X.
      • Qian X.
      DC-AL GAN: pseudoprogression and true tumor progression of glioblastoma multiform image classification based on DCGAN and AlexNet.
      Other research in the prediction of pseudo-progression with deep learning achieved comparable results (AUC = 0.83 and 0.80).
      • Jang B.S.
      • Jeon S.H.
      • Kim I.H.
      • Kim I.A.
      Prediction of pseudoprogression versus progression using machine learning algorithm in glioblastoma.
      ,
      • Bacchi S.
      • Zerner T.
      • Dongas J.
      • et al.
      Deep learning in the detection of high-grade glioma recurrence using multiple MRI sequences: a pilot study.

       Radiogenomics for non-invasive insight into tumour biology for precision medicine

      Radiogenomics is the field of research where imaging phenotypes are linked to genetic characteristics (such as gene expression and gene mutation).
      • Rutman A.M.
      • Kuo M.D.
      Radiogenomics: creating a link between molecular diagnostics and diagnostic imaging.
      ,
      • Kuo M.D.
      • Jamshidi N.
      Behind the numbers: decoding molecular phenotypes with radiogenomics – guiding principles and technical considerations.
      The goal is to associate imaging features with previously determined gene expression profiles. In the past years, the term ‘radiogenomics’ has expanded to encompass more than ‘just genomics’. Radiogenomics currently aims to link imaging markers with other biological parameters, such as proteomics, metabolomics and microenvironmental data.
      • Bodalal Z.
      • Trebeschi S.
      • Nguyen-Kim T.D.L.
      • Schats W.
      • Beets-Tan R.
      Radiogenomics: bridging imaging and genomics.
      While still a field of research in its infancy, radiogenomics has the potential to add significant value to the future clinical workup of a patient. In an era where targeted therapies are being used increasingly, knowing the genetic profile of the tumour can help in the formulation of a customized treatment plan. Radiogenomics offers the ability to profile the tumour non-invasively and mitigate several challenges facing traditional biopsy-based approaches. In radiogenomics, the full tumour burden could be analysed non-invasively and cost-effectively, accounting for both intra- and inter-lesional heterogeneity.
      • Lee J.
      • Narang S.
      • Martinez J.J.
      • Rao G.
      • Rao A.
      Associating spatial diversity features of radiologically defined tumor habitats with epidermal growth factor receptor driver status and 12-month survival in glioblastoma: methods and preliminary investigation.
      ,
      • Grimm L.J.
      • Zhang J.
      • Mazurowski M.A.
      Computational approach to radiogenomics of breast cancer: luminal A and luminal B molecular subtypes are associated with imaging features on routine breast MRI extracted using computer vision algorithms.
      Radiogenomics is also able to analyse patient images at several different time points (e.g. longitudinal imaging of the genomic landscape) and tumour sites; this is not always possible with traditional biopsy-based diagnostic modalities.
      • Sadot E.
      • Simpson A.L.
      • Do R.K.G.
      • et al.
      Cholangiocarcinoma: correlation between molecular profiling and imaging phenotypes.
      Radiogenomics has been applied in many different tumour types. In the field of breast cancer, Yamamoto et al. found correlation between quantitative features extracted from dynamic contrast-enhanced MRI scans and early metastasis and metastasis-free survival biomarkers in breast cancer.
      • Yamamoto S.
      • Han W.
      • Kim Y.
      • et al.
      Breast cancer: radiogenomic biomarker reveals associations among dynamic contrast-enhanced MR imaging, long noncoding RNA, and metastasis.
      Additionally, several studies have used non-invasive imaging features to predict the molecular subtype of the tumour (i.e. luminal A
      • Mazurowski M.A.
      • Zhang J.
      • Grimm L.J.
      • Yoon S.C.
      • Silber J.I.
      Radiogenomic analysis of breast cancer: luminal B molecular subtype is associated with enhancement dynamics at MR imaging.
      and luminal B
      • Grimm L.J.
      • Zhang J.
      • Mazurowski M.A.
      Computational approach to radiogenomics of breast cancer: luminal A and luminal B molecular subtypes are associated with imaging features on routine breast MRI extracted using computer vision algorithms.
      ). Luminal A subtype breast cancer is less aggressive and has a better response to therapy than luminal B subtype breast cancer.
      • Grimm L.J.
      • Zhang J.
      • Mazurowski M.A.
      Computational approach to radiogenomics of breast cancer: luminal A and luminal B molecular subtypes are associated with imaging features on routine breast MRI extracted using computer vision algorithms.
      Being able to gain insight into tumour biology non-invasively can impact the treatment plan significantly.
      In lung cancer, radiomic research discovered a relation between specific hypoxia markers (vascular endothelial growth factor and EGFR) and quantitative texture features derived from cholangiocarcinoma CT scans. The presence of this imaging phenotype could then be used to indicate a hypoxic microenvironment non-invasively.
      • Sadot E.
      • Simpson A.L.
      • Do R.K.G.
      • et al.
      Cholangiocarcinoma: correlation between molecular profiling and imaging phenotypes.
      Similarly, radiomic features have been used to predict the presence of many clinically relevant mutations in NSCLC, namely EGFR,
      • Sacconi B.
      • Anzidei M.
      • Leonardi A.
      • et al.
      Analysis of CT features and quantitative texture analysis in patients with lung adenocarcinoma: a correlation with EGFR mutations and survival rates.
      • Caramella C.
      • Bluthgen M.V.
      • Rosellini S.
      • et al.
      Prognostic value of texture analysis and correlation with molecular profile in EGFR mutated/ALK rearranged advanced non-small cell lung cancer (NSCLC).
      • Aerts H.J.W.L.
      • Grossmann P.
      • Tan Y.
      • et al.
      Defining a radiomic response phenotype: a pilot study using targeted therapy in NSCLC.
      • Ozkan E.
      • West A.
      • Dedelow J.A.
      • et al.
      CT gray-level texture analysis as a quantitative imaging biomarker of epidermal growth factor receptor mutation status in adenocarcinoma of the lung.
      KRAS
      • De Jong E.E.C.
      • Van Elmpt W.
      • Hendriks L.E.L.
      • Leijenaar R.T.H.
      • Dingemans A.M.C.
      • Lambin P.
      Radiomic CT features for evaluation of EGFR and KRAS mutation status in patients with advanced NSCLC.
      • Weiss G.J.
      • Ganeshan B.
      • Miles K.A.
      • et al.
      Noninvasive image texture analysis differentiates K-ras mutation from pan-wildtype NSCLC and is prognostic.
      • Rios Velazquez E.
      • Parmar C.
      • Liu Y.
      • et al.
      Somatic mutations drive distinct imaging phenotypes in lung cancer.
      and ALK.
      • Yoon H.J.
      • Sohn I.
      • Cho J.H.
      • et al.
      Decoding tumor phenotypes for ALK, ROS1, and RET fusions in lung adenocarcinoma using a radiomics approach.

      Challenges in radiomics

      Despite its promising results, the field of radiomics also faces several challenges that need to be overcome.

       Generalizability and reproducibility

      A requirement for the implementation of biomarkers in the clinic, imaging or otherwise, is that they are independent, informative and reproducible.
      • McShane L.M.
      In pursuit of greater reproducibility and credibility of early clinical biomarker research.
      Imaging biomarkers that meet these criteria are often derived from large heterogeneous datasets and validated on independent test sets. Reproducibility, however, may be hampered due to differences in radiomic values resulting from differences in scanning equipment,
      • Fave X.
      • MacKin D.
      • Yang J.
      • et al.
      Can radiomics features be reproducibly measured from CBCT images for patients with non-small cell lung cancer?.
      image acquisition protocols,
      • Lecler A.
      • Duron L.
      • Balvay D.
      • et al.
      Combining multiple magnetic resonance imaging sequences provides independent reproducible radiomics features.
      reconstruction algorithms,
      • Zhao B.
      • Tan Y.
      • Tsai W.Y.
      • et al.
      Reproducibility of radiomics for deciphering tumor phenotype with imaging.
      ,
      • Kim H.
      • Park C.M.
      • Lee M.
      • et al.
      Impact of reconstruction algorithms on CT radiomic features of pulmonary tumors: analysis of intra- and inter-reader variability and inter-reconstruction algorithm variability.
      image pre-processing
      • Moradmand H.
      • Aghamiri S.M.R.
      • Ghaderi R.
      Impact of image preprocessing methods on reproducibility of radiomic features in multimodal magnetic resonance imaging in glioblastoma.
      and feature extraction methods.
      • Traverso A.
      • Wee L.
      • Dekker A.
      • Gillies R.
      Repeatability and reproducibility of radiomic features: a systematic review.
      Essentially, this results in a situation where, based on the healthcare centre and scanning equipment, different radiomic signatures can be found to be prognostic/predictive for the same endpoint.
      • Zwanenburg A.
      Radiomics in nuclear medicine: robustness, reproducibility, standardization, and how to avoid data analysis traps and replication crisis.
      Standardization of scan protocols between different institutions could aid in the generalizability and reproducibility of radiomic features/signatures.
      • Boellaard R.
      Standards for PET image acquisition and quantitative data analysis.
      ,
      • Öllers M.
      • Bosmans G.
      • van Baardwijk A.
      • et al.
      The integration of PET-CT scans from different hospitals into radiotherapy treatment planning.
      Initiatives such as the Quantitative Imaging Network initiative have started to address this concern by putting forward a set of standards and best practices for image acquisition and feature extraction.
      • Kumar V.
      • Gu Y.
      • Basu S.
      • et al.
      QIN ‘radiomics: the process and the challenges’.
      Lambin et al. have even proposed a radiomic quality score where specific practices within the methodology of an article would be rewarded or penalized.
      • Lambin P.
      • Leijenaar R.T.H.
      • Deist T.M.
      • et al.
      Radiomics: the bridge between medical imaging and personalized medicine.
      For example, a study with external validation, biological correlation and multiple segmentations per image would receive additional points for its quality score.

       Radiomic workflow

      The radiomic pipeline can be divided into four main steps: image acquisition, ROI delineation/segmentation, feature extraction/selection, and statistical analysis/AI training. Errors can occur at any point in this pipeline. The build-up of these errors is generally termed ‘propagation of the error’.
      As a human-dependent step, ROI delineation/segmentation presents many challenges, most notably inter- and intra-observer variation.
      • Rizzo S.
      • Botta F.
      • Raimondi S.
      • et al.
      Radiomics: the facts and the challenges of image analysis.
      Any variation in the radiomic workflow can result in massive changes to the radiomic feature values.
      • Yip S.S.F.
      • Aerts H.J.W.L.
      Applications and limitations of radiomics.
      Kocak et al. demonstrated that each step in the pipeline was prone to a change of segmentation margin of 2 mm; a margin that can be of high impact and should be taken into account.
      • Kocak B.
      • Ates E.
      • Durmaz E.S.
      • Ulusan M.B.
      • Kilickesmez O.
      Influence of segmentation margin on machine learning–based high-dimensional quantitative CT texture analysis: a reproducibility study on renal clear cell carcinomas.
      Additionally, as expert radiologists are required for accurate delineations, time and availability constraints can often serve as ‘bottlenecks’ for data inclusion in radiomic studies.
      • Beam C.A.
      • Layde P.M.
      • Sullivan D.C.
      Variability in the interpretation of screening mammograms by us radiologists: findings from a national sample.
      ,
      • Armato III, S.G.
      • McNitt-Gray M.F.
      • Reeves A.P.
      • et al.
      The Lung Image Database Consortium (LIDC): an evaluation of radiologist variability in the identification of lung nodules on CT scans.
      Automatic AI-driven ROI delineation approaches have therefore become a growing field of research within radiology, but the technology remains in its infancy as it suffers from its own set of caveats.
      • Hesamian M.H.
      • Jia W.
      • He X.
      • Kennedy P.
      Deep learning techniques for medical image segmentation: achievements and challenges.

      Future directions

      Notwithstanding the challenges mentioned above, the field of radiomics has a bright future, particularly with the research currently being carried out in the following domains.

       The rise of deep learning

      As indicated in the Introduction, classical radiomics is characterized by a mandatory tumour delineation step, followed by a handcrafted feature extraction step. Delineation is vulnerable to inter- and intra-observer variability as well as human bias, limiting reproducibility. AI methods, particularly deep learning, can mitigate many of the limitations in classical radiomics. Alongside the potential for automatic ROI segmentation, CNNs learn image features internally and link these features to the desired outcomes, all without human involvement. The use of automatic AI methods removes many of the potential sources of human-induced bias from the radiomic pipeline.

       Less data-hungry AI

      Generalizable deep learning models typically require a large number of well-annotated samples as training data. AI training datasets used by Google, for example, consist of hundreds of thousands of images.
      • Nichols J.A.
      • Herbert Chan H.W.
      • Baker M.A.B.
      Machine learning: applications of artificial intelligence to imaging and diagnosis.
      One of the significant challenges in medical AI, especially in the case of patients treated with immunotherapy, is the limited amount of publically available high-quality well-annotated medical/imaging data.
      • Shaikh F.A.
      • Kolowitz B.J.
      • Awan O.
      • et al.
      Technical challenges in the clinical application of radiomics.
      ,
      • Colen R.R.
      • Fujii T.
      • Bilen M.A.
      • et al.
      Radiomics to predict immunotherapy-induced pneumonitis: proof of concept.
      Current AI approaches, especially deep learning, achieve better predictive performance when trained on large amounts of well-annotated data. Obtaining this high-quality medical data remains challenging, as evidenced by the fact that many of the articles highlighted in this review had low sample sizes (N < 100).
      Data augmentation can be used to artificially increase the amount of usable training data to overcome this limitation. Being able to use unlabelled data would unlock even more of the hospital's data for AI development. Towards this, unsupervised neural networks have gained significant interest in medical AI research. In unsupervised learning, the network learns features in the data without having access to the ground truth.

       Model explainability techniques: understanding the ‘black box’

      Something that raises concern in all of society is ‘fear of the AI uprising’. This sentiment, which is not well founded, reflects the concern that society has with AI. A significant part of this fear stems from the perceived inexplicability of the predictions, regardless of the accuracy (i.e. the ‘black box’). In the context of medicine, clinicians will not accept and adopt technology that seems to behave without any fundamental basis. However, this fear has already existed for more than 30 years, and new methods have been developed to shed light on the inner workings of neural networks.
      One approach that has been gaining traction in AI radiological research is saliency mapping. A saliency map is an intensity map that represents the individual importance of each individual pixel/voxel in the generation of the model's prediction. In essence, saliency maps allow for the creation of heat maps that can be overlaid on the anatomical image to indicate which parts of the image were deemed important/relevant by the algorithm for its prediction.

       Tumour delineation

      Within the radiomic pipeline, tumour segmentation is considered a ‘bottleneck’ as reliable segmentation is time-consuming and requires experienced radiologists.

      Khan S, Yong SP. A comparison of deep learning and hand crafted features in medical image modality classification. 2016 3rd Int Conf Comput Inf Sci. Published online 2016:633-638.

      AI has two approaches to mitigate this limitation: automatic segmentation algorithms and non-segmentation-dependent deep learning.
      In automatic segmentation, algorithms are created to automatically detect ROIs (based on prior training) and generate delineations/segmentations. Subsequently, features can be extracted from the automatically segmented region. As there is minimal human involvement in this approach, inter- and intra-observer variability can be minimized. One caveat is that in order to create an automatic segmentation algorithm, the AI model must first be trained using human-generated segmentations. It is advised that great care should be paid to the training data to prevent any bias in the data that would lead to a bias in the model.
      • Nanni L.
      • Ghidoni S.
      • Brahnam S.
      Handcrafted vs. non-handcrafted features for computer vision classification.
      Another mitigation strategy for the segmentation ‘bottleneck’ is the use of CNNs to detect either ‘global’ imaging markers (on the entire non-delineated image instead of an ROI) or have readers generate broad ROIs, termed ‘bounding boxes’. With bounding boxes, the radiologist does not need to delineate the tumour in detail, but can instead indicate the general area of the tumour, for example, in the image.

       Multimodal AI models/integrated healthcare systems

      Within the routine clinical workflow, patients undergo a wide array of tests and analysis ranging from blood tests to different types of imaging to organ-specific function tests. In essence, from the moment of admission to the moment of discharge, a single patient can generate large volumes of clinical, pathological, radiological and genetic data (among others), all encoding potentially complementary information. AI methods allow for the integration of these different data types to drastically enhance predictive performance. It is envisaged that, in the future, integration systems will be established in hospitals where different diagnostic data are combined (see Figure 3).
      • Lu M.
      • Zhan X.
      The crucial role of multiomic approach in cancer research and clinically relevant outcomes.
      • Bodalal Z.
      • Trebeschi S.
      • Beets-Tan R.
      Radiomics: a critical step towards integrated healthcare.
      • Zanfardino M.
      • Franzese M.
      • Pane K.
      • et al.
      Bringing radiomics into a multi-omics framework for a comprehensive genotype-phenotype characterization of oncological diseases.
      The resulting clinical decision support system will then aid the multidisciplinary tumour boards to design the best possible management plan on an individual patient basis.
      Figure 3
      Figure 3A schematic representation of multimodal artificial intelligence (AI) models.
      With the increase in available data and knowledge, a combination of different diagnostic data into one integrated diagnostic system can improve AI model outcomes. The outcomes of the integrated diagnostic models can be discussed in multidisciplinary tumour boards to optimize the treatment management plan for patients.

      Conclusions

      Radiomics is a promising research field with future potential for implementation in the clinical workflow. Whether it is in the context of prognostication, response prediction or tumour biology assessment, radiomic features have been studied extensively with a large number of proof-of-concept studies. Notwithstanding the challenges that need to be addressed, significant progress has been made, especially with the implementation of novel AI methods. It is believed that imaging markers, in general, and radiomics, in particular, will find their place within an integrated diagnostics system that harnesses multimodal patient data.

      Funding

      This research did not receive any specific grant from funding agencies in the public, commercial or not-for-profit sectors.

      Disclosure

      The authors have declared no conflicts of interest.

      References

        • Fass L.
        Imaging and cancer: a review.
        Mol Oncol. 2008; 2: 115-152
        • Kisilev P.
        • Walach E.
        • Hashoul S.
        • Barkan E.
        • Ophir B.
        • Alpert S.
        Semantic description of medical image findings: structured learning approach.
        in: Proceedings of the British Machine Vision Conference (BMVC). BMVA Press, London2016: 171.1-171.11
      1. Bergan, RC, Raza A, Mundle SD, et al. Imaging in Oncology. (Blake MA, Kalra MK, eds.); 2008.

      2. Altun E, Jewells VS, Fielding JR. Imaging in Oncology. 2nd ed. (Husband J, Reznek RH, eds.). Taylor and Francis Group; 2015.

        • Reilly R.M.
        Medical Imaging for Health Professionals: Technologies and Clinical Applications.
        Wiley, Hoboken, NJ2019
        • Thoeny H.C.
        • Ross B.D.
        Predicting and monitoring cancer treatment response with DW-MRI.
        J Magn Reson Imaging. 2010; 32: 2-16
        • Eisenhauer E.A.
        • Therasse P.
        • Bogaerts J.
        • et al.
        New response evaluation criteria in solid tumours: revised RECIST guideline (version 1.1).
        Eur J Cancer. 2009; 45: 228-247
        • Therasse P.
        • Arbuck S.G.
        • Eisenhauer E.A.
        • et al.
        New guidelines to evaluate the response to treatment in solid tumors.
        J Natl Cancer Inst. 2000; 92: 205-216
        • Esfahani K.
        • Roudaia L.
        • Buhlaiga N.
        • Del Rincon S.V.
        • Papneja N.
        • Miller W.H.
        A review of cancer immunotherapy: from the past, to the present, to the future.
        Curr Oncol. 2020; 27: 87-97
        • Sambi M.
        • Bagheri L.
        • Szewczuk M.R.
        Current challenges in cancer immunotherapy: multimodal approaches to improve efficacy and patient response rates.
        J Oncol. 2019; 2019: 4508794
        • Dromain C.
        • Beigelman C.
        • Pozzessere C.
        • Duran R.
        • Digklia A.
        Imaging of tumour response to immunotherapy.
        Eur Radiol Exp. 2020; 4: 1-15
        • Erinjeri J.P.
        • Fine G.C.
        • Adema G.J.
        • et al.
        Immunotherapy and the interventional oncologist: challenges and opportunities-a society of interventional oncology white paper.
        Radiology. 2019; 292: 25-34
        • Galldiks N.
        • Kocher M.
        • Ceccon G.
        • et al.
        Imaging challenges of immunotherapy and targeted therapy in patients with brain metastases: response, progression, and pseudoprogression.
        Neuro Oncol. 2020; 22: 17-30
        • Seymour L.
        • Bogaerts J.
        • Perrone A.
        • et al.
        iRECIST: guidelines for response criteria for use in trials testing immunotherapeutics.
        Lancet Oncol. 2017; 18: e143-e152
        • Park H.J.
        • Kim G.H.
        • Kim K.W.
        • et al.
        Comparison of RECIST 1.1 and irecist in patients treated with immune checkpoint inhibitors: a systematic review and meta-analysis.
        Cancers (Basel). 2021; 13: 1-14
        • Sharma P.
        • Hu-Lieskovan S.
        • Wargo J.A.
        • Ribas A.
        Primary, adaptive, and acquired resistance to cancer immunotherapy.
        Cell. 2017; 168: 707-723
        • Cho S.Y.
        • Lipson E.J.
        • Im H.-J.
        • et al.
        Prediction of response to immune checkpoint inhibitor therapy using early-time-point 18F-FDG PET/CT imaging in patients with advanced melanoma.
        J Nucl Med. 2017; 58: 1421-1428
        • Anwar H.
        • Sachpekidis C.
        • Winkler J.
        • et al.
        Absolute number of new lesions on 18F-FDG PET/CT is more predictive of clinical response than SUV changes in metastatic melanoma patients receiving ipilimumab.
        Eur J Nucl Med Mol Imaging. 2018; 45: 376-383
        • Sachpekidis C.
        • Anwar H.
        • Winkler J.
        • et al.
        The role of interim 18F-FDG PET/CT in prediction of response to ipilimumab treatment in metastatic melanoma.
        Eur J Nucl Med Mol Imaging. 2018; 45: 1289-1296
        • Unterrainer M.
        • Ruzicka M.
        • Fabritius M.P.
        • et al.
        PET/CT imaging for tumour response assessment to immunotherapy: current status and future directions.
        Eur Radiol Exp. 2020; 4: 63
        • Yip S.S.F.
        • Liu Y.
        • Parmar C.
        • et al.
        Associations between radiologist-defined semantic and automatically computed radiomic features in non-small cell lung cancer.
        Sci Rep. 2017; 7: 3519
        • Bodalal Z.
        • Trebeschi S.
        • Nguyen-Kim T.D.L.
        • Schats W.
        • Beets-Tan R.
        Radiogenomics: bridging imaging and genomics.
        Abdom Radiol. 2019; 44: 1960-1984
        • Rahul P.
        • Schabath M.
        • Balagurunathan Y.
        • et al.
        Explaining deep features using radiologist-defined semantic features and traditional quantitative features.
        Tomography. 2019; 5: 192-200
        • Rogers W.
        • Thulasi Seetha S.
        • Refaee T.A.G.
        • et al.
        Radiomics: from qualitative to quantitative imaging.
        Br J Radiol. 2020; 93: 20190948
        • Opulencia P.
        • Channin D.S.
        • Raicu D.S.
        • Furst J.D.
        Mapping LIDC, RadLexTM, and lung nodule image features.
        J Digit Imaging. 2011; 24: 256-270
        • Langlotz C.P.
        RadLex: a new method for indexing online educational materials.
        Radiographics. 2006; 26: 1595-1597
        • Liu Y.
        • Balagurunathan Y.
        • Atwater T.
        • et al.
        Radiological image traits predictive of cancer status in pulmonary nodules.
        Clin Cancer Res. 2017; 23: 1442-1449
        • Rizzo S.
        • Petrella F.
        • Buscarino V.
        • et al.
        CT radiogenomic characterization of EGFR, K-RAS, and ALK mutations in non-small cell lung cancer.
        Eur Radiol. 2016; 26: 32-42
        • Lv J.
        • Zhang H.
        • Ma J.
        • et al.
        Comparison of CT radiogenomic and clinical characteristics between EGFR and KRAS mutations in lung adenocarcinomas.
        Clin Radiol. 2018; 73: 590.e1-590.e8
        • Halpenny D.F.
        • Riely G.J.
        • Hayes S.
        • et al.
        Are there imaging characteristics associated with lung adenocarcinomas harboring ALK rearrangements?.
        Lung Cancer. 2014; 86: 190-194
        • Li Q.
        • Balagurunathan Y.
        • Liu Y.
        • et al.
        Comparison between radiological semantic features and lung-RADS in predicting malignancy of screen-detected lung nodules in the National Lung Screening Trial.
        Clin Lung Cancer. 2018; 19: 148-156.e3
        • Wu W.
        • Pierce L.A.
        • Zhang Y.
        • et al.
        Comparison of prediction models with radiological semantic features and radiomics in lung cancer diagnosis of the pulmonary nodules: a case-control study.
        Eur Radiol. 2019; 29: 6100-6108
        • Charbonnier J.P.
        • Chung K.
        • Scholten E.T.
        • et al.
        Automatic segmentation of the solid core and enclosed vessels in subsolid pulmonary nodules.
        Sci Rep. 2018; 8: 1-11
        • Wetzel S.G.
        • Cha S.
        • Johnson G.
        • et al.
        Relative cerebral blood volume measurements in intracranial mass lesions: interobserver and intraobserver reproducibility study.
        Radiology. 2002; 224: 797-803
        • Van Riel S.J.
        • Sánchez C.I.
        • Bankier A.A.
        • et al.
        Observer variability for classification of pulmonary nodules on low-dose CT images and its effect on nodule management.
        Radiology. 2015; 277: 863-871
        • Chen S.
        • Qin J.
        • Ji X.
        • et al.
        Automatic scoring of multiple semantic attributes with multi-task feature leverage: a study on pulmonary nodules in CT images.
        IEEE Trans Med Imaging. 2017; 36: 802-814
        • Ridge C.A.
        • Yildirim A.
        • Boiselle P.M.
        • et al.
        Differentiating between subsolid and solid pulmonary nodules at CT: inter- and intraobserver agreement between experienced thoracic radiologists.
        Radiology. 2016; 278: 888-896
        • Sun K.Y.
        • Hu H.T.
        • Chen S.L.
        • et al.
        CT-based radiomics scores predict response to neoadjuvant chemotherapy and survival in patients with gastric cancer.
        BMC Cancer. 2020; 20: 468
        • Aerts H.J.W.L.
        • Velazquez E.R.
        • Leijenaar R.T.H.
        • et al.
        Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach.
        Nat Commun. 2014; 5: 4406
        • Cook G.J.R.
        • Siddique M.
        • Taylor B.P.
        • Yip C.
        • Chicklore S.
        • Goh V.
        Radiomics in PET: principles and applications.
        Clin Transl Imaging. 2014; 2: 269-276
        • Chicklore S.
        • Goh V.
        • Siddique M.
        • Roy A.
        • Marsden P.K.
        • Cook G.J.R.
        Quantifying tumour heterogeneity in 18F-FDG PET/CT imaging by texture analysis.
        Eur J Nucl Med Mol Imaging. 2013; 40: 133-140
        • Depeursinge A.
        • Kurtz C.
        • Beaulieu C.
        • Napel S.
        • Rubin D.
        Predicting visual semantic descriptive terms from radiological image data: preliminary results with liver lesions in CT.
        IEEE Trans Med Imaging. 2014; 33: 1669-1676
        • Miller G.A.
        The magical number two, plus or minus one: some limits on our capacity for processing musical information.
        Psychol Rev. 1994; 101: 343-352
        • Wang Y.-X.J.
        • Ng C.K.
        The impact of quantitative imaging in medicine and surgery: charting our course for the future.
        Quant Imaging Med Surg. 2011; 1: 1-3
        • Hectors S.J.
        • Lewis S.
        • Besa C.
        • et al.
        MRI radiomics features predict immuno-oncological characteristics of hepatocellular carcinoma.
        Eur Radiol. 2020; 30: 3759-3769
        • van Griethuysen J.J.
        • Fedorov A.
        • Parmar C.
        • et al.
        Computational radiomics system to decode the radiographic phenotype.
        Cancer Res. 2017; 77: e104-e107
        • Shaver M.M.
        • Kohanteb P.A.
        • Chiou C.
        • et al.
        Optimizing neuro-oncology imaging: a review of deep learning approaches for glioma imaging.
        Cancers (Basel). 2019; 11: 1-14
        • Rizzo S.
        • Botta F.
        • Raimondi S.
        • et al.
        Radiomics: the facts and the challenges of image analysis.
        Eur Radiol Exp. 2018; 2: 36
        • Schweitzer A.D.
        • Chiang G.C.
        • Ivanidze J.
        • Baradaran H.
        • Young R.J.
        • Zimmerman R.D.
        Computer-extracted texture features to distinguish cerebral radionecrosis from recurrent brain tumors on multiparametric MRI: a feasibility study.
        Am J Neuroradiol. 2017; 38: 2231-2236
        • Peng L.
        • Parekh V.
        • Huang P.
        • et al.
        Distinguishing true progression from radionecrosis after stereotactic radiation therapy for brain metastases with machine learning and radiomics.
        Int J Radiat Oncol Biol Phys. 2018; 102: 1236-1243
        • Zwanenburg A.
        • Vallières M.
        • Abdalah M.A.
        • et al.
        The image biomarker standardization initiative: standardized quantitative radiomics for high-throughput image-based phenotyping.
        Radiology. 2020; 295: 328-338
        • Gillies R.J.
        • Kinahan P.E.
        • Hricak H.
        Radiomics: images are more than pictures, they are data.
        Radiology. 2016; 278: 563-577
        • Song S.H.
        • Park H.
        • Lee G.
        • et al.
        Imaging phenotyping using radiomics to predict micropapillary pattern within lung adenocarcinoma.
        J Thorac Oncol. 2017; 12: 624-632
        • Coroller T.P.
        • Agrawal V.
        • Huynh E.
        • et al.
        Radiomic-based pathological response prediction from primary tumors and lymph nodes in NSCLC.
        J Thorac Oncol. 2017; 12: 467-476
        • Coroller T.P.
        • Grossmann P.
        • Hou Y.
        • et al.
        CT-based radiomic signature predicts distant metastasis in lung adenocarcinoma.
        Radiother Oncol. 2015; 114: 345-350
        • Huang Y.
        • Liu Z.
        • He L.
        • et al.
        Radiomics signature: a potential biomarker for the prediction of disease-free survival in early-stage (I or II) non-small cell lung cancer.
        Radiology. 2016; 281: 947-957
        • Huang Y.Q.
        • Liang C.H.
        • He L.
        • et al.
        Development and validation of a radiomics nomogram for preoperative prediction of lymph node metastasis in colorectal cancer.
        J Clin Oncol. 2016; 34: 2157-2164
        • Wibmer A.
        • Hricak H.
        • Gondo T.
        • et al.
        Haralick texture analysis of prostate MRI: utility for differentiating non-cancerous prostate from prostate cancer and differentiating prostate cancers with different Gleason scores.
        Eur Radiol. 2016; 25: 2840-2850
        • Afshar P.
        • Mohammadi A.
        • Plataniotis K.N.
        • Oikonomou A.
        • Benali H.
        From handcrafted to deep-learning-based cancer radiomics: challenges and opportunities.
        IEEE Signal Process Mag. 2019; 36: 132-160
        • Mazzei M.A.
        • Nardone V.
        • Di Giacomo L.
        • et al.
        The role of delta radiomics in gastric cancer.
        Quant Imaging Med Surg. 2018; 8: 719-721
        • Fave X.
        • Zhang L.
        • Yang J.
        • et al.
        Delta-radiomics features for the prediction of patient outcomes in non-small cell lung cancer.
        Sci Rep. 2017; 7: 1-11
        • Traverso A.
        • Wee L.
        • Dekker A.
        • Gillies R.
        Repeatability and reproducibility of radiomic features: a systematic review.
        Int J Radiat Oncol Biol Phys. 2018; 102: 1143-1158
        • Chalkidou A.
        • O'Doherty M.J.
        • Marsden P.K.
        False discovery rates in PET and CT studies with texture features: a systematic review.
        PLoS One. 2015; 10: 1-18
        • Hosny A.
        • Aerts H.J.
        • Mak R.
        Handcrafted versus deep learning radiomics for prediction of cancer therapy response.
        Lancet Digit Health. 2019; 1: e106-e107
        • O'Mahony N.
        • Campbell S.
        • Carvalho A.
        • et al.
        Deep learning vs. traditional computer vision.
        Adv Intell Syst Comput. 2020; 943: 128-144
        • Ribas A.
        • Wolchok J.D.
        Cancer immunotherapy using checkpoint blockade.
        Science. 2018; 359: 1350-1355
        • Meiliana A.
        • Wijaya A.
        • Sriwijaya U.
        Cancer immunotherapy: a review.
        Indones Biomed J. 2016; 8: 1-20
        • Zhang H.
        • Chen J.
        Current status and future directions of cancer immunotherapy.
        J Cancer. 2018; 9: 1773-1781
        • Trebeschi S.
        • Drago S.G.
        • Birkbak N.J.
        • et al.
        Predicting response to cancer immunotherapy using noninvasive radiomic biomarkers.
        Ann Oncol. 2019; 30: 998-1004
        • Velcheti V.
        • Alilou M.
        • Khunger M.
        • Thawani R.
        • Madabhushi A.
        Changes in computer extracted features of vessel tortuosity on CT scans post-treatment in responders compared to non-responders for non-small cell lung cancer on immunotherapy.
        J Clin Oncol. 2017; 35: 11518
        • Nardone V.
        • Tini P.
        • Pastina P.
        • et al.
        Radiomics predicts survival of patients with advanced non-small cell lung cancer undergoing PD-1 blockade using Nivolumab.
        Oncol Lett. 2020; 19: 1559-1566
        • Alilou M.
        • Bera K.
        • Vaidya P.
        • et al.
        Quantitative vessel tortuosity radiomics on baseline non-contrast lung CT predict response to immunotherapy and are prognostic of overall survival.
        Med Imaging. 2019; 109501F
        • Khorrami M.
        • Prasanna P.
        • Gupta A.
        • et al.
        Changes in CT radiomic features associated with lymphocyte distribution predict overall survival and response to immunotherapy in non-small cell lung cancer.
        Cancer Immunol Res. 2020; 8: 108-119
        • Mu W.
        • Tunali I.
        • Gray J.E.
        • Qi J.
        • Schabath M.B.
        • Gillies R.J.
        Radiomics of 18F-FDG PET/CT images predicts clinical benefit of advanced NSCLC patients to checkpoint blockade immunotherapy.
        Eur J Nucl Med Mol Imaging. 2020; 47: 1168-1182
        • He B.
        • Dong D.
        • She Y.
        • et al.
        Predicting response to immunotherapy in advanced non-small-cell lung cancer using tumor mutational burden radiomic biomarker.
        J Immunother Cancer. 2020; 8: 1-10
        • Rundo F.
        • Spampinato C.
        • Banna G.L.
        • Conoci S.
        Advanced deep learning embedded motion radiomics pipeline for predicting anti-PD-1/PD-L1 immunotherapy response in the treatment of bladder cancer: preliminary results.
        Electronics. 2019; 8: 1134
        • Tunali I.
        • Gray J.E.
        • Qi J.
        • et al.
        Novel clinical and radiomic predictors of rapid disease progression phenotypes among lung cancer patients treated with immunotherapy: an early report.
        Lung Cancer. 2019; 129: 75-79
        • Sun R.
        • Limkin E.J.
        • Vakalopoulou M.
        • et al.
        A radiomics approach to assess tumour-infiltrating CD8 cells and response to anti-PD-1 or anti-PD-L1 immunotherapy: an imaging biomarker, retrospective multicohort study.
        Lancet Oncol. 2018; 19: 1180-1191
        • Wang Z.
        • Mao L.
        • Zhou Z.
        • Si L.
        • Zhu H.
        • Chen X.
        Pilot study of CT-based radiomics model for early evaluation of response to immunotherapy in patients with metastatic melanoma.
        Front Oncol. 2020; 10: 1-8
        • Durot C.
        • Mulé S.
        • Soyer P.
        • Marchal A.
        • Grange F.
        • Hoeffel C.
        Metastatic melanoma: pretreatment contrast-enhanced CT texture parameters as predictive biomarkers of survival in patients treated with pembrolizumab.
        Eur Radiol. 2019; 29: 3183-3191
        • Bhatia A.
        • Birger M.
        • Veeraraghavan H.
        • et al.
        MRI radiomic features are associated with survival in melanoma brain metastases treated with immune checkpoint inhibitors.
        Neuro Oncol. 2019; 21: 1578-1586
        • Colen R.R.
        • Fujii T.
        • Bilen M.A.
        • et al.
        Radiomics to predict immunotherapy-induced pneumonitis: proof of concept.
        Invest New Drugs. 2018; 36: 601-607
        • Nguyen-Kim T.D.L.
        • Trebeschi S.
        • Pouw J.E.E.
        • et al.
        Deep learning radiomics distinguishes intrapulmonary disease from metastases in immunotherapy-treated melanoma patients.
        Ann Oncol. 2019; 30: v529
        • Elshafeey N.
        • Kotrotsou A.
        • Hassan A.
        • et al.
        Multicenter study demonstrates radiomic features derived from magnetic resonance perfusion images identify pseudoprogression in glioblastoma.
        Nat Commun. 2019; 10: 1-9
        • Hu X.
        • Wong K.K.
        • Young G.S.
        • Guo L.
        Support Vector Machine (SVM) multi-parametric MRI identification of pseudoprogression from tumor recurrence in patients with resected glioblastoma.
        J Magn Reson Imaging. 2011; 33: 296-305
        • Li M.
        • Tang H.
        • Chan M.D.
        • Zhou X.
        • Qian X.
        DC-AL GAN: pseudoprogression and true tumor progression of glioblastoma multiform image classification based on DCGAN and AlexNet.
        Med Phys. 2020; 47: 1139-1150
        • Jang B.S.
        • Jeon S.H.
        • Kim I.H.
        • Kim I.A.
        Prediction of pseudoprogression versus progression using machine learning algorithm in glioblastoma.
        Sci Rep. 2018; 8: 1-9
        • Bacchi S.
        • Zerner T.
        • Dongas J.
        • et al.
        Deep learning in the detection of high-grade glioma recurrence using multiple MRI sequences: a pilot study.
        J Clin Neurosci. 2019; 70: 11-13
        • Basler L.
        • Gabryś H.S.
        • Hogan S.A.
        • et al.
        Radiomics, tumor volume, and blood biomarkers for early prediction of pseudoprogression in patients with metastatic melanoma treated with immune checkpoint inhibition.
        Clin Cancer Res. 2020; 26: 4414-4425
        • Rutman A.M.
        • Kuo M.D.
        Radiogenomics: creating a link between molecular diagnostics and diagnostic imaging.
        Eur J Radiol. 2009; 70: 232-241
        • Kuo M.D.
        • Jamshidi N.
        Behind the numbers: decoding molecular phenotypes with radiogenomics – guiding principles and technical considerations.
        Radiology. 2014; 270: 320-325
        • Lee J.
        • Narang S.
        • Martinez J.J.
        • Rao G.
        • Rao A.
        Associating spatial diversity features of radiologically defined tumor habitats with epidermal growth factor receptor driver status and 12-month survival in glioblastoma: methods and preliminary investigation.
        J Med Imaging. 2015; 2: 1-9
        • Grimm L.J.
        • Zhang J.
        • Mazurowski M.A.
        Computational approach to radiogenomics of breast cancer: luminal A and luminal B molecular subtypes are associated with imaging features on routine breast MRI extracted using computer vision algorithms.
        J Magn Reson Imaging. 2015; 42: 902-907
        • Sadot E.
        • Simpson A.L.
        • Do R.K.G.
        • et al.
        Cholangiocarcinoma: correlation between molecular profiling and imaging phenotypes.
        PLoS One. 2015; 10: 1-12
        • Yamamoto S.
        • Han W.
        • Kim Y.
        • et al.
        Breast cancer: radiogenomic biomarker reveals associations among dynamic contrast-enhanced MR imaging, long noncoding RNA, and metastasis.
        Radiology. 2015; 275: 384-392
        • Mazurowski M.A.
        • Zhang J.
        • Grimm L.J.
        • Yoon S.C.
        • Silber J.I.
        Radiogenomic analysis of breast cancer: luminal B molecular subtype is associated with enhancement dynamics at MR imaging.
        Radiology. 2014; 273: 365-372
        • Sacconi B.
        • Anzidei M.
        • Leonardi A.
        • et al.
        Analysis of CT features and quantitative texture analysis in patients with lung adenocarcinoma: a correlation with EGFR mutations and survival rates.
        Clin Radiol. 2017; 72: 443-450
        • Caramella C.
        • Bluthgen M.V.
        • Rosellini S.
        • et al.
        Prognostic value of texture analysis and correlation with molecular profile in EGFR mutated/ALK rearranged advanced non-small cell lung cancer (NSCLC).
        Eur J Cancer. 2015; 51: S647-S648
        • Aerts H.J.W.L.
        • Grossmann P.
        • Tan Y.
        • et al.
        Defining a radiomic response phenotype: a pilot study using targeted therapy in NSCLC.
        Sci Rep. 2016; 6: 1-8
        • Ozkan E.
        • West A.
        • Dedelow J.A.
        • et al.
        CT gray-level texture analysis as a quantitative imaging biomarker of epidermal growth factor receptor mutation status in adenocarcinoma of the lung.
        Am J Roentgenol. 2015; 205: 1016-1025
        • De Jong E.E.C.
        • Van Elmpt W.
        • Hendriks L.E.L.
        • Leijenaar R.T.H.
        • Dingemans A.M.C.
        • Lambin P.
        Radiomic CT features for evaluation of EGFR and KRAS mutation status in patients with advanced NSCLC.
        Radiother Oncol. 2016; 119: S290-S291
        • Weiss G.J.
        • Ganeshan B.
        • Miles K.A.
        • et al.
        Noninvasive image texture analysis differentiates K-ras mutation from pan-wildtype NSCLC and is prognostic.
        PLoS One. 2014; 9: e100244
        • Rios Velazquez E.
        • Parmar C.
        • Liu Y.
        • et al.
        Somatic mutations drive distinct imaging phenotypes in lung cancer.
        Cancer Res. 2017; 77: 3922-3930
        • Yoon H.J.
        • Sohn I.
        • Cho J.H.
        • et al.
        Decoding tumor phenotypes for ALK, ROS1, and RET fusions in lung adenocarcinoma using a radiomics approach.
        Medicine (Baltimore). 2015; 94: 1-8
        • McShane L.M.
        In pursuit of greater reproducibility and credibility of early clinical biomarker research.
        Clin Transl Sci. 2017; 10: 58-60
        • Fave X.
        • MacKin D.
        • Yang J.
        • et al.
        Can radiomics features be reproducibly measured from CBCT images for patients with non-small cell lung cancer?.
        Med Phys. 2015; 42: 6784-6797
        • Lecler A.
        • Duron L.
        • Balvay D.
        • et al.
        Combining multiple magnetic resonance imaging sequences provides independent reproducible radiomics features.
        Sci Rep. 2019; 9: 1-8
        • Zhao B.
        • Tan Y.
        • Tsai W.Y.
        • et al.
        Reproducibility of radiomics for deciphering tumor phenotype with imaging.
        Sci Rep. 2016; 6: 1-7
        • Kim H.
        • Park C.M.
        • Lee M.
        • et al.
        Impact of reconstruction algorithms on CT radiomic features of pulmonary tumors: analysis of intra- and inter-reader variability and inter-reconstruction algorithm variability.
        PLoS One. 2016; 11: 1-11
        • Moradmand H.
        • Aghamiri S.M.R.
        • Ghaderi R.
        Impact of image preprocessing methods on reproducibility of radiomic features in multimodal magnetic resonance imaging in glioblastoma.
        J Appl Clin Med Phys. 2020; 21: 179-190
        • Zwanenburg A.
        Radiomics in nuclear medicine: robustness, reproducibility, standardization, and how to avoid data analysis traps and replication crisis.
        Eur J Nucl Med Mol Imaging. 2019; 46: 2638-2655
        • Boellaard R.
        Standards for PET image acquisition and quantitative data analysis.
        J Nucl Med. 2009; 50: 11S-20S
        • Öllers M.
        • Bosmans G.
        • van Baardwijk A.
        • et al.
        The integration of PET-CT scans from different hospitals into radiotherapy treatment planning.
        Radiother Oncol. 2008; 87: 142-146
        • Kumar V.
        • Gu Y.
        • Basu S.
        • et al.
        QIN ‘radiomics: the process and the challenges’.
        Magn Reson Imaging. 2013; 30: 1234-1248
        • Lambin P.
        • Leijenaar R.T.H.
        • Deist T.M.
        • et al.
        Radiomics: the bridge between medical imaging and personalized medicine.
        Nat Rev Clin Oncol. 2017; 14: 749-762
        • Yip S.S.F.
        • Aerts H.J.W.L.
        Applications and limitations of radiomics.
        Phys Med Biol. 2016; 61: R150-R166
        • Kocak B.
        • Ates E.
        • Durmaz E.S.
        • Ulusan M.B.
        • Kilickesmez O.
        Influence of segmentation margin on machine learning–based high-dimensional quantitative CT texture analysis: a reproducibility study on renal clear cell carcinomas.
        Eur Radiol. 2019; 29: 4765-4775
        • Beam C.A.
        • Layde P.M.
        • Sullivan D.C.
        Variability in the interpretation of screening mammograms by us radiologists: findings from a national sample.
        Arch Intern Med. 1996; 156: 209-213
        • Armato III, S.G.
        • McNitt-Gray M.F.
        • Reeves A.P.
        • et al.
        The Lung Image Database Consortium (LIDC): an evaluation of radiologist variability in the identification of lung nodules on CT scans.
        Acad Radiol. 2007; 14: 1409-1421
        • Hesamian M.H.
        • Jia W.
        • He X.
        • Kennedy P.
        Deep learning techniques for medical image segmentation: achievements and challenges.
        J Digit Imaging. 2019; 32: 582-596
        • Nichols J.A.
        • Herbert Chan H.W.
        • Baker M.A.B.
        Machine learning: applications of artificial intelligence to imaging and diagnosis.
        Biophys Rev. 2019; 11: 111-118
        • Shaikh F.A.
        • Kolowitz B.J.
        • Awan O.
        • et al.
        Technical challenges in the clinical application of radiomics.
        JCO Clin Cancer Informatics. 2017; : 1-8
        • Colen R.R.
        • Fujii T.
        • Bilen M.A.
        • et al.
        Radiomics to predict immunotherapy-induced pneumonitis: proof of concept.
        Invest New Drugs. 2018; 36: 601-607
      3. Khan S, Yong SP. A comparison of deep learning and hand crafted features in medical image modality classification. 2016 3rd Int Conf Comput Inf Sci. Published online 2016:633-638.

        • Nanni L.
        • Ghidoni S.
        • Brahnam S.
        Handcrafted vs. non-handcrafted features for computer vision classification.
        Pattern Recognit. 2017; 71: 158-172
        • Lu M.
        • Zhan X.
        The crucial role of multiomic approach in cancer research and clinically relevant outcomes.
        EPMA J. 2018; 9: 77-102
        • Bodalal Z.
        • Trebeschi S.
        • Beets-Tan R.
        Radiomics: a critical step towards integrated healthcare.
        Insights Imaging. 2018; 9: 911-914
        • Zanfardino M.
        • Franzese M.
        • Pane K.
        • et al.
        Bringing radiomics into a multi-omics framework for a comprehensive genotype-phenotype characterization of oncological diseases.
        J Transl Med. 2019; 17: 1-21