Validating retinal fundus image analysis algorithms: issues and a proposal

Emanuele Trucco, Alfredo Ruggeri, Thomas Karnowski, Luca Giancardo, Edward Chaum, Jean Pierre Hubschman, Bashir al-Diri, Carol Y. Cheung, Damon Wong, Michael Abramoff, Gilbert Lim, Dinesh Kumar, Philippe Burlina, Neil M. Bressler, Herbert F. Jelinek, Fabrice Meriaudeau, Gwenole Quellec, Tom MacGillivray, Bal Dhillon

    Research output: Contribution to journalArticle

    71 Citations (Scopus)

    Abstract

    This paper concerns the validation of automatic retinal image analysis (ARIA) algorithms. For reasons of space and consistency, we concentrate on the validation of algorithms processing color fundus camera images, currently the largest section of the ARIA literature. We sketch the context (imaging instruments and target tasks) of ARIA validation, summarizing the main image analysis and validation techniques. We then present a list of recommendations focusing on the creation of large repositories of test data created by international consortia, easily accessible via moderated Web sites, including multicenter annotations by multiple experts, specific to clinical tasks, and capable of running submitted software automatically on the data stored, with clear and widely agreed-on performance criteria, to provide a fair comparison.

    Original languageEnglish
    Pages (from-to)3546-3559
    Number of pages14
    JournalInvestigative Ophthalmology & Visual Science
    Volume54
    Issue number5
    DOIs
    Publication statusPublished - 24 May 2013

    Keywords

    • AUTOMATED DETECTION
    • VESSEL SEGMENTATION
    • validation
    • GLAUCOMA ANALYSIS
    • fundus image analysis
    • TO-DISC RATIO
    • COST-EFFECTIVENESS
    • MACULAR DEGENERATION
    • CARDIAC CYCLE
    • OPTIC DISC
    • DIABETIC-RETINOPATHY
    • reference standards
    • SCREENING TOOL

    Cite this

    Trucco, E., Ruggeri, A., Karnowski, T., Giancardo, L., Chaum, E., Hubschman, J. P., ... Dhillon, B. (2013). Validating retinal fundus image analysis algorithms: issues and a proposal. Investigative Ophthalmology & Visual Science, 54(5), 3546-3559. https://doi.org/10.1167/iovs.12-10347
    Trucco, Emanuele ; Ruggeri, Alfredo ; Karnowski, Thomas ; Giancardo, Luca ; Chaum, Edward ; Hubschman, Jean Pierre ; al-Diri, Bashir ; Cheung, Carol Y. ; Wong, Damon ; Abramoff, Michael ; Lim, Gilbert ; Kumar, Dinesh ; Burlina, Philippe ; Bressler, Neil M. ; Jelinek, Herbert F. ; Meriaudeau, Fabrice ; Quellec, Gwenole ; MacGillivray, Tom ; Dhillon, Bal. / Validating retinal fundus image analysis algorithms : issues and a proposal. In: Investigative Ophthalmology & Visual Science. 2013 ; Vol. 54, No. 5. pp. 3546-3559.
    @article{33e6776ec0ca439ba47e9b3d13f0a99a,
    title = "Validating retinal fundus image analysis algorithms: issues and a proposal",
    abstract = "This paper concerns the validation of automatic retinal image analysis (ARIA) algorithms. For reasons of space and consistency, we concentrate on the validation of algorithms processing color fundus camera images, currently the largest section of the ARIA literature. We sketch the context (imaging instruments and target tasks) of ARIA validation, summarizing the main image analysis and validation techniques. We then present a list of recommendations focusing on the creation of large repositories of test data created by international consortia, easily accessible via moderated Web sites, including multicenter annotations by multiple experts, specific to clinical tasks, and capable of running submitted software automatically on the data stored, with clear and widely agreed-on performance criteria, to provide a fair comparison.",
    keywords = "AUTOMATED DETECTION, VESSEL SEGMENTATION, validation, GLAUCOMA ANALYSIS, fundus image analysis, TO-DISC RATIO, COST-EFFECTIVENESS, MACULAR DEGENERATION, CARDIAC CYCLE, OPTIC DISC, DIABETIC-RETINOPATHY, reference standards, SCREENING TOOL",
    author = "Emanuele Trucco and Alfredo Ruggeri and Thomas Karnowski and Luca Giancardo and Edward Chaum and Hubschman, {Jean Pierre} and Bashir al-Diri and Cheung, {Carol Y.} and Damon Wong and Michael Abramoff and Gilbert Lim and Dinesh Kumar and Philippe Burlina and Bressler, {Neil M.} and Jelinek, {Herbert F.} and Fabrice Meriaudeau and Gwenole Quellec and Tom MacGillivray and Bal Dhillon",
    year = "2013",
    month = "5",
    day = "24",
    doi = "10.1167/iovs.12-10347",
    language = "English",
    volume = "54",
    pages = "3546--3559",
    journal = "Investigative Ophthalmology & Visual Science",
    issn = "0146-0404",
    publisher = "Association for Research in Vision and Ophthalmology",
    number = "5",

    }

    Trucco, E, Ruggeri, A, Karnowski, T, Giancardo, L, Chaum, E, Hubschman, JP, al-Diri, B, Cheung, CY, Wong, D, Abramoff, M, Lim, G, Kumar, D, Burlina, P, Bressler, NM, Jelinek, HF, Meriaudeau, F, Quellec, G, MacGillivray, T & Dhillon, B 2013, 'Validating retinal fundus image analysis algorithms: issues and a proposal', Investigative Ophthalmology & Visual Science, vol. 54, no. 5, pp. 3546-3559. https://doi.org/10.1167/iovs.12-10347

    Validating retinal fundus image analysis algorithms : issues and a proposal. / Trucco, Emanuele; Ruggeri, Alfredo; Karnowski, Thomas; Giancardo, Luca; Chaum, Edward; Hubschman, Jean Pierre; al-Diri, Bashir; Cheung, Carol Y.; Wong, Damon; Abramoff, Michael; Lim, Gilbert; Kumar, Dinesh; Burlina, Philippe; Bressler, Neil M.; Jelinek, Herbert F.; Meriaudeau, Fabrice; Quellec, Gwenole; MacGillivray, Tom; Dhillon, Bal.

    In: Investigative Ophthalmology & Visual Science, Vol. 54, No. 5, 24.05.2013, p. 3546-3559.

    Research output: Contribution to journalArticle

    TY - JOUR

    T1 - Validating retinal fundus image analysis algorithms

    T2 - issues and a proposal

    AU - Trucco, Emanuele

    AU - Ruggeri, Alfredo

    AU - Karnowski, Thomas

    AU - Giancardo, Luca

    AU - Chaum, Edward

    AU - Hubschman, Jean Pierre

    AU - al-Diri, Bashir

    AU - Cheung, Carol Y.

    AU - Wong, Damon

    AU - Abramoff, Michael

    AU - Lim, Gilbert

    AU - Kumar, Dinesh

    AU - Burlina, Philippe

    AU - Bressler, Neil M.

    AU - Jelinek, Herbert F.

    AU - Meriaudeau, Fabrice

    AU - Quellec, Gwenole

    AU - MacGillivray, Tom

    AU - Dhillon, Bal

    PY - 2013/5/24

    Y1 - 2013/5/24

    N2 - This paper concerns the validation of automatic retinal image analysis (ARIA) algorithms. For reasons of space and consistency, we concentrate on the validation of algorithms processing color fundus camera images, currently the largest section of the ARIA literature. We sketch the context (imaging instruments and target tasks) of ARIA validation, summarizing the main image analysis and validation techniques. We then present a list of recommendations focusing on the creation of large repositories of test data created by international consortia, easily accessible via moderated Web sites, including multicenter annotations by multiple experts, specific to clinical tasks, and capable of running submitted software automatically on the data stored, with clear and widely agreed-on performance criteria, to provide a fair comparison.

    AB - This paper concerns the validation of automatic retinal image analysis (ARIA) algorithms. For reasons of space and consistency, we concentrate on the validation of algorithms processing color fundus camera images, currently the largest section of the ARIA literature. We sketch the context (imaging instruments and target tasks) of ARIA validation, summarizing the main image analysis and validation techniques. We then present a list of recommendations focusing on the creation of large repositories of test data created by international consortia, easily accessible via moderated Web sites, including multicenter annotations by multiple experts, specific to clinical tasks, and capable of running submitted software automatically on the data stored, with clear and widely agreed-on performance criteria, to provide a fair comparison.

    KW - AUTOMATED DETECTION

    KW - VESSEL SEGMENTATION

    KW - validation

    KW - GLAUCOMA ANALYSIS

    KW - fundus image analysis

    KW - TO-DISC RATIO

    KW - COST-EFFECTIVENESS

    KW - MACULAR DEGENERATION

    KW - CARDIAC CYCLE

    KW - OPTIC DISC

    KW - DIABETIC-RETINOPATHY

    KW - reference standards

    KW - SCREENING TOOL

    UR - http://www.scopus.com/inward/record.url?scp=84883286209&partnerID=8YFLogxK

    U2 - 10.1167/iovs.12-10347

    DO - 10.1167/iovs.12-10347

    M3 - Article

    VL - 54

    SP - 3546

    EP - 3559

    JO - Investigative Ophthalmology & Visual Science

    JF - Investigative Ophthalmology & Visual Science

    SN - 0146-0404

    IS - 5

    ER -