Abstract
This paper presents an automatic approach for measurement of the superficial spreading depth of cutaneous melanomas based on microscopic hyperspectral imaging technology. To extract the skin granular layer， an edge detection method combined with kernel minimum noise fraction is proposed. Then least squares support vector machine based on characteristic spectrum supervision is used to identify malignant melanocytes. The measurement of tumor superficial spreading depth depends on the vertical distance from the skin granular layer to the deepest malignant melanocytes. Experimental results illustrate that the proposed method is possible to provide an effective reference for the diagnosis and treatment of cutaneous melanoma.
According to the GLOBOCAN2012 database of World Health Organization （WHO）， in 2015， the number of new cases of cutaneous melanomas （CM） worldwide reached 250 178， including 130 800 males and 119 378 females. There were 60 098 global cutaneous melanoma deaths， including 34 143 males and 25 955 female
Detecting melanoma in skin tissue and determining the size of the lesion area are common diagnostic procedures. Researchers at home and abroad have combined numerous modern optical imaging techniques with image processing methods for medical diagnosis. Innovative approach such as dermoscopy can be well applied clinicall
The medical field recognizes that pathological diagnosis remains the gold standard for complex tumor diagnosis. In the treatment of melanoma， the depth of tumor invasion （DoI）， namely Breslow thickness in skin tissue is a crucial prognostic factor
The application of hyperspectral imaging technology in biomedical imaging points out a new direction for the identification and diagnosis of cutaneous melanoma
Microscopic hyperspectral imaging （MHSI） technology highlights the abundant spatial and spectral information in the pathological structure of cutaneous melanoma. In this article， to measure the superficial spreading depth of cutaneous melanoma， we use a segmentation approach combined a kernel minimum noise fraction （KMNF） algorithm and two morphological edge detection ways to identify the skin granular layer. Subsequently， a least squares support vector machine based on characteristic spectrum supervision （CSS-LSSVM） segmentation algorithm is applied to separate malignant melanocytes. Comparing the results of different algorithms， our method has realistic significance and may offer a quantitative basis for CM pathological diagnosis.
Hyperspectral images can provide an assessment of tissue pathophysiology based on the spectral characteristics of different tissue
Fig.1 The main flowchart of the proposed methods
图1 本论文方法流程图
Conventional pathological sections preparation is extremely important in histopathology and is the basis of all pathological analyses. The main steps involved in the production process are tissue sample extraction， fixation， dehydration， waxing， embedding， slicing， baking， dyeing， and sealing
The structure of skin tissue is complex， and skin melanoma has a wide variety of types. However， under the microscopic field of view， it is not possible to completely contain all the structures of the skin tissue. The cutaneous melanoma samples used in this research are superficial spreading melanoma （SSM）. The lesions are mostly in the epidermis and dermis. The lesion boundaries are obvious， and melanocytes are scattered individually or in-cluster in various layers of the epidermis， with few tumor cell nests and no top-to-bottom maturation. Therefore， we define the tumor depth of invasion within the SSM sample as superficial spreading depth in our experiment. As shown in
Fig.2 （a） Normal skin tissue, （b） melanoma sample
图2 （a） 正常皮肤组织, （b） 黑素瘤样本
The microscopic hyperspectral imaging system as shown in
Fig.3 The main schematic and optical path diagram of microscopic hyperspectral imaging system
图3 显微高光谱成像系统主要结构及光路图
In hyperspectral data acquisition， single-wavelength image data are consistent with ordinary two-dimensional image data that contain only spatial information （
Fig.4 （a） Spectrum of a sampling point, （b） the data cube of the cutaneous melanoma, and （c） the single band image
图4 （a）从数据中提取的光谱曲线,（b） 黑色素瘤样本数据立方体,（c） 单波段图像
When the MHSI system is collecting image data， changes in external conditions such as illumination intensity， CCD electrical noise， and sample background have a certain impact on the quality of imaged data. Since the absorption characteristics of biological tissues to the spectrum are much smaller than the transmission and reflection characteristics of slide， the unprocessed sample data exhibit similar spectral characteristics in the same wavelength range. The spectral characteristics of target tissue are weakened， which cannot accurately reflect biochemical features of the sample. Hence samples’ spectrum should be corrected first before data processing.
Lambert-Beer's （LB） law
| , | （1） |
| . | （2） |
In this experiment， we simultaneously acquired hyperspectral images of biological tissues and blank areas of the same slides. The transmittance and absorbance of the cutaneous melanoma samples are：
| , | （3） |
| ， | （4） |
where D（n，m；λ） is the pixel value of the target sample in the
The purpose of this section is to achieve segmentation of the skin granular layer. In H&E-staining pathological images， the limitation of RGB information makes it difficult to distinguish different tissues with extremely high similarity. However， in microscopic hyperspectral images， the spectral characteristics of granular layer are different from other tissues. We utilized a KMNF and morphological filtering segmentation method to provide an accurate starting boundary for the calculation of melanoma superficial spreading depth.
Minimum noise fraction （MNF）
KMNF
| , | （5） |
| . | （6） |
After linearization， the MNF maximization is：
| . | （7） |
If a hyperspectral image is regarded as a data set of n pixels and p spectral bands， then
| . | （8） |
Let a map of be Rn， then the maximization of KMNF is
| ， | （9） |
where Rn is a two-dimensional matrix， the element of the asymmetric matrix = is the kernel function k（xi， xNj）， i， j=1，···，n， and the average of and kernel matrix columns is zero. An important result of the dual representation is that the dimension of feature space no longer affects calculation. The kernel function is added based on the minimum noise separation transform， and the mapping from original data space to feature space can be completed. The first few dimensions contain a large number of eigenvalues of the image data and a small noise. As the dimension increases， the noise becomes larger and the eigenvalue becomes smaller.
KMNF retains nonlinear characteristics of hyperspectral data effectively. In cutaneous melanoma samples， the granular layer exhibits a black band in hyperspectral grayscale image. The next step is to extract main morphology of the granular layer by morphological filtering. A conventional morphological operator is an N by N （N=3，5，7...） square filter. In this section， in order to better adapt to the shape of the skin granular layer， a multi-size filter operator tending to the diamond shape is designed. We mixed three kinds of filter operators to achieve better morphological extraction results. Namely， the skin granular layer uses a 3 by 3 template， the epidermis layer except the granular layer adopts a 5 by 5 template， and the irregularly shaped fiber structure and malignant melanocytes employ a 7 by 7 template.
The skin granular layer is generally used as starting boundary for the measurement of skin melanoma superficial spreading depth
We studied the image contour extraction method based on level set segmentation
In pathological diagnosis of cutaneous melanoma， the morphology and distribution of malignant melanocytes are one of the main diagnostic criteria. It is also the termination boundary for calculating melanoma superficial spreading depth. In this research， we utilized the least squares support vector machine method based on characteristic spectrum supervision to segment melanocytes.
The least squares support vector machine （LSSVM） is an improved data mining-based machine learning method
| . | （10） |
The least squares support vector machine based on characteristic spectrum supervision （CSS-LSSVM） segmentation method add characteristic spectra of the target to match in the LSSVM model and segment the LSSVM results again to achieve higher accuracy. The matching way calculates spectral similarity between melanocytes and the pixel of segmentation image one by one and determine the type of the segmented pixel. A spectral angle matching method is typically used for evaluation of spectral similarity. It is known that the number of wavelengths of experimental sample is λ， the spectral characteristic of a single pixel is X=［x1，x2，···，xλ］， and the characteristic spectra of the melanocytes are X’=［x’1，x’2，···，x’λ］， then the spectral similarity
| . | （11） |
The threshold α is usually set artificially， beyond which it is considered not to be a homogeneous biological tissue. The CSS-LSSVM segmentation method includes the following steps：
1） The spectra of region of interest （ROI） are selected as the training set.
2） A feature spectrum library is constructed， and the average spectrum of a plurality of adjacent pixels in the target region is taken as a characteristic spectrum.
3） The ROI data set is trained to establish LSSVM classification model.
The spectra of remaining pixels are sequentially inputted to the LSSVM model. If the model output is determined to be the target class， output is the target class； if not， the pixel’s spectrum is matched with the feature spectral library. If the matching result is a target class， it is determined to be the target class， otherwise， it is not.
The samples used in this section are magnified 100-fold images of cutaneous melanoma， as they contain both granular layers and melanocytes in the field of view. The boundary of granular layers is divided into an inner boundary （near the dermal layer portion） and an outer boundary （near the stratum corneum portion）， and the outer boundary is calculated for superficial spreading depth. For the malignant melanocytes in the samples， the data of the nearer melanocytes are removed with reference to the slope of the outer boundary of granular layer， and the farthest data are retained.
Let the set of outer boundary curve points of the granular layer be D（Φi（x，y）|i∈1，2，···，p）， and p is the number of pixels of the outer boundary curve. Let the set of malignant melanocyte boundary points be M（Γi（x，y）|j∈1，2，···，q）， q is the number of pixels of malignant melanocytes border. The procedures for calculating superficial spreading depth of skin melanoma are：
1） Since the distance between adjacent pixels is small， in order to reduce the amount of calculation， the curves D and M are resampled. Let the pixel points of sampling distance be d， and the symbol ∟ indicates rounding down. The boundary sets of granular layer and malignant melanocytes are respectively expressed as：
| ， | （12） |
| . | （13） |
2） Let Ek be a set of coordinate pairs， and the
| , | （14） |
| , | （15） |
where ||•|| represents the Euclidean distance between two points， and the coordinate pair indicates the point at which the malignant melanocytes are furthest from granular layer.
3） To improve the accuracy of the tumor superficial spreading depth calculation results， the maximum of the neighborhood is solved by the maximum distance of the previous step. Let Deep be the depth of melanoma， then：
| . | （16） |
We operated the microscopic hyperspectral imaging system to capture images of cutaneous melanoma samples at different magnifications. A magnified 100x image contained epidermal layer is displayed in
Fig.5 （a） 100x microscopic image of melanoma sample, （c） 200x image, （b） and （d） Single-band images at 810 nm.
图5 （a）（c） 放大100倍与200倍的显微镜彩色图像, （b）（d） 810 nm的单波段图像
After spectral calibration of hyperspectral image， the spectral differences of various pathological tissue components can be relatively truly reflected.
Fig.6 （a） and （b） Single-band images after preprocessing. （c） and （d） Spectra at the same position before and after processing
图6 （a）、（b）预处理后的单波段图像, （c）、（d）预处理前后相同位置的光谱曲线
The skin melanoma samples used in the experiment are thick in stratum corneum and have no distinct boundary with granular layer. It is almost difficult to identify and segment the granular layer based on common pathological images. Combining the spectral information and spatial information of hyperspectral images， we used the KMNF algorithm to focus the spectral features of image to the first few wavebands and reduce the image noise. The segmentation process of skin granular layer is demonstrated in
Fig.7 （a） The color microscope image, （b） the single-wavelength hyperspectral image, （c） the KMNF-based result at sixth waveband, （d） the morphological filtering result of skin granular layer, （e） the contour extraction based on level set segmentation, and （f） the finished segmentation result.
图7 （a）彩色显微图像, （b）单波段图像, （c）KMNF结果, （d）形态滤波结果, （e）轮廓提取结果, （f）颗粒层分割结果
In this part， three methods for feature extraction of hyperspectral images are studied， and some samples are employed for experiments.
Fig.8 Comparison of granular layer segmentation results of different methods
图8 不同方法颗粒层分割结果比较
Skin melanoma is derived from the cancerization of melanocytes. Therefore， malignant melanocytes are the key targets in the diagnosis of cutaneous melanoma. Under the microscopic field of H&E staining samples （
Fig.9 （a） A microscope image of malignant melanocytes, （b） the single band image at 810 nm, （c） the SVM segmentation result, and （d） the CSS-LSSVM segmentation result.
图9 （a）黑色素细胞彩色图像, （b）单波段图像, （c）SVM分割结果, （d）CSS-LSSVM分割结果
The results of quantitative analysis of four samples are listed in
The CSS-LSSVM algorithm has higher segmentation accuracy than the SVM for malignant melanocytes in the samples and can reach more than 85%. Referring to
Surgical treatment is one of the most effective and preferred treatment methods for cutaneous melanoma. It is to prevent the spread of the tumor by resection of the primary lesion. Whether the resection of the primary lesion is complete is also a direct factor affecting the prognosis. The most important thing in surgical treatment is to determine the boundaries of the tumor. We adopted the CSS-LSSVM algorithm to segment the malignant melanocytes in 100-fold amplified samples， and extracted the skin granular layer structure based on the KMNF algorithm， and retained the outer boundary. During the measurement， the calculated superficial spreading depth denotes the farthest distance between the malignant melanocytes and the granular layer. The yellow arrows in
Fig.10 Measurement results of cutaneous melanoma superficial spreading depth.
图10 皮肤黑色素瘤浅表扩散深度.
Since the superficial spreading depth is calculated based on pixels， in order to obtain the true physical value corresponding to the pixel size of the image， we conducted the following experiments： at different microscope magnifications， an objective lens calibration ruler was moved to count the distance of pixels in the image. The results are shown in
To evaluate the depth measurement， we compared the melanoma sample with normal skin tissue， as shown in the
The results show the distribution and spread of malignant melanocytes of superficial spreading melanoma. If patients are not treated in time， tumor deterioration and metastasis are likely to occur. we expect to provide a new technique to assist pathological diagnosis. The method proposed in this article is also applicable to advanced skin melanoma samples. However， due to the complexity of the melanoma samples， the morphological and spectral similarity of individual lymphocytes and malignant melanocytes in the melanoma sample may affect the accuracy of identification.
In view of the problem that the traditional pathological diagnosis of cutaneous melanoma cannot quantitatively analyze the tumor tissue， this paper applied microscopic hyperspectral imaging technology to identify and analyze melanoma superficial spreading depth. First， we segmented skin granular layer. Three multi-scale morphological filter operators were designed to extract the main morphological features， and the level set segmentation algorithm was used to obtain the contour of granular layer. By comparing PCA， MNF， and KMNF， the experimental results show that the KMNF-based algorithm has a segmentation accuracy of more than 80%. Second， we explored the segmentation of the malignant melanocytes. After adding the characteristic spectrum supervision of malignant melanocytes， the segmentation accuracy of the LSSVM is greater than 85%， which is better than the SVM algorithm. Finally， on the basis of the above researches， the superficial spreading depth of melanoma was quantitatively calculated. We consider that there is room for improvement in this study. Using microscopic hyperspectral imaging technology to identify and analyze superficial spreading melanoma samples is still a preliminary test， and the experimental results indicate the feasibility of this method. We have found that the thickness of granular layer in different samples is not exactly the same， and the adaptive ability of the filter operator in the experiment is insufficient. Further， we intend to verify on a large number of complex samples， so as to achieve the flexibility of MHSI technology for pathological diagnosis. Improving the accuracy of segmentation and measurement is also the direction of our efforts to developing method.
References
Ferlay J， Soerjomataram I， Dikshit R， et al. Cancer incidence and mortality worldwide： Sources， methods and major patterns in GLOBOCAN 2012 ［J］. International Journal of Cancer， 2015， 136（5）： E359-E386. [百度学术]
Erdmann F， Lortet-Tieulent J， Schuz J， et al. International trends in the incidence of malignant melanoma 1953-2008uare recent generations at higher or lower risk？ ［J］. International Journal of Cancer， 2013， 132（2）： 385-400. [百度学术]
Gray-Schopfer V， Wellbrock C， Marais R. Melanoma biology and new targeted therapy ［J］. Nature， 2007， 445（7130）： 851-857. [百度学术]
Eggermont A M M， Chiarion-Sileni V， Grob J J， et al. Prolonged Survival in Stage III Melanoma with Ipilimumab Adjuvant Therapy ［J］. New England Journal of Medicine， 2016， 375（19）： 1845-1855. [百度学术]
Garbe C， Peris K， Hauschild A， et al. Diagnosis and treatment of melanoma. European consensus-based interdisciplinary guideline - Update 2016 ［J］. European Journal of Cancer， 2016， 63： 201-217. [百度学术]
Thomas L， Puig S. Dermoscopy， Digital Dermoscopy and Other Diagnostic Tools in the Early Detection of Melanoma and Follow-up of High-risk Skin Cancer Patients ［J］. Acta Dermato-Venereologica， 2017， 97： 14-21. [百度学术]
Yu L Q， Chen H， Dou Q， et al. Automated Melanoma Recognition in Dermoscopy Images via Very Deep Residual Networks ［J］. Ieee Transactions on Medical Imaging， 2017， 36（4）： 994-1004. [百度学术]
Wang T， Hou G， Zhang N， et al. Melanoma Computer-aided Diagnosis Algorithms Based on Laser Scanning Confocal Microscope Images ［J］. Chinese Journal of Medical Imaging， 2013， 21（2）： 130-133. [百度学术]
Rastrelli M， Tropea S， Rossi C R， et al. Melanoma： Epidemiology， Risk Factors， Pathogenesis， Diagnosis and Classification ［J］. In Vivo， 2014， 28（6）： 1005-1011. [百度学术]
Silveira M， Nascimento J C， Marques J S， et al. Comparison of Segmentation Methods for Melanoma Diagnosis in Dermoscopy Images ［J］. Ieee Journal of Selected Topics in Signal Processing， 2009， 3（1）： 35-45. [百度学术]
Binder M， Schwarz M， Winkler A， et al. Epiluminescence microscopy： A useful tool for the diagnosis of pigmented skin lesions for formally trained dermatologists ［J］. Archives of Dermatology， 1995， 131（3）： 286-291. [百度学术]
Garbe C， Leiter U. Melanoma epidemiology and trends ［J］. Clinics in Dermatology， 2009， 27（1）： 3-9. [百度学术]
MacKie R M， Dunitz M. Skin cancer ： an illustrated guide to the aetiology， clinical features， pathology and management of benign and malignant cutaneous tumours，1989. [百度学术]
Mokhtari M， Rezaeian M， Gharibzadeh S， et al. Computer aided measurement of melanoma depth of invasion in microscopic images ［J］. Micron， 2014， 61： 40-48. [百度学术]
Noroozi N， Zakerolhosseini A. Computerized measurement of melanocytic tumor depth in skin histopathological images ［J］. Micron， 2015， 77： 44-56. [百度学术]
Hongming X， Huiquan W， Berendt R， et al. Computerized measurement of melanoma depth of invasion in skin biopsy images ［J］. 2017 IEEE EMBS International Conference on Biomedical & Health Informatics （BHI）， 2017： 17-20. [百度学术]
Dinehart M S， Dinehart S M， Sukpraprut-Braaten S， et al. Immunohistochemistry utilization in the diagnosis of melanoma ［J］. Journal of cutaneous pathology， 2020. [百度学术]
Kamyab-Hesary K， Ghanadan A， Balighi K， et al. Immunohistochemical Staining in the Assessment of Melanoma Tumor Thickness ［J］. Pathology oncology research ： POR， 2019. [百度学术]
Chopra A， Sharma R， Rao U N M. Pathology of Melanoma ［J］. Surgical Clinics of North America， 2020， 100（1）： 43 [百度学术]
-+. [百度学术]
Stamatas G N， Balas C J， Kollias N. Hyperspectral image acquisition and analysis of skin ［J］. Proceedings of the SPIE - The International Society for Optical Engineering， 2003， 4959： 77-82. [百度学术]
Nagaoka T， Nakamura A， Okutani H， et al. A possible melanoma discrimination index based on hyperspectral data： a pilot study ［J］. Skin Research and Technology， 2012， 18（3）： 301-310. [百度学术]
Zheludev V， Polonen I， Neittaanmaki-Perttu N， et al. Delineation of malignant skin tumors by hyperspectral imaging using diffusion maps dimensionality reduction ［J］. Biomedical Signal Processing and Control， 2015， 16： 48-60. [百度学术]
Pardo A， Gutierrez-Gutierrez J A， Lihacova I， et al. On the spectral signature of melanoma： a non-parametric classification framework for cancer detection in hyperspectral imaging of melanocytic lesions ［J］. Biomedical Optics Express， 2018， 9（12）： 6283-6301. [百度学术]
Hosking A M， Coakley B J， Chang D， et al. Hyperspectral imaging in automated digital dermoscopy screening for melanoma ［J］. Lasers in Surgery and Medicine， 2019， 51（3）： 214-222. [百度学术]
Panasyuk S V， Yang S， Faller D V， et al. Medical hyperspectral imaging to facilitate residual tumor identification during surgery ［J］. Cancer Biology & Therapy， 2007， 6（3）： 439-446. [百度学术]
Wang J S， Li Q L. Quantitative analysis of liver tumors at different stages using microscopic hyperspectral imaging technology ［J］. Journal of Biomedical Optics， 2018， 23（10）. [百度学术]
Zonios G， Perelman L T， Backman V M， et al. Diffuse reflectance spectroscopy of human adenomatous colon polyps in vivo ［J］. Applied Optics， 1999， 38（31）： 6628-6637. [百度学术]
He S， Chen H， Jiao F. Improved Preparation Method for Animal Histopathological Section ［J］. Progress in Veterinary Medicine， 2011， 32（11）： 130-132. [百度学术]
Li Q L， Xu D R， He X F， et al. AOTF based molecular hyperspectral imaging system and its applications on nerve morphometry ［J］. Applied Optics， 2013， 52（17）： 3891-3901. [百度学术]
Li Q L， Zhou M， Liu H Y， et al. Red Blood Cell Count Automation Using Microscopic Hyperspectral Imaging Technology ［J］. Applied Spectroscopy， 2015， 69（12）： 1372-1380. [百度学术]
Li Q L， Wang Y T， Liu H Y， et al. A combined spatial-spectral method for automated white blood cells segmentation ［J］. Optics and Laser Technology， 2013， 54： 225-231. [百度学术]
Huebschman M L， Schultz R A， Garner H R. Characteristics and capabilities of the Hyperspectral Imaging Microscope ［J］. Ieee Engineering in Medicine and Biology Magazine， 2002， 21（4）： 104-117. [百度学术]
Green A A， Berman M， Switzer P， et al. A transformation for ordering multispectral data in terms of image quality with implications for noise removal ［J］. IEEE Transactions on Geoscience and Remote Sensing， 1988， 26（1）： 65-74. [百度学术]
Randeberg L L， Larsen E L， Svaasand L O. Characterization of vascular structures and skin bruises using hyperspectral imaging， image analysis and diffusion theory ［J］. Journal of Biophotonics， 2010， 3（1-2）： 53-65. [百度学术]
Nielsen A A. Kernel Maximum Autocorrelation Factor and Minimum Noise Fraction Transformations ［J］. Ieee Transactions on Image Processing， 2011， 20（3）： 612-624. [百度学术]
Yang X， Gao X B， Tao D C， et al. An Efficient MRF Embedded Level Set Method for Image Segmentation ［J］. Ieee Transactions on Image Processing， 2015， 24（1）： 9-21. [百度学术]
Wang B， Li J， Gao X. An Edge-and Region-Based Level Set Method with Shape Priors for Image Segmentation ［J］. Chinese Journal of Computers， 2012， 35（5）： 1067-1072. [百度学术]
Chan T F， Vese L A. Active contours without edges ［J］. Ieee Transactions on Image Processing， 2001， 10（2）： 266-277. [百度学术]
de Kruif B J， de Vries T J A. Pruning error minimization in least squares support vector machines ［J］. Ieee Transactions on Neural Networks， 2003， 14（3）： 696-702. [百度学术]
Bui D T， Tuan T A， Hoang N D， et al. Spatial prediction of rainfall-induced landslides for the Lao Cai area （Vietnam） using a hybrid intelligent approach of least squares support vector machines inference model and artificial bee colony optimization ［J］. Landslides， 2017， 14（2）： 447-458. [百度学术]
Dong C-x， Rao X， Yang S-q， et al. Research on estimating the generalization performance of RBF-SVM ［J］. Journal of Xidian University， 2004， 31（4）： 557-561. [百度学术]
Wei X， Yu X， Fu Q， et al. Spectral Matching Classification Approach and Experiment Combined with Spectral Angle Cosine and Spectral Correlation Coefficient ［J］. Geography and Geo-information Science， 2016， 32（3）： 29-33. [百度学术]
Elmore J G， Barnhill R L， Elder D E， et al. Pathologists' diagnosis of invasive melanoma and melanocytic proliferations： observer accuracy and reproducibility study ［J］. Bmj-British Medical Journal， 2017， 357： 11. [百度学术]