Face Hallucination with Weighted Nuclear Norm Constraint

Abstract

Abstract:Face hallucination via patch-pairs leaning based methods has been wildly used in the past several years. Some position-patch based face hallucination methods have been proposed to improve the representation power of image patch and obtain the optimal regressive weighted vector. The rationale behind the position-patch based face hallucination is the fact that human face is always highly structured and consequently positioned and it plays an increasingly important role in the reconstruction. However, in the existing position-patch based methods, the probe image patch is usually represented as a linear combination of the corresponding patches of some training images, and the reconstruction residual is usually measured using the vector norm such as 1-norm and 2-norm. Since the vector norms neglect two-dimensional structures inside the residual, the final reconstruction performance is not very satisfactory. To cope with this problem, we present a weighted nuclear-norm constrained sparse coding (WNCSC) model for position-patch based face hallucination. In addition, an efficient algorithm for the WNCSC is developed using the alternating direction method of multipliers (ADMM) and the method of augmented Lagrange multipliers (ALM). The advantages of the proposed model are twofold: in order to fully make use of low-rank structure information of the reconstruction residual, the weighted nuclear norm is applied to measure the residual matrix, which is able to alleviate the bias between input patches and training data, and it is more robust than the Euclidean distance (2-norm); the more flexible selection method for rank components can determine the optimal combination weights and adaptively choose the relevant and nearest hallucinated neighbors. Finally, experimental results prove that the proposed method outperforms the related state-of-the-art methods in both quantitative and visual comparisons.

Key words:face hallucination| weighted nuclear norm| position-patch

摘要：Face hallucination via patch-pairs leaning based methods has been wildly used in the past several years. Some position-patch based face hallucination methods have been proposed to improve the representation power of image patch and obtain the optimal regressive weighted vector. The rationale behind the position-patch based face hallucination is the fact that human face is always highly structured and consequently positioned and it plays an increasingly important role in the reconstruction. However, in the existing position-patch based methods, the probe image patch is usually represented as a linear combination of the corresponding patches of some training images, and the reconstruction residual is usually measured using the vector norm such as 1-norm and 2-norm. Since the vector norms neglect two-dimensional structures inside the residual, the final reconstruction performance is not very satisfactory. To cope with this problem, we present a weighted nuclear-norm constrained sparse coding (WNCSC) model for position-patch based face hallucination. In addition, an efficient algorithm for the WNCSC is developed using the alternating direction method of multipliers (ADMM) and the method of augmented Lagrange multipliers (ALM). The advantages of the proposed model are twofold: in order to fully make use of low-rank structure information of the reconstruction residual, the weighted nuclear norm is applied to measure the residual matrix, which is able to alleviate the bias between input patches and training data, and it is more robust than the Euclidean distance (2-norm); the more flexible selection method for rank components can determine the optimal combination weights and adaptively choose the relevant and nearest hallucinated neighbors. Finally, experimental results prove that the proposed method outperforms the related state-of-the-art methods in both quantitative and visual comparisons.

关键词:face hallucination| weighted nuclear norm| position-patch

CLC Number:

TP 319

TANG Songze (唐松泽), LI Heng (李恒), XIAO Liang (肖亮). Face Hallucination with Weighted Nuclear Norm Constraint[J]. Journal of Shanghai Jiao Tong University (Science), 2018, 23(5): 627-635.

References31

[1]	FREEMAN W T, PASZTOR E C, CARMICHAEL OT. Learning low-level vision [J]. International Journalof Computer Vision, 2000, 40(1): 25-47.
[2]	TANG S Z, XIAO L, LIU P F, et al. Coupled learningbased on singular-values-unique and hog for facehallucination [C]//Proceedings of IEEE InternationalConference on Acoustics, Speech and Signal Processing.Brisbane, Australia: IEEE, 2015: 1315-1319.
[3]	JIANG J J, HU R M, WANG Z Y, et al. Noise robustface hallucination via locality-constrained representation[J]. IEEE Transactions on Multimedia, 2014,16(5): 1268-1281.
[4]	BAKER S, KANADE T. Limits on super-resolutionand how to break them [J]. IEEE Transactions on PatternAnalysis and Machine Intelligence, 2002, 24(9):1167-1183.
[5]	WANG X Y, TANG X O. Hallucinating face by eigentransformation [J]. IEEE Transactions on Systems,Man, and Cybernetics, Part C, 2005, 35(3): 425-434.
[6]	HU Y, LAM K M, SHEN T Z, et al. A novel kernelbasedframework for facial-image hallucination [J]. Imageand Vision Computing, 2011, 29: 219-229.
[7]	HUANG H, HE H T, FAN X, et al. Super-resolution ofhuman face image using canonical correlation analysis[J]. Pattern Recognition, 2010, 43: 2532-2543.
[8]	AN L, BHANU B. Face image super-resolution using2D CCA [J]. Signal Processing, 2014, 103: 184-194.
[9]	TANG S Z, XIAO L, LIU P F, et al. Partial leastsquaresregression on common feature space for singleimage superresolution [J]. Journal of Electronic Imaging,2014, 23(5): 053006.
[10]	ROWEIS S T, SAUL L K. Nonlinear dimensionality reductionby locally linear embedding [J]. Science, 2000,290(5500): 2323-2326.
[11]	CHANG H, YEUNG D Y, XIONG Y M. Superresolutionthrough neighbor embedding [C]//Proceedingsof 17th IEEE Computer Society Conferenceon Computer Vision and Pattern Recognition. Washington,USA: IEEE, 2004: 1275-1282.
[12]	ZHU Q D, SUN L, CAI C T. Non-local neighbor embeddingfor image super-resolution through FoE features[J]. Neurocomputing, 2014, 141: 211-222.
[13]	CHEN X X, QI C. Low-rank neighbor embedding forsingle image super-resolution [J]. IEEE Signal ProcessingLetters, 2014, 21(1): 79-82.
[14]	JIANG J J, HU R M, WANG Z Y, et al. Facial imagehallucination through coupled-layer neighbor embedding[J]. IEEE Transactions on Circuits and Systemsfor Video Technology, 2015, 26(9): 1674-1684.
[15]	YANG J C, WRIGHT J, HUANG T S, et al. Imagesuper-resolution via sparse representation [J]. IEEETransactions on Image Processing, 2010, 19(11): 2861-2873.
[16]	MA X, ZHANG J P, QI C. Hallucinating face byposition-patch [J]. Pattern Recognition, 2010, 43:2224-2236.
[17]	JUNG C, JIAO L C, LIU B, et al. Position-patch basedface hallucination using convex optimization [J]. IEEESignal Processing Letters, 2011, 18(6): 367-370.
[18]	JIANG J J, CHEN C, MA J Y, et al. SRLSP: A faceimage super-resolution algorithm using smooth regressionwith local structure prior [J]. IEEE Transactionson Multimedia, 2016, 19(1): 27-40.
[19]	DONG C, LOY C C, HE K M, et al. Image superresolutionusing deep convolutional networks [J]. IEEETransactions on Pattern Analysis and Machine Intelligence,2016, 38(2): 295-307.
[20]	GAO G W, YANG J, LAI Z H, et al. Nuclear normregularized coding with local position-patch and nonlocalsimilarity for face hallucination [J]. Digital SignalProcessing, 2017, 64: 107-120.
[21]	CAI J F, CAND`ES E J, SHEN Z W. A singularvalue thresholding algorithm for matrix completion [J].SIAM Journal on Optimization, 2010, 20(4): 1956-1982.
[22]	GU S H, ZHANG L, ZUO W M, et al. Weightednuclear norm minimization with application to imagedenoising [C]//Proceedings of IEEE Conference onComputer Vision and Pattern Recognition. Columbus,USA: IEEE, 2014: 2862-2869.
[23]	YANG J, LUO L, QIAN J J, et al. Nuclear norm basedmatrix regression with applications to face recognitionwith occlusion and illumination changes [J]. IEEETransactions on Pattern Analysis and Machine Intelligence,2016, 39(1): 156-171.
[24]	LIN Z C, CHEN M M, MA Y. The augmentedLagrange multiplier method for exact recovery ofcorrupted low-rank matrices [EB/OL]. (2010-09-26)[2017-12-17]. https://arxiv.org/abs/1009.5055.
[25]	HANSSON A, LIU Z, VANDENBERGHE L. Subspacesystem identification via weighted nuclear norm optimization[C]//Proceedings of 51st IEEE Conference onDecision and Control. Hawaii, USA: IEEE, 2012: 3439-3444.
[26]	BECK A, TEBOULLE M. A fast iterative shrinkagethresholdingalgorithm for linear inverse problems [J].SIAM Journal on Imaging Sciences, 2009, 2(1): 183-202.
[27]	GU S H, XIE Q, MENG D Y, et al. Weighted nuclearnorm minimization and its applications to low levelvision [J]. International Journal of Computer Vision,2017, 121(2): 183-208.
[28]	THOMAZ C E, GIRALDI G A. A new ranking methodfor principal components analysis and its applicationto face image analysis [J]. Image and Vision Computing,2010, 28: 902-913.
[29]	WANG Z, BOVIK A C, SHEIKH H R, et al. Imagequality assessment: From error visibility to structuralsimilarity [J]. IEEE Transactions on Image Processing,2004, 13(4): 600-612.
[30]	QU SM, HU R M, CHEN S H, et al. Robust face superresolutionvia position-patch neighborhood preserving[C]//Proceedings of 15th IEEE International Conferenceon Multimedia and Expo Workshops (ICMEW).Chengdu, China: IEEE, 2014: 1-5.
[31]	XIE Y, QU Y Y, TAO D C, et al. Hyperspectral imagerestoration via iteratively regularized weighted Schattenp-norm minimization [J]. IEEE Transactions onGeoscience and Remote Sensing, 2016, 54(8): 4642-4659.