More Web Proxy on the site http://driver.im/

research-article

Pixel-wise warping for deep image stitching

AUTHORs:

Hyeokjun Kweon,

Hyeonseong Kim,

Wooseong Jeong,

Kuk-Jin YoonAuthors Info & Claims

AAAI'23/IAAI'23/EAAI'23: Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence

Article No.: 133, Pages 1196 - 1204

https://doi.org/10.1609/aaai.v37i1.25202

Published: 07 February 2023 Publication History

Abstract

Existing image stitching approaches based on global or local homography estimation are not free from the parallax problem and suffer from undesired artifacts. In this paper, instead of relying on the homography-based warp, we propose a novel deep image stitching framework exploiting the pixel-wise warp field to handle the large-parallax problem. The proposed deep image stitching framework consists of a Pixel-wise Warping Module (PWM) and a Stitched Image Generating Module (SIGMo). For PWM, we obtain pixel-wise warp in a similar manner as estimating an optical flow (OF). In the stitching scenario, the input images usually include non-overlap (NOV) regions of which warp cannot be directly estimated, unlike the overlap (OV) regions. To help the PWM predict a reasonable warp on the NOV region, we impose two geometrical constraints: an epipolar loss and a line-preservation loss. With the obtained warp field, we relocate the pixels of the target image using forward warping. Finally, the SIGMo is trained by the proposed multi-branch training framework to generate a stitched image from a reference image and a warped target image. For training and evaluating the proposed framework, we build and publish a novel dataset including image pairs with corresponding pixel-wise ground truth warp and stitched result images. We show that the results of the proposed framework are qualitatively and quantitatively superior to those of the conventional methods.

References

[1]

Armeni, I.; Sener, O.; Zamir, A. R.; Jiang, H.; Brilakis, I.; Fischer, M.; and Savarese, S. 2016. 3d semantic parsing of large-scale indoor spaces. In Proceedings of the IEEE conference on computer vision and pattern recognition, 1534-1543.

[2]

Chaohao Xie, C. L. M.-M. C.-W. Z. X. L. S. W. E. D., Shaohui Liu. 2019. Image inpainting with learnable bidirectional attention maps. IEEE/CVF International Conference on Computer Vision.

[3]

Dosovitskiy, A.; Ros, G.; Codevilla, F.; Lopez, A.; and Koltun, V. 2017. CARLA: An open urban driving simulator. In Conference on robot learning, 1-16. PMLR.

[4]

Gao, J.; Kim, S. J.; and Brown, M. S. 2011. Constructing image panoramas using dual-homography warping. In CVPR 2011, 49-56. IEEE.

[5]

Guilin Liu, K. J. S. T.-C. W. A. T. B. C., Fitsum A. Reda. 2018. Image inpainting for irregular holes using partial convolutions. European conference on computer vision.

[6]

Guilin Liu, K. J. S. T.-C. W. A. T. B. C., Fitsum A. Reda. 2020. Rethinking image inpainting via a mutual encoderdecoder with feature equalizations. European conference on computer vision.

[7]

Hoang, V.-D.; Tran, D.-P.; Nhu, N. G.; Pham, V.-H.; et al. 2020. Deep feature extraction for panoramic image stitching. In Asian Conference on Intelligent Information and Database Systems, 141-151. Springer.

[8]

Isola, P.; Zhu, J.-Y.; Zhou, T.; and Efros, A. A. 2017. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, 1125-1134.

[9]

Jia, Q.; Li, Z.; Fan, X.; Zhao, H.; Teng, S.; Ye, X.; and Latecki, L. J. 2021. Leveraging Line-Point Consistence To Preserve Structures for Wide Parallax Image Stitching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 12186-12195.

[10]

Johnson, J.; Alahi, A.; and Fei-Fei, L. 2016. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision, 694-711. Springer.

[11]

Krishna, V.; Joshi, A.; Vrabac, D.; Bulterys, P.; Yang, E.; Fernandez-Pol, S.; Ng, A. Y.; and Rajpurkar, P. 2021. GloFlow: Whole Slide Image Stitching from Video Using Optical Flow and Global Image Alignment. In International Conference on Medical Image Computing and ComputerAssisted Intervention, 519-528. Springer.

[12]

Krizhevsky, A.; Sutskever, I.; and Hinton, G. E. 2012. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25.

[13]

Lee, K.-Y.; and Sim, J.-Y. 2020. Warping residual based image stitching for large parallax. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8198-8206.

[14]

Levin, A.; Zomet, A.; Peleg, S.; and Weiss, Y. 2004. Seamless image stitching in the gradient domain. In European conference on computer vision, 377-389. Springer.

[15]

Li, H.; Yu, H.; Yang, W.; Yu, L.; and Scherer, S. 2020. ULSD: Unified Line Segment Detection across Pinhole, Fisheye, and Spherical Cameras. arXiv:2011.03174.

[16]

Li, J.; Zhao, Y.; Ye, W.; Yu, K.; and Ge, S. 2019. Attentive Deep Stitching and Quality Assessment for 360° Omnidirectional Images. IEEE Journal of Selected Topics in Signal Processing, 14(1): 209-221.

[17]

Li, L.; Yao, J.; Xie, R.; Xia, M.; and Zhang, W. 2017. A unified framework for street-view panorama stitching. Sensors, 17(1): 1.

[18]

Lin, C.-C.; Pankanti, S. U.; Natesan Ramamurthy, K.; and Aravkin, A. Y. 2015. Adaptive as-natural-as-possible image stitching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1155-1163.

[19]

Lin, K.; Jiang, N.; Cheong, L.-F.; Do, M.; and Lu, J. 2016. Seagull: Seam-guided local alignment for parallax-tolerant image stitching. In European conference on computer vision, 370-385. Springer.

[20]

Lin, W.-Y.; Liu, S.; Matsushita, Y.; Ng, T.-T.; and Cheong, L.-F. 2011. Smoothly varying affine stitching. In CVPR 2011, 345-352. IEEE.

[21]

Loshchilov, I.; and Hutter, F. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101.

[22]

Meng, M.; and Liu, S. 2020. High-quality Panorama Stitching based on Asymmetric Bidirectional Optical Flow. In 2020 5th International Conference on Computational Intelligence and Applications (ICCIA), 118-122. IEEE.

[23]

Mittal, A.; Moorthy, A. K.; and Bovik, A. C. 2012. No-reference image quality assessment in the spatial domain. In IEEE Transactions on image processing, volume 21, 4695-4708. IEEE.

[24]

Nie, L.; Lin, C.; Liao, K.; Liu, M.; and Zhao, Y. 2020a. A view-free image stitching network based on global homography. Journal of Visual Communication and Image Representation, 73: 102950.

Digital Library

[25]

Nie, L.; Lin, C.; Liao, K.; Liu, S.; and Zhao, Y. 2021. Unsupervised Deep Image Stitching: Reconstructing Stitched Features to Images. IEEE Transactions on Image Processing, 30: 6184-6197.

[26]

Nie, L.; Lin, C.; Liao, K.; and Zhao, Y. 2020b. Learning edge-preserved image stitching from large-baseline deep homography. arXiv preprint arXiv:2012.06194.

[27]

Niklaus, S.; and Liu, F. 2020. Softmax splatting for video frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5437-5446.

[28]

Peleg, S.; Rousso, B.; Rav-Acha, A.; and Zomet, A. 2000. Mosaicing on adaptive manifolds. IEEE Transactions on pattern analysis and machine intelligence, 22(10): 1144-1154.

Digital Library

[29]

Shi, Z.; Li, H.; Cao, Q.; Ren, H.; and Fan, B. 2020. An image mosaic method based on convolutional neural network semantic features extraction. Journal of Signal Processing Systems, 92(4): 435-444.

[30]

Simonyan, K.; and Zisserman, A. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.

[31]

Teed, Z.; and Deng, J. 2020. Raft: Recurrent all-pairs field transforms for optical flow. In European conference on computer vision, 402-419. Springer.

[32]

Wang, Z.; Bovik, A. C.; Sheikh, H. R.; and Simoncelli, E. P. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4): 600-612.

[33]

Zaragoza, J.; Chin, T.-J.; Brown, M. S.; and Suter, D. 2013. As-projective-as-possible image stitching with moving DLT. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2339-2346.

[34]

Zhang, F.; and Liu, F. 2014. Parallax-tolerant image stitching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3262-3269.

[35]

Zhang, R.; Isola, P.; Efros, A. A.; Shechtman, E.; and Wang, O. 2018. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, 586-595.

[36]

Zheng, J.; Wang, Y.; Wang, H.; Li, B.; and Hu, H.-M. 2019. A novel projective-consistent plane based image stitching method. IEEE Transactions on Multimedia, 21(10): 2561-2575.

Digital Library

Cited By

Kim SHong SPark KCho HSohn KCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)OmniStitch: Depth-Aware Stitching Framework for Omnidirectional Vision with Multiple CamerasProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681208(10210-10219)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681208

Recommendations

Learning Pixel-wise Alignment for Unsupervised Image Stitching
MM '23: Proceedings of the 31st ACM International Conference on Multimedia

Image stitching aims to align a pair of images in the same view. Generating precise alignment with natural structures is challenging for image stitching, as there is no wider field-of-view image as a reference, especially in non-coplanar practical ...
Deep Learning on Image Stitching With Multi-viewpoint Images: A Survey
Abstract
Multi-viewpoint image stitching aims to stitch images taken from different viewpoints into pictures with a broader field of view. The stitched images are subject to artifacts, geometric distortion, and blur distortion due to the mismatch of ...
Feature Based Stitching of a Clear/Blurred Image Pair
CMSP '11: Proceedings of the 2011 International Conference on Multimedia and Signal Processing - Volume 01

Clear image stitching becomes mature now, but how to stitch blurred images with clear ones is still tough, mainly because of the ill-posed image deblurring. Existing deblurring methods assume spatial-invarint blur kernel, pixelwise constant blur kernel ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Guide Proceedings

AAAI'23/IAAI'23/EAAI'23: Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence

February 2023

16496 pages

ISBN:978-1-57735-880-0

Copyright © 2023 Association for the Advancement of Artificial Intelligence.

Sponsors

Association for the Advancement of Artificial Intelligence

Publisher

AAAI Press

Publication History

Published: 07 February 2023

Qualifiers

Research-article
Research
Refereed limited

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 21 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kim SHong SPark KCho HSohn KCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)OmniStitch: Depth-Aware Stitching Framework for Omnidirectional Vision with Multiple CamerasProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681208(10210-10219)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681208

View Options

View options

Media

Figures

Other

Tables

View Table of Contents