More Web Proxy on the site http://driver.im/

research-article

PatchNet: a patch-based image representation for interactive library-driven image editing

Authors:

Fang-Lue Zhang,

Ralph R. Martin,

Jue WangAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 32, Issue 6

Article No.: 196, Pages 1 - 12

https://doi.org/10.1145/2508363.2508381

Published: 01 November 2013 Publication History

Abstract

We introduce PatchNets, a compact, hierarchical representation describing structural and appearance characteristics of image regions, for use in image editing. In a PatchNet, an image region with coherent appearance is summarized by a graph node, associated with a single representative patch, while geometric relationships between different regions are encoded by labelled graph edges giving contextual information. The hierarchical structure of a PatchNet allows a coarse-to-fine description of the image. We show how this PatchNet representation can be used as a basis for interactive, library-driven, image editing. The user draws rough sketches to quickly specify editing constraints for the target image. The system then automatically queries an image library to find semantically-compatible candidate regions to meet the editing goal. Contextual image matching is performed using the PatchNet representation, allowing suitable regions to be found and applied in a few seconds, even from a library containing thousands of images.

References

[1]

Arbelaez, P., Maire, M., Fowlkes, C., and Malik, J. 2011. Contour detection and hierarchical image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 33, 5 (May).

Digital Library

[2]

Baeza-Yates, R., and Valiente, G. 2000. An image similarity measure based on graph matching. In Proc. International Symposium on String Processing and Information Retrieval, IEEE, 28--38.

Digital Library

[3]

Bagon, B., and Irani, M. 2008. What is a good image segment? a unified approach to segment extraction. In Proc. ECCV.

Digital Library

[4]

Barnes, C., Shechtman, E., Finkelstein, A., and Goldman, D. B. 2009. Patchmatch: a randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28, 3 (July), 24:1--24:11.

Digital Library

[5]

Barnes, C., Shechtman, E., Goldman, D. B., and Finkelstein, A. 2010. The generalized PatchMatch correspondence algorithm. In Proc. ECCV, 29--43.

Digital Library

[6]

Belongie, S., Malik, J., and Puzicha, J. 2002. Shape matching and object recognition using shape contexts. IEEE Trans. Pattern Anal. Mach. Intell. 24, 4, 509--522.

Digital Library

[7]

Besse, F., Rother, C., Fitzgibbon, A., and Kautz, J. 2012. Pmbp: Patchmatch belief propagation for correspondence field estimation. In Proc. BMVC.

[8]

Bosch, A., Zisserman, A., and Munoz, X. 2007. Representing shape with a spatial pyramid kernel. In Proc. CIVR.

Digital Library

[9]

Brox, T., Bregler, C., and Malik, J. 2009. Large displacement optical flow. In Proc. CVPR, 41--48.

[10]

Buades, A., and Coll, B. 2005. A non-local algorithm for image denoising. In Proc. CVPR, 60--65.

Digital Library

[11]

Chen, T., Cheng, M.-M., Tan, P., Shamir, A., and Hu, S.-M. 2009. Sketch2photo: internet image montage. ACM Trans. Graph. 28, 5 (Dec.), 124:1--124:10.

Digital Library

[12]

Chevalier, F., Domenger, J.-P., Benois-Pineau, J., and Delest, M. 2007. Retrieval of objects in video by similarity based on graph matching. Pattern Recognition Letters 28, 8, 939--949.

Digital Library

[13]

Comaniciu, D., Meer, P., and Member, S. 2002. Mean shift: A robust approach toward feature space analysis. IEEE Trans. Pattern Anal. Mach. Intell. 24, 603--619.

Digital Library

[14]

Darabi, S., Shechtman, E., Barnes, C., Goldman, D. B., and Sen, P. 2012. Image melding: combining inconsistent images using patch-based synthesis. ACM Trans. Graph. 31, 4 (July), 82:1--82:10.

Digital Library

[15]

Efros, A. A., and Freeman, W. T. 2001. Image quilting for texture synthesis and transfer. In Proc. SIGGRAPH, ACM, 341--346.

Digital Library

[16]

Eitz, M., Richter, R., Hildebrand, K., Boubekeur, T., and Alexa, M. 2011. Photosketcher: interactive sketch-based image synthesis. IEEE Computer Graphics and Applications.

Digital Library

[17]

Fei-Fei, L., and Perona, P. 2005. A bayesian hierarchical model for learning natural scene categories. In Proc. CVPR, 524--531.

Digital Library

[18]

Fisher, M., and Hanrahan, P. 2010. Context-based search for 3d models. ACM Trans. Graph. 29, 6, 182:1--182:10.

Digital Library

[19]

Freedman, G., and Fattal, R. 2011. Image and video upscaling from local self-examples. ACM Trans. Graph. 30, 2 (Apr.), 12:1--12:11.

Digital Library

[20]

Galun, M., Sharon, E., Basri, R., and Brandt, A. 2003. Texture segmentation by multiscale aggregation of filter responses and shape elements. In Proc. ICCV.

Digital Library

[21]

Gould, S., and Zhang, Y. 2012. Patchmatchgraph: building a graph of dense patch correspondences for label transfer. In Proc. ECCV, Springer, 439--452.

Digital Library

[22]

HaCohen, Y., Shechtman, E., Goldman, D. B., and Lischinski, D. 2011. Non-rigid dense correspondence with applications for image enhancement. ACM Trans. Graph. 30, 4 (July), 70:1--70:10.

Digital Library

[23]

Hays, J., and Efros, A. A. 2007. Scene completion using millions of photographs. ACM Trans. Graph. 26, 3 (July), 4.

Digital Library

[24]

Hlaoui, A., and Wang, S. 2002. A new algorithm for graph matching with application to content-based image retrieval. In Structural, Syntactic, and Statistical Pattern Recognition. Springer, 291--300.

Digital Library

[25]

Hu, R., Barnard, M., and Collomosse, J. 2010. Gradient field descriptor for sketch based retrieval and localization. In Proc. ICIP, IEEE, 1025--1028.

[26]

Hu, S.-M., Chen, T., Xu, K., Cheng, M.-M., and Martin, R. R. 2013. Internet visual media processing: a survey with graphics and vision applications. The Visual Computer 29, 5, 393--405.

Digital Library

[27]

Jain, A., Thormählen, T., Ritschel, T., and Seidel, H.-P. 2012. Material memex: automatic material suggestions for 3d objects. ACM Trans. Graph. 31, 6, 143:1--143:8.

Digital Library

[28]

Johnson, M., Brostow, G. J., Shotton, J., Arandjelovic, O., Kwatra, V., and Cipolla, R. 2006. Semantic photo synthesis. Computer Graphics Forum 25, 3, 407--413.

[29]

Johnson, M. K., Dale, K., Avidan, S., Pfister, H., Freeman, W. T., and Matusik, W. 2011. Cg2real: Improving the realism of computer generated images using a large collection of photographs. IEEE Transactions on Visualization and Computer Graphics 17, 9, 1273--1285.

Digital Library

[30]

Jojic, N. 2003. Epitomic analysis of appearance and shape. In Proc. ICCV.

Digital Library

[31]

Kannan, A., Winn, J., and Rother, C. 2006. Clustering appearance and shape by learning jigsaws. In Proc. NIPS.

[32]

Kopf, J., Kienzle, W., Drucker, S., and Kang, S. B. 2012. Quality prediction for image completion. ACM Trans. Graph. 31, 6 (Nov.), 131:1--131:8.

Digital Library

[33]

Kuettel, D., Guillaumin, M., and Ferrari, V. 2012. Figure-ground segmentation by transferring window masks. In Proc. CVPR.

Digital Library

[34]

Kwatra, V., Schödl, A., Essa, I., Turk, G., and Bobick, A. 2003. Graphcut textures: image and video synthesis using graph cuts. ACM Trans. Graph. 22, 3, 277--286.

Digital Library

[35]

Lalonde, J.-F., Hoiem, D., Efros, A. A., Rother, C., Winn, J., and Criminisi, A. 2007. Photo clip art. ACM Trans. Graph. 26, 3, 3.

Digital Library

[36]

Lasram, A., and Lefebvre, S. 2012. Parallel patch-based texture synthesis. In Proceedings of the Fourth ACM SIGGRAPH/Eurographics conference on High-Performance Graphics, 115--124.

Digital Library

[37]

Lee, Y. J., and Grauman, K. 2010. Object-graphs for context-aware category discovery. In Proc. CVPR, IEEE, 1--8.

[38]

Lee, Y. J., Zitnick, C. L., and Cohen, M. F. 2011. Shadowdraw: real-time user guidance for freehand drawing. ACM Trans. Graph. 30, 4 (July), 27:1--27:10.

Digital Library

[39]

Levin, A., Lischenski, D., and Weiss, Y. 2008. A closed-form solution to natural image matting. IEEE Trans. Pattern Anal. Mach. Intell. 30, 2, 228--242.

Digital Library

[40]

Lim, J. J., Arbeláez, P., Gu, C., and Malik, J. 2009. Context by region ancestry. In Proc. ICCV, IEEE, 1978--1985.

[41]

Liu, Y., and Yu, Y. 2011. Free appearance-editing with improved poisson image cloning. Journal of Computer Science and Technology 26, 6, 1011--1016.

Digital Library

[42]

Liu, Y., and Yu, Y. 2012. Interactive image segmentation based on level sets of probabilities. IEEE Transactions on Visualization and Computer Graphics 18, 2, 202--213.

Digital Library

[43]

Liu, C., Yuen, J., Torralba, A., Sivic, J., and Freeman, W. T. 2008. Sift flow: Dense correspondence across different scenes. In Proc. ECCV, vol. 3, 28--42.

Digital Library

[44]

Lowe, D. G. 2004. Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vision 60, 2, 91--110.

Digital Library

[45]

Malisiewicz, T., and Efros, A. 2009. Beyond categories: The visual memex model for reasoning about object relationships. In Proc. NIPS.

[46]

Oliva, A., and Torralba, A. 2001. Modeling the shape of the scene: A holistic representation of the spatial envelope. International Journal of Computer Vision 42, 3, 145--175.

Digital Library

[47]

Perazzi, F., Krahenbuhl, P., Pritch, Y., and Hornung, A. 2012. Saliency filters: Contrast based filtering for salient region detection. In Proc. CVPR.

Digital Library

[48]

Pérez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. ACM Trans. Graph. 22, 3 (July), 313--318.

Digital Library

[49]

Risser, E., Han, C., Dahyot, R., and Grinspun, E. 2010. Synthesizing structured image hybrids. ACM Trans. Graph. 29 (July), 85:1--85:6.

Digital Library

[50]

Russakovsky, O., Lin, Y., Yu, K., and Fei-Fei, L. 2012. Object-centric spatial pooling for image classification. In Proc. ECCV, Springer, 1--15.

Digital Library

[51]

Shrivastava, A., Malisiewicz, T., Gupta, A., and Efros, A. A. 2011. Data-driven visual similarity for cross-domain image matching. ACM Trans. Graph. 30, 6, 154.

Digital Library

[52]

Sivic, J., and Zisserman, A. 2003. Video google: A text retrieval approach to object matching in videos. In Proc. ICCV, IEEE, 1470--1477.

Digital Library

[53]

Sun, J., Yuan, L., Jia, J., and Shum, H.-Y. 2005. Image completion with structure propagation. ACM Trans. Graph. 24, 3, 861--868.

Digital Library

[54]

Wei, L.-Y., Han, J., Zhou, K., Bao, H., Guo, B., and Shum, H.-Y. 2008. Inverse texture synthesis. ACM Trans. Graph. 27, 3 (Aug.), 52:1--52:9.

Digital Library

[55]

Wei, L. Y., Lefebvre, S., Kwatra, V., and Turk, G. 2009. State of the art in example-based texture synthesis. In EG STAR, 93--117.

[56]

Wexler, Y., Shechtman, E., and Irani, M. 2007. Spacetime completion of video. IEEE Trans. Pattern Anal. Mach. Intell. 29, 3, 463--476.

Digital Library

[57]

Wilczkowiak, M., Brostow, G. J., Tordoff, B., and Cipolla, R. 2005. Hole filling through photomontage. In Proc. BMVC.

[58]

Wu, Y., and Fan, J. 2009. Contextual flow. In Proc. CVPR, IEEE, 33--40.

[59]

Xiao, C., Liu, M., Nie, Y., and Dong, Z. 2011. Fast exact nearest patch matching for patch-based image editing and processing. IEEE Transactions on Visualization and Computer Graphics 17, 8, 1122--1134.

Digital Library

[60]

Zhang, Y., and Tong, R. 2011. Environment-sensitive cloning in images. The Visual Computer 27, 6--8, 739--748.

Digital Library

[61]

Zhang, S.-H., Tong, Q., Hu, S.-M., and Martin, R. R. 2011. Painting patches: reducing flicker in painterly re-rendering of video. Science China-Information Sciences 54, 12, 2592--2601.

[62]

Zhang, F.-L., Cheng, M.-M., Jia, J., and Hu, S.-M. 2012. Imageadmixture: Putting together dissimilar objects from groups. IEEE Transactions on Visualization and Computer Graphics 18, 11, 1849--1857.

Digital Library

Cited By

Huang YCao YLai YShan YGao L(2024)NeRF-Texture: Synthesizing Neural Radiance Field TexturesIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.338219846:9(5986-6000)Online publication date: Sep-2024
https://doi.org/10.1109/TPAMI.2024.3382198
Yi BSun Y(2023)Enhancing Educational Outcomes with a High-performance Online Exam SystemAcademic Journal of Science and Technology10.54097/ajst.v5i2.69945:2(239-243)Online publication date: 2-Apr-2023
https://doi.org/10.54097/ajst.v5i2.6994
Gao LLiu FChen SJiang KLi CLai YFu H(2023)SketchFaceNeRF: Sketch-based Facial Generation and Editing in Neural Radiance FieldsACM Transactions on Graphics10.1145/359210042:4(1-17)Online publication date: 26-Jul-2023
https://dl.acm.org/doi/10.1145/3592100
Show More Cited By

Index Terms

PatchNet: a patch-based image representation for interactive library-driven image editing
1. Computing methodologies
  1. Computer graphics
2. Social and professional topics
  1. Professional topics

Recommendations

Application of PatchNet in image representation

PatchNet, as a graph model with hierarchical structure, is a new technology of image representation. Its description structure for images can well conform to the cognitive features of human visual system. The semantic information and geometric ...
Rectangle NAM Image Representation and Contour Extraction of Binary Image Represented by NAM
CISP '08: Proceedings of the 2008 Congress on Image and Signal Processing, Vol. 3 - Volume 03

Less data and faster operation are the aims of image representation. The popular matrix image representation needs huge storage space and its processing speed is usually slow because of its operation pattern of pixel to pixel. Some traditional image ...
Representing images using points on image surfaces

This paper presents a new approach to represent an image by "verge points," which are defined as high-curvature points on the image surface. This representation offers a compact and reversible way to preserve the essence of the original image. Various ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 32, Issue 6

November 2013

671 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/2508363

Issue’s Table of Contents

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 November 2013

Published in TOG Volume 32, Issue 6

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Tsinghua University
National Natural Science Foundation of China
Ministry of Science and Technology of the People's Republic of China
Program for Changjiang Scholars and Innovative Research Team in University

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

39
Total Citations
View Citations
812
Total Downloads

Downloads (Last 12 months)26
Downloads (Last 6 weeks)2

Reflects downloads up to 25 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Huang YCao YLai YShan YGao L(2024)NeRF-Texture: Synthesizing Neural Radiance Field TexturesIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.338219846:9(5986-6000)Online publication date: Sep-2024
https://doi.org/10.1109/TPAMI.2024.3382198
Yi BSun Y(2023)Enhancing Educational Outcomes with a High-performance Online Exam SystemAcademic Journal of Science and Technology10.54097/ajst.v5i2.69945:2(239-243)Online publication date: 2-Apr-2023
https://doi.org/10.54097/ajst.v5i2.6994
Gao LLiu FChen SJiang KLi CLai YFu H(2023)SketchFaceNeRF: Sketch-based Facial Generation and Editing in Neural Radiance FieldsACM Transactions on Graphics10.1145/359210042:4(1-17)Online publication date: 26-Jul-2023
https://dl.acm.org/doi/10.1145/3592100
Huang YCao YLai YShan YGao L(2023)NeRF-Texture: Texture Synthesis with Neural Radiance FieldsACM SIGGRAPH 2023 Conference Proceedings10.1145/3588432.3591484(1-10)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.1145/3588432.3591484
Wang HJi XPeng KWang WWang S(2023)PVONet: point-voxel-based semi-supervision monocular three-dimensional object detection using LiDAR camera systemsJournal of Electronic Imaging10.1117/1.JEI.32.5.05301532:05Online publication date: 1-Sep-2023
https://doi.org/10.1117/1.JEI.32.5.053015
Jiang NSheng BLi PLee T(2023)PhotoHelper: Portrait Photographing Guidance Via Deep Feature Retrieval and FusionIEEE Transactions on Multimedia10.1109/TMM.2022.314489025(2226-2238)Online publication date: 2023
https://doi.org/10.1109/TMM.2022.3144890
Farshad AYeganeh YChi YShen COmmer BNavab N(2023)SceneGenie: Scene Graph Guided Diffusion Models for Image Synthesis2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)10.1109/ICCVW60793.2023.00016(88-98)Online publication date: 2-Oct-2023
https://doi.org/10.1109/ICCVW60793.2023.00016
Liu XHao CSu ZQi ZFu SLi YHan H(2022)Image inpainting algorithm based on tensor decomposition and weighted nuclear normMultimedia Tools and Applications10.1007/s11042-022-12635-382:3(3433-3458)Online publication date: 5-Jul-2022
https://dl.acm.org/doi/10.1007/s11042-022-12635-3
Zhong LLi FHuang HZhang YLu SWang J(2021)Aesthetic-guided outward image croppingACM Transactions on Graphics10.1145/3478513.348056640:6(1-13)Online publication date: 10-Dec-2021
https://dl.acm.org/doi/10.1145/3478513.3480566
Zhao KHan QZhang CXu JCheng M(2021)Deep Hough Transform for Semantic Line DetectionIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2021.3077129(1-1)Online publication date: 2021
https://doi.org/10.1109/TPAMI.2021.3077129
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents