CN108810512B - A kind of object-based stereo-picture depth method of adjustment - Google Patents

Abstract

The invention discloses a kind of object-based stereo-picture depth methods of adjustment, it keeps energy, total object control energy, total depth to control energy and total background holding energy by the corresponding total edge of left view point image and right visual point image for extracting stereo-picture, and by optimization so that gross energy is minimum, obtain best similitude transformation matrix, it aloows depth stereo-picture adjusted to retain accurate object shapes, comfort with higher and sense of depth in this way, and can adaptively control the scaling of important content according to the user's choice；It controls the deformation of quadrilateral mesh by all coordinate positions for falling in the characteristic point in important area and in background area in control stereo-picture in turn, so as to guarantee the comfort and sense of depth of depth stereo-picture adjusted.

Description

Object-based stereo image depth adjustment method

Technical Field

The present invention relates to a method for processing an image signal, and more particularly, to a method for adjusting a depth of a stereoscopic image based on an object.

Background

With the rapid development of 3D technology, stereo images and stereo videos are more and more noticed and favored by people. Especially, with the development of mobile phones, tablets and personal computers, the display of the mobile terminal is more and more popular among users. However, when displaying a stereoscopic image and a stereoscopic video on a mobile terminal screen, the stereoscopic sensation may be reduced or even disappear, and a movie maker tries to increase the stereoscopic sensation of a specific object by adjusting the size and depth of the object to focus the viewer on the object. Accordingly, for displaying a stereoscopic image and a stereoscopic video on a mobile terminal screen, depth adjustment based on an object can enhance the attention and depth perception of the object.

In the depth adjustment of a stereoscopic image, two methods are roughly classified: depth adjustment using the depth map and depth adjustment without the depth map. The former method requires an accurate depth map and generates an image with adjusted depth using a virtual viewpoint rendering technique; the latter method achieves the purpose of depth adjustment directly by moving pixels in an image, however, the method often generates a hole or causes deformation of an object after depth adjustment, and therefore, how to reduce image deformation of a stereoscopic image after depth adjustment and how to adaptively control adjustment of the object according to selection of a user to highlight significant content are problems which need to be researched and solved in the process of depth adjustment of the stereoscopic image.

Disclosure of Invention

The technical problem to be solved by the present invention is to provide an object-based depth adjustment method for a stereoscopic image, which enables the stereoscopic image after depth adjustment to retain an accurate object shape, have high comfort and depth feeling, and can adaptively control the scaling of important content according to the selection of a user.

The technical scheme adopted by the invention for solving the technical problems is as follows: a method for adjusting depth of a stereoscopic image based on an object is characterized by comprising the following steps:

the method comprises the following steps: the left viewpoint image, the right viewpoint image, and the left parallax image of the stereoscopic image having the width W and the height H to be processed are correspondingly denoted as { L (x, y) }, { R (x, y) }, and { d_L(x, y) }; wherein x is more than or equal to 1 and less than or equal to W, y is more than or equal to 1 and less than or equal to H, W and H can be evenly divided by 8, L (x, y) represents the pixel value of the pixel point with the coordinate position (x, y) in { L (x, y) }, R (x, y) represents the pixel value of the pixel point with the coordinate position (x, y) in { R (x, y) }, d_L(x, y) represents { d }_LThe coordinate position in (x, y) is the pixel value of the pixel point of (x, y);

step two: dividing { L (x, y) } intoFour-sided grids of 8 × 8 size and not overlapping each other, and the kth four-sided grid in { L (x, y) } is marked as U_L,kAnd the set of all quadrilateral meshes in { L (x, y) } is denoted as V_L，V_L＝{U_L,kL 1 is more than or equal to k and less than or equal to M }; then according to all quadrilateral meshes in { L (x, y) } and { d_L(x, y) }, acquiring all non-overlapping quadrilateral grids with the size of 8 multiplied by 8 in the { R (x, y) }, and marking the kth quadrilateral grid in the { R (x, y) }asU_R,kAnd the set of all quadrilateral meshes in { R (x, y) } is denoted as V_R，V_R＝{U_R,kL 1 is more than or equal to k and less than or equal to M }; wherein k is a positive integer, k is not less than 1 and not more than M, M represents the total number of quadrilateral meshes contained in { L (x, y) } and also represents the total number of quadrilateral meshes contained in { R (x, y) },U_L,kdescribed by its set of 4 mesh vertices above left, below left, above right and below right, corresponds to and represents U_L,kA left upper grid vertex as a1 st grid vertex, a left lower grid vertex as a2 nd grid vertex, a right upper grid vertex as a 3 rd grid vertex, a right lower grid vertex as a 4 th grid vertex,to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that,U_R,kdescribed by its set of 4 mesh vertices above left, below left, above right and below right, corresponds to and represents U_R,kA left upper grid vertex as a1 st grid vertex, a left lower grid vertex as a2 nd grid vertex, a right upper grid vertex as a 3 rd grid vertex, a right lower grid vertex as a 4 th grid vertex,to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, represents { d_L(x, y) } coordinate position ofThe pixel value of the pixel point of (a),to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, represents { d_L(x, y) } coordinate position ofThe pixel value of the pixel point of (a),to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, represents { d_L(x, y) } coordinate position ofThe pixel value of the pixel point of (a),to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, represents { d_L(x, y) } coordinate position ofThe pixel value of the pixel point of (1);

extracting all feature points in the { L (x, y) } by adopting scale-invariant feature conversion; then, the q-th feature point in { L (x, y) } is noted as Then according to each feature in { L (x, y) }Dot sum { d_L(x, y) }, acquiring feature points in the { R (x, y) } which are matched with each feature point in the { L (x, y) }, and matching the feature points in the { R (x, y) } with the feature points in the { L (x, y) }The matched feature points are recorded as Wherein Q is a positive integer, 1. ltoreq. q.ltoreq.Q, Q represents the total number of feature points in { L (x, y) }, and also represents the total number of feature points in { R (x, y) },to representThe position of the abscissa of the bar (a),to representThe position of the ordinate of (a) is,to representThe position of the abscissa of the bar (a), represents { d_L(x, y) } coordinate position ofThe pixel value of the pixel point of (a),to representThe position of the ordinate of (a) is,

step three: calculating the total edge retention energy of the target quadrilateral meshes corresponding to all quadrilateral meshes in the { L (x, y) } and the { R (x, y) } according to a matrix formed by the edges of all grid vertexes of each quadrilateral mesh in the { L (x, y) } and a matrix formed by the edges of all grid vertexes of the target quadrilateral mesh corresponding to each quadrilateral mesh in the { L (x, y) }, a matrix formed by the edges of all grid vertexes of each quadrilateral mesh in the { R (x, y) } and a matrix formed by the edges of all grid vertexes of the target quadrilateral mesh corresponding to each quadrilateral mesh in the { R (x, y) }, and recording the total edge retention energy as E_edge；

According to the important content selected by the user, calculating the total object control energy of target quadrilateral grids corresponding to all quadrilateral grids in the rectangular area where the important content selected by the user in { L (x, y) } and { R (x, y) } is located, and marking the total object control energy as E_object；

According to all feature points in the rectangular areas where the important contents selected by the user are located in the { L (x, y) } and the { R (x, y) } the total depth control energy of the target quadrilateral grids corresponding to all quadrilateral grids in the rectangular areas where the important contents selected by the user are located in the { L (x, y) } and the { R (x, y) } is calculated and recorded as E_depth；

Calculating the total of target quadrilateral grids corresponding to all quadrilateral grids in the { L (x, y) } and the { R (x, y) } in the background area according to all the feature points in the { L (x, y) } and the corresponding target feature points, and all the feature points in the { R (x, y) } in the background area and the corresponding target feature pointsBackground Retention energy, denoted E_back(ii) a The background area is an area except a rectangular area where the important content selected by the user is located;

step four: according to E_edge、E_object、E_depthAnd E_backCalculating the total energy of the target quadrilateral grids corresponding to all the quadrilateral grids in the { L (x, y) } and the { R (x, y) }, and recording the total energy as E_total，E_total＝E_edge+λ_object×E_object+λ_depth×E_depth+λ_back×E_back(ii) a Then solving by least squares optimizationObtaining a set formed by the optimal target quadrilateral grids corresponding to all quadrilateral grids in the { L (x, y) } and a set formed by the optimal target quadrilateral grids corresponding to all quadrilateral grids in the { R (x, y) }, and recording the optimal target quadrilateral grids corresponding to all quadrilateral grids in the { L (x, y) }asthe optimal target quadrilateral grids corresponding to all quadrilateral grids in the { R (x, y) } correspondinglyAnd then according toCalculating the optimal similarity transformation matrix of the optimal target quadrilateral grids corresponding to each quadrilateral grid in the { L (x, y) }, and converting U into U_L,kCorresponding optimal target quadrilateral meshIs recorded as the optimal similarity transformation matrix And according toCalculating the optimal similarity transformation matrix of the optimal target quadrilateral grids corresponding to each quadrilateral grid in the { R (x, y) }, and converting U into U_R,kCorresponding optimal target quadrilateral meshIs recorded as the optimal similarity transformation matrix

Wherein λ is_objectIs E_objectA weighting parameter of_depthIs E_depthA weighting parameter of_backIs E_backMin () is a function taking the minimum value,represents a set of target quadrilateral meshes corresponding to all quadrilateral meshes in the { L (x, y) },represents a set of target quadrilateral meshes corresponding to all quadrilateral meshes in R (x, y),represents U_L,kThe corresponding optimal target quadrilateral mesh is selected from the set of target quadrilateral meshes, corresponding representation1 st mesh vertex, 2 nd mesh vertex, 3 rd mesh vertex, 4 th mesh vertex,represents U_R,kThe corresponding optimal target quadrilateral mesh is selected from the set of target quadrilateral meshes, corresponding representation1 st mesh vertex, 2 nd mesh vertex, 3 rd mesh vertex, 4 th mesh vertex of (a)_L,k)^TIs A_L,kTranspose of (A) ((A)_L,k)^TA_L,k)^-1Is (A)_L,k)^TA_L,kThe inverse of (a) is,andcorresponding representationA horizontal coordinate position and a vertical coordinate position of,andcorresponding representationA horizontal coordinate position and a vertical coordinate position of,andcorresponding representationA horizontal coordinate position and a vertical coordinate position of,andcorresponding representation(ii) a horizontal coordinate position and a vertical coordinate position of (A)_R,k)^TIs A_R,kTranspose of (A) ((A)_R,k)^TA_R,k)^-1Is (A)_R,k)^TA_R,kThe inverse of (a) is,andcorresponding representationA horizontal coordinate position and a vertical coordinate position of,andcorresponding representationHorizontal coordinate ofThe position and the position of the vertical coordinate,andcorresponding representationA horizontal coordinate position and a vertical coordinate position of,andcorresponding representationHorizontal coordinate position and vertical coordinate position of (a);

step five: according to the optimal similarity transformation matrix of the optimal target quadrilateral grid corresponding to each quadrilateral grid in the { L (x, y) }, calculating the horizontal coordinate position and the vertical coordinate position of each pixel point in each quadrilateral grid in the { L (x, y) } after the optimal similarity transformation rectangular transformation, and converting the U into the U-shaped coordinate position_L,kThe position of the middle horizontal coordinate is x'_L,kAnd vertical coordinate position y'_L,kThe correspondence of the horizontal coordinate position and the vertical coordinate position of the pixel point after the optimal similarity transformation matrix is recorded asAnd then according to the horizontal coordinate position sum of each pixel point in each quadrilateral grid in the { L (x, y) } after the rectangular transformation of the optimal similarity transformationThe vertical coordinate position is used for obtaining a left viewpoint image after the depth adjustment and recording the left viewpoint image asWherein x is not less than 1'_L,k≤W，1≤y'_L,k≤H，X 'is more than or equal to 1 and less than or equal to W', y 'is more than or equal to 1 and less than or equal to H, W' represents the width of the three-dimensional image after the depth adjustment, H is the height of the three-dimensional image after the depth adjustment,to representThe pixel value of the pixel point with the middle coordinate position (x ', y');

similarly, according to the optimal similarity transformation matrix of the optimal target quadrilateral grid corresponding to each quadrilateral grid in the { R (x, y) }, calculating the horizontal coordinate position and the vertical coordinate position of each pixel point in each quadrilateral grid in the { R (x, y) } after the optimal similarity transformation rectangular transformation, and converting the U into the U shape_R,kThe position of the middle horizontal coordinate is x'_R,kAnd vertical coordinate position y'_R,kThe correspondence of the horizontal coordinate position and the vertical coordinate position of the pixel point after the optimal similarity transformation matrix is recorded asAnd then, according to the horizontal coordinate position and the vertical coordinate position of each pixel point in each quadrilateral grid in the { R (x, y) } after the optimal similarity transformation rectangular transformation, a right viewpoint image after depth adjustment is obtained and recorded as a right viewpoint imageWherein x is not less than 1'_R,k≤W，1≤y'_R,k≤H，1≤x'≤W'，1≤y'≤H，To representAnd the pixel value of the pixel point with the middle coordinate position of (x ', y').

E in the third step_edgeThe calculation process of (2) is as follows:

a1, calculating the edge holding energy of the target quadrilateral grids corresponding to all quadrilateral grids in the { L (x, y) }, and recording the edge holding energy as Wherein e is_L,kRepresents U_L,kIs used to form a matrix of edges of all mesh vertices,(e_L,k)^Tis e_L,kTranspose of (e) ((e)_L,k)^Te_L,k)^-1Is (e)_L,k)^Te_L,kThe inverse of (a) is,to representIs used to form a matrix of edges of all mesh vertices, represents U_L,kThe corresponding target quadrilateral mesh is then selected,described by its set of 4 mesh vertices above left, below left, above right and below right, corresponding representationA left upper grid vertex as a1 st grid vertex, a left lower grid vertex as a2 nd grid vertex, a right upper grid vertex as a 3 rd grid vertex, a right lower grid vertex as a 4 th grid vertex,to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that,the symbol "| | |" is to ask EuropeA number of distance symbols;

similarly, the edge holding energy of the target quadrilateral mesh corresponding to all quadrilateral meshes in { R (x, y) } is calculated and recorded as Wherein e is_R,kRepresents U_R,kIs used to form a matrix of edges of all mesh vertices,(e_R,k)^Tis e_R,kTranspose of (e) ((e)_R,k)^Te_R,k)^-1Is (e)_R,k)^Te_R,kThe inverse of (a) is,to representIs used to form a matrix of edges of all mesh vertices, represents U_R,kThe corresponding target quadrilateral mesh is then selected,described by its set of 4 mesh vertices above left, below left, above right and below right, corresponding representationA left upper grid vertex as a1 st grid vertex, a left lower grid vertex as a2 nd grid vertex, a right upper grid vertex as a 3 rd grid vertex, a right lower grid vertex as a 4 th grid vertex,to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that,

a2, according toAndcalculating the total edge-preserving energy E of the target quadrilateral meshes corresponding to all quadrilateral meshes in the { L (x, y) } and the { R (x, y) } according to the total edge-preserving energy E_edge，

E in the third step_objectThe calculation process of (2) is as follows:wherein,a range of a rectangular area in which important contents selected by the user are indicated,denotes a horizontal coordinate position of a mesh vertex of { L (x, y) } which is jth in the horizontal direction and ith in the vertical direction,denotes a horizontal coordinate position of a mesh vertex of { L (x, y) } which is j +1 th in the horizontal direction and i-th in the vertical direction,denotes a horizontal coordinate position of a mesh vertex in the target quadrangular mesh, of a mesh vertex which is jth in the horizontal direction and ith in the vertical direction in { L (x, y) },denotes the horizontal coordinate position of the mesh vertex in the target quadrangular mesh of the mesh vertex of j +1 th in the horizontal direction and i th in the vertical direction in { L (x, y) },denotes a horizontal coordinate position of a mesh vertex of { R (x, y) } which is jth in the horizontal direction and ith in the vertical direction,denotes a horizontal coordinate position of a mesh vertex of { R (x, y) } which is j +1 th in the horizontal direction and i-th in the vertical direction,denotes the horizontal direction in { R (x, y) }To the horizontal coordinate position of the mesh vertex in the target quadrilateral mesh of the mesh vertex which is jth and ith in the vertical direction,denotes the horizontal coordinate position, s, of the mesh vertex in the target quadrilateral mesh of the mesh vertex in { R (x, y) } which is j +1 th in the horizontal direction and i-th in the vertical direction_xA horizontal scaling factor representing user-specified important content.

E in the third step_depthThe calculation process of (2) is as follows:

b1, calculating comfort level maintenance energy of all feature points in the rectangular area where the important content selected by the user is located in the { L (x, y) }, and recording the comfort level maintenance energy as E_co_mfo_rt， Wherein,representing a set formed by the serial numbers of all the characteristic points which fall in the rectangular area of the important content selected by the user after renumbering in the { L (x, y) },indicates the depth value of the p-th feature point in the L (x, y) within the rectangular area where the important content selected by the user is located,exp () represents an exponential function with the natural base e as the base, the symbol "|" is an absolute value symbol, Z_maxDenotes the maximum depth value of { L (x, y) }, Z_minDenotes the minimum depth value of { L (x, y) }, CVZ_minA minimum comfortable viewing zone range is indicated,H_erepresenting a horizontal baseline distance between left and right viewpoints of the stereoscopic image to be processed, D representing a viewing distance between the left and right viewpoints of the stereoscopic image to be processed and the display, W_dIndicating the horizontal width, R, of the display_dIndicating the horizontal resolution of the display, η₁Indicating minimum comfortable viewing angle, CVZ_maxIndicating the maximum comfortable viewing zone range,η₂indicating the maximum comfortable viewing angle of view,the parallax value representing the p-th feature point in { L (x, y) } that falls within the rectangular region in which the important content selected by the user is located also represents { d }_LThe coordinate position in (x, y) is the pixel value of the pixel point of the p-th characteristic point in the rectangular area where the important content selected by the user is located,to representThe target depth value of (2);

b2, calculating the feature retention energy of all feature points in the rectangular area where the important content selected by the user is located in the { L (x, y) }, and recording the feature retention energy as E_feature，Wherein,representing that all the feature points in the eight neighborhood range adjacent to the p-th feature point in the rectangular region in which the important content selected by the user is located in the rectangular region are renumbered in the { L (x, y) } stateIs a set of sequence numbers of (a),indicating the depth value of the p' th characteristic point in the eight neighborhood range adjacent to the p-th characteristic point in the rectangular area in which the important content selected by the user is located in the L (x, y),to representThe target depth value of (2);

b3 by solving for min (E)_comfort+λ_feature×E_feature) Obtaining the optimal target depth value set of all feature points in the rectangular region where the important content selected by the user falls in { L (x, y) } and recording the optimal target depth value set as Is composed ofThe optimal target depth value; then according toAcquiring the optimal target disparity value set of all feature points in the rectangular region where the important content selected by the user is located in the { L (x, y) } and recording the optimal target disparity value set as Is composed ofThe optimal target disparity value of (a) is,wherein min () is a function of taking the minimum value, λ_featureIs E_featureThe weighting parameter of (2);

b4, according toCalculating the total depth control energy E of target quadrilateral grids corresponding to all quadrilateral grids in the rectangular area where the important content selected by the user is located in the { L (x, y) } and the { R (x, y) } are located_depth，

Wherein the symbol "| | |" is a euclidean distance-solving symbol,indicating the p-th feature point in the { L (x, y) } that falls within the rectangular area where the important content selected by the user is located,represents the p-th feature point in the R (x, y) within the rectangular area where the important content selected by the user is located,to representThe corresponding target feature points are set to be in a same shape,to representThe corresponding target feature points are set to be in a same shape,to representThe position of the abscissa of the bar (a),to representThe position of the abscissa of the bar (a),to representThe position of the abscissa of the bar (a),to representThe horizontal coordinate position of the X-ray detector is that i 'is more than or equal to 1 and less than or equal to 4, k' is more than or equal to 1 and less than or equal to M,to representQuadrilateral grid U_L,k'ith' mesh vertex, U_L,k'is the kth' quadrilateral mesh in { L (x, y) },to representQuadrilateral grid U_L,k' as the 1 st mesh vertex,to representQuadrilateral grid U_L,k' as the 2 nd mesh vertex,to representQuadrilateral grid U_L,k' as the top-right mesh vertex of the 3 rd mesh vertex,to representQuadrilateral grid U_L,k' as the lower right mesh vertex of the 4 th mesh vertex,to representAndthe euclidean distance between them,to representTarget quadrilateral meshThe ith' mesh vertex of (a),is U_L,k' a corresponding target quadrilateral mesh,to representTarget quadrilateral meshAs the 1 st mesh vertex,to representTarget quadrilateral meshAs the 2 nd mesh vertex,to representTarget quadrilateral meshAs the top-right mesh vertex of the 3 rd mesh vertex,to representTarget quadrilateral meshAs the 4 th mesh vertex,to representQuadrilateral grid U_R,k'ith' mesh vertex, U_R,k'is the kth' quadrilateral mesh in { R (x, y) },denotes fR_pQuadrilateral grid U_R,k' as the 1 st mesh vertex,to representQuadrilateral grid U_R,k' as the 2 nd mesh vertex,to representQuadrilateral grid U_R,k' as the top-right mesh vertex of the 3 rd mesh vertex,to representQuadrilateral grid U_R,k' as the lower right mesh vertex of the 4 th mesh vertex,to representAndthe euclidean distance between them,to representTarget quadrilateral meshThe ith' mesh vertex of (a),is U_R,k' a corresponding target quadrilateral mesh,representing characteristic pointsTarget quadrilateral meshAs the 1 st mesh vertex,to representTarget quadrilateral meshAs the 2 nd mesh vertex,to representTarget quadrilateral meshAs the top-right mesh vertex of the 3 rd mesh vertex,representing characteristic pointsTarget quadrilateral meshAs the 4 th mesh vertex.

E in the third step_backThe calculation process of (2) is as follows: wherein,representing a set of sequence numbers formed by renumbering all feature points falling in the background area in { L (x, y) } and { R (x, y) },represents the g-th feature point falling within the background region in { L (x, y) },represents the g-th feature point falling within the background region in { R (x, y) },to representThe corresponding target feature points are set to be in a same shape,to representCorresponding target characteristic points, the symbol "| | |" is a Euclidean distance-calculating symbol, i' is more than or equal to 1 and less than or equal to 4, k is more than or equal to 1 and less than or equal to M,to representQuadrilateral grid U_L,k"ith' mesh vertex, U_L,k"is the kth" quadrilateral mesh in { L (x, y) },to representQuadrilateral grid U_L,k"left upper grid vertex as the 1 st grid vertex,to representQuadrilateral grid U_L,k"as the 2 nd mesh vertex,to representQuadrilateral grid U_L,k"top right mesh vertex as the 3 rd mesh vertex,to representQuadrilateral grid U_L,k"as the lower right mesh vertex of the 4 th mesh vertex,to representAndthe euclidean distance between them,to representTarget quadrilateral meshThe ith' mesh vertex of (a),is U_L,k"the corresponding target quadrilateral mesh,to representIs located atTarget quadrilateral mesh ofAs the 1 st mesh vertex,to representTarget quadrilateral meshAs the 2 nd mesh vertex,to representTarget quadrilateral meshAs the top-right mesh vertex of the 3 rd mesh vertex,to representTarget quadrilateral meshAs the 4 th mesh vertex,to representQuadrilateral grid U_R,kThe ith' ofMesh vertex, U_R,k"is the kth" quadrilateral mesh in { R (x, y) },to representQuadrilateral grid U_R,k"left upper grid vertex as the 1 st grid vertex,to representQuadrilateral grid U_R,k"as the 2 nd mesh vertex,to representQuadrilateral grid U_R,k"top right mesh vertex as the 3 rd mesh vertex,to representQuadrilateral grid U_R,k"as the lower right mesh vertex of the 4 th mesh vertex,to representAndthe euclidean distance between them,to representTarget quadrilateral meshThe ith' mesh vertex of (a),is U_R,k"the corresponding target quadrilateral mesh,to representTarget quadrilateral meshAs the 1 st mesh vertex,representing characteristic pointsTarget quadrilateral meshAs the 2 nd mesh vertex,representing characteristic pointsTarget quadrilateral meshAs the top-right mesh vertex of the 3 rd mesh vertex,representing characteristic pointsTarget quadrilateral meshAs the 4 th mesh vertex.

Compared with the prior art, the invention has the advantages that:

1) the method extracts the total edge holding energy, the total object control energy, the total depth control energy and the total background holding energy corresponding to the left viewpoint image and the right viewpoint image of the stereo image, minimizes the total energy through optimization, and obtains the optimal similarity transformation matrix, so that the stereo image after depth adjustment can keep accurate object shape, has higher comfort and depth feeling, and can adaptively control the scaling of important contents according to the selection of a user.

2) The method of the invention controls the coordinate positions of all the characteristic points in the stereo image in the important area and the background area, and further controls the deformation of the quadrilateral grids, thereby ensuring the comfort and the depth feeling of the stereo image after depth adjustment.

Drawings

FIG. 1 is a block diagram of an overall implementation of the method of the present invention;

FIG. 2a is a "red/green" view of the original stereo Image of "Image 1";

FIG. 2b is a "red/green" diagram of "Image 1" after object depth adjustment;

FIG. 3a is a "red/green" view of the original stereo Image of "Image 2";

FIG. 3b is a "red/green" diagram of "Image 2" after object depth adjustment;

FIG. 4a is a "red/green" view of the original stereo Image of "Image 3";

FIG. 4b is a "red/green" diagram of "Image 3" after object depth adjustment;

FIG. 5a is a "red/green" view of the original stereo Image of "Image 4";

fig. 5b is a "red/green" diagram after the object depth adjustment of "Image 4".

Detailed Description

The invention is described in further detail below with reference to the accompanying examples.

The general implementation block diagram of the method for adjusting the depth of a stereoscopic image based on an object, which is provided by the invention, is shown in fig. 1, and the method comprises the following steps:

the method comprises the following steps: the left viewpoint image, the right viewpoint image, and the left parallax image of the stereoscopic image having the width W and the height H to be processed are correspondingly denoted as { L (x, y) }, { R (x, y) }, and { d_L(x, y) }; wherein x is more than or equal to 1 and less than or equal to W, y is more than or equal to 1 and less than or equal to H, W and H can be evenly divided by 8, L (x, y) represents the pixel value of the pixel point with the coordinate position (x, y) in { L (x, y) }, R (x, y) represents the pixel value of the pixel point with the coordinate position (x, y) in { R (x, y) }, d_L(x, y) represents { d }_LAnd the coordinate position in the (x, y) is the pixel value of the pixel point of (x, y).

Step two: dividing { L (x, y) } intoA grid of non-overlapping 8 × 8 quadrilateral dimensions, which are defined by { L (x, y) } ofThe kth quadrilateral mesh is marked as U_L,kAnd the set of all quadrilateral meshes in { L (x, y) } is denoted as V_L，V_L＝{U_L,kL 1 is more than or equal to k and less than or equal to M }; then according to all quadrilateral meshes in { L (x, y) } and { d_L(x, y) }, acquiring all non-overlapping quadrilateral grids with the size of 8 multiplied by 8 in the { R (x, y) }, and marking the kth quadrilateral grid in the { R (x, y) }asU_R,kAnd the set of all quadrilateral meshes in { R (x, y) } is denoted as V_R，V_R＝{U_R,kL 1 is more than or equal to k and less than or equal to M }; wherein k is a positive integer, k is not less than 1 and not more than M, M represents the total number of quadrilateral meshes contained in { L (x, y) } and also represents the total number of quadrilateral meshes contained in { R (x, y) },U_L,kdescribed by its set of 4 mesh vertices above left, below left, above right and below right, corresponds to and represents U_L,kA left upper grid vertex as a1 st grid vertex, a left lower grid vertex as a2 nd grid vertex, a right upper grid vertex as a 3 rd grid vertex, a right lower grid vertex as a 4 th grid vertex,to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, to be provided withHorizontal coordinate position ofAnd vertical coordinate positionDevice for placingTo be described, the method has the advantages that,U_R,kdescribed by its set of 4 mesh vertices above left, below left, above right and below right, corresponds to and represents U_R,kA left upper grid vertex as a1 st grid vertex, a left lower grid vertex as a2 nd grid vertex, a right upper grid vertex as a 3 rd grid vertex, a right lower grid vertex as a 4 th grid vertex,to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, represents { d_L(x, y) } coordinate position ofThe pixel value of the pixel point of (a),to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, represents { d_L(x, y) } coordinate position ofThe pixel value of the pixel point of (a),to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, represents { d_L(x, y) } coordinate position ofThe pixel value of the pixel point of (a),to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, represents { d_L(x, y) } coordinate position ofThe pixel value of the pixel point of (1).

Extracting all Feature points in { L (x, y) } by adopting the conventional Scale Invariant Feature Transform (SIFT); then, the q-th feature point in { L (x, y) } is noted as Let a set of all feature points in { L (x, y) } be denoted asThen according to each feature point in { L (x, y) } and { d }_L(x, y) }, acquiring feature points in the { R (x, y) } which are matched with each feature point in the { L (x, y) }, and matching the feature points in the { R (x, y) } with the feature points in the { L (x, y) }The matched feature points are recorded as Let a set of all feature points in { R (x, y) } be denoted asWherein Q is a positive integer, 1. ltoreq. q.ltoreq.Q, Q represents the total number of feature points in { L (x, y) }, and also represents the total number of feature points in { R (x, y) },to representThe position of the abscissa of the bar (a),to representThe position of the ordinate of (a) is,to representThe position of the abscissa of the bar (a), represents { d_L(x, y) } coordinate position ofThe pixel value of the pixel point of (a),to representThe position of the ordinate of (a) is,

step three: calculating the total edge retention energy of the target quadrilateral meshes corresponding to all quadrilateral meshes in the { L (x, y) } and the { R (x, y) } according to a matrix formed by the edges of all grid vertexes of each quadrilateral mesh in the { L (x, y) } and a matrix formed by the edges of all grid vertexes of the target quadrilateral mesh corresponding to each quadrilateral mesh in the { L (x, y) }, a matrix formed by the edges of all grid vertexes of each quadrilateral mesh in the { R (x, y) } and a matrix formed by the edges of all grid vertexes of the target quadrilateral mesh corresponding to each quadrilateral mesh in the { R (x, y) }, and recording the total edge retention energy as E_edge。

In this embodiment, step E of step three_edgeThe calculation process of (2) is as follows:

a1, calculating the edge holding energy of the target quadrilateral grids corresponding to all quadrilateral grids in the { L (x, y) }, and recording the edge holding energy as Wherein e is_L,kRepresents U_L,kAll nets ofThe matrix of edges at the vertices of the grid,(e_L,k)^Tis e_L,kTranspose of (e) ((e)_L,k)^Te_L,k)^-1Is (e)_L,k)^Te_L,kThe inverse of (a) is,to representIs used to form a matrix of edges of all mesh vertices, represents U_L,kThe corresponding target quadrilateral mesh is then selected,described by its set of 4 mesh vertices above left, below left, above right and below right, corresponding representationA left upper grid vertex as a1 st grid vertex, a left lower grid vertex as a2 nd grid vertex, a right upper grid vertex as a 3 rd grid vertex, a right lower grid vertex as a 4 th grid vertex,to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that,the symbol "| | |" is a euclidean distance solving symbol.

Similarly, the edge holding energy of the target quadrilateral mesh corresponding to all quadrilateral meshes in { R (x, y) } is calculated and recorded as Wherein e is_R,kRepresents U_R,kIs used to form a matrix of edges of all mesh vertices,(e_R,k)^Tis e_R,kTranspose of (e) ((e)_R,k)^Te_R,k)^-1Is (e)_R,k)^Te_R,kThe inverse of (a) is,to representIs used to form a matrix of edges of all mesh vertices, represents U_R,kThe corresponding target quadrilateral mesh is then selected,described by its set of 4 mesh vertices above left, below left, above right and below right, corresponding representationA left upper grid vertex as a1 st grid vertex, a left lower grid vertex as a2 nd grid vertex, a right upper grid vertex as a 3 rd grid vertex, a right lower grid vertex as a 4 th grid vertex,to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that,

a2, according toAndcalculating the total edge-preserving energy E of the target quadrilateral meshes corresponding to all quadrilateral meshes in the { L (x, y) } and the { R (x, y) } according to the total edge-preserving energy E_edge，

According to the important content selected by the user, calculating the total object control energy of target quadrilateral grids corresponding to all quadrilateral grids in the rectangular area where the important content selected by the user in { L (x, y) } and { R (x, y) } is located, and marking the total object control energy as E_object。

In this embodiment, step E of step three_objectThe calculation process of (2) is as follows:wherein,a range of a rectangular area in which important contents selected by the user are indicated,denotes a horizontal coordinate position of a mesh vertex of { L (x, y) } which is jth in the horizontal direction and ith in the vertical direction,denotes a horizontal coordinate position of a mesh vertex of { L (x, y) } which is j +1 th in the horizontal direction and i-th in the vertical direction,denotes the horizontal direction in { L (x, y) }To the horizontal coordinate position of the mesh vertex in the target quadrilateral mesh of the mesh vertex which is jth and ith in the vertical direction,denotes the horizontal coordinate position of the mesh vertex in the target quadrangular mesh of the mesh vertex of j +1 th in the horizontal direction and i th in the vertical direction in { L (x, y) },denotes a horizontal coordinate position of a mesh vertex of { R (x, y) } which is jth in the horizontal direction and ith in the vertical direction,denotes a horizontal coordinate position of a mesh vertex of { R (x, y) } which is j +1 th in the horizontal direction and i-th in the vertical direction,denotes a horizontal coordinate position of a mesh vertex in the target quadrangular mesh of a mesh vertex jth in the horizontal direction and ith in the vertical direction in { R (x, y) },denotes the horizontal coordinate position, s, of the mesh vertex in the target quadrilateral mesh of the mesh vertex in { R (x, y) } which is j +1 th in the horizontal direction and i-th in the vertical direction_xHorizontal scaling factor, s, representing user-specified important content_xCan be set according to the needs of users, and s is taken in the embodiment_x1.8, namely the adjusted important content size is 1.8 times of the original size.

According to all feature points in the rectangular areas where the important contents selected by the user are located in the { L (x, y) } and the { R (x, y) } the total depth control energy of the target quadrilateral grids corresponding to all quadrilateral grids in the rectangular areas where the important contents selected by the user are located in the { L (x, y) } and the { R (x, y) } is calculated and recorded as the total depth control energy of the target quadrilateral grids corresponding to all quadrilateral grids located in the rectangular areas where the important contents selected by the user are located in the { L (x, y) }E_depth。

In this embodiment, step E of step three_depthThe calculation process of (2) is as follows:

b1, calculating comfort level maintenance energy of all feature points in the rectangular area where the important content selected by the user is located in the { L (x, y) }, and recording the comfort level maintenance energy as E_co_mfo_rt， Wherein,representing a set formed by the serial numbers of all the characteristic points which fall in the rectangular area of the important content selected by the user after renumbering in the { L (x, y) },indicates the depth value of the p-th feature point in the L (x, y) within the rectangular area where the important content selected by the user is located,exp () represents an exponential function based on a natural base e, e is 2.71828183 …, the symbol "|" is an absolute value symbol, Z_maxDenotes the maximum depth value of { L (x, y) }, Z_minDenotes the minimum depth value of { L (x, y) }, CVZ_minA minimum comfortable viewing zone range is indicated,H_erepresenting a horizontal baseline distance between left and right viewpoints of the stereoscopic image to be processed, D representing a viewing distance between the left and right viewpoints of the stereoscopic image to be processed and the display, W_dIndicating the horizontal width, R, of the display_dRepresents the horizontal resolution of the display, taken H in this example_e65 mm, 1200 mm, W_d750 mm and R_d1920 mm, η₁Indicating a minimum comfortable viewing angle, in this example η₁＝-1o，CVZ_maxIndicating the maximum comfortable viewing zone range,η₂indicating the maximum comfortable viewing angle, in this example η₂＝1o，The parallax value representing the p-th feature point in { L (x, y) } that falls within the rectangular region in which the important content selected by the user is located also represents { d }_LThe coordinate position in (x, y) is the pixel value of the pixel point of the p-th characteristic point in the rectangular area where the important content selected by the user is located,to representThe target depth value of (2).

B2, calculating the feature retention energy of all feature points in the rectangular area where the important content selected by the user is located in the { L (x, y) }, and recording the feature retention energy as E_feature，Wherein,representing a set formed by the serial numbers of all feature points in the eight neighborhood range adjacent to the p-th feature point in the rectangular region in which the important content selected by the user is located in the { L (x, y) } region,indicating that the important content falls within the user-selected important content in { L (x, y) }The depth value of the p' th characteristic point in the eight adjacent domains adjacent to the p-th characteristic point in the rectangular region,to representThe target depth value of (2).

B3 by solving for min (E)_comfort+λ_feature×E_feature) Obtaining the optimal target depth value set of all feature points in the rectangular region where the important content selected by the user falls in { L (x, y) } and recording the optimal target depth value set as Is composed ofThe optimal target depth value; then according toAcquiring the optimal target disparity value set of all feature points in the rectangular region where the important content selected by the user is located in the { L (x, y) } and recording the optimal target disparity value set as Is composed ofThe optimal target disparity value of (a) is,wherein min () is a function of taking the minimum value，λ_featureIs E_featureIn this embodiment, take λ as the weighting parameter of_feature＝0.4。

B4, according toCalculating the total depth control energy E of target quadrilateral grids corresponding to all quadrilateral grids in the rectangular area where the important content selected by the user is located in the { L (x, y) } and the { R (x, y) } are located_depth，

Wherein the symbol "| | |" is a euclidean distance-solving symbol,indicating the p-th feature point in the { L (x, y) } that falls within the rectangular area where the important content selected by the user is located,represents the p-th feature point in the R (x, y) within the rectangular area where the important content selected by the user is located,to representThe corresponding target feature points are set to be in a same shape,to representCorresponding purposeThe characteristic points are marked on the surface of the steel plate,to representThe position of the abscissa of the bar (a),to representThe position of the abscissa of the bar (a),to representThe position of the abscissa of the bar (a),to representThe horizontal coordinate position of the X-ray detector is that i 'is more than or equal to 1 and less than or equal to 4, k' is more than or equal to 1 and less than or equal to M,to representQuadrilateral grid U_L,k'ith' mesh vertex, U_L,k'is the kth' quadrilateral mesh in { L (x, y) },to representQuadrilateral grid U_L,k' as the 1 st mesh vertex,to representQuadrilateral grid U_L,k' as the 2 nd mesh vertex,to representQuadrilateral grid U_L,k' as the top-right mesh vertex of the 3 rd mesh vertex,to representQuadrilateral grid U_L,k' as the lower right mesh vertex of the 4 th mesh vertex,to representAndthe euclidean distance between them,to representTarget quadrilateral meshThe ith' mesh vertex of (a),is U_L,k' a corresponding target quadrilateral mesh,to representTarget quadrilateral meshAs the 1 st mesh vertex,to representTarget quadrilateral meshAs the 2 nd mesh vertex,to representTarget quadrilateral meshAs the top-right mesh vertex of the 3 rd mesh vertex,to representTarget quadrilateral meshAs the 4 th mesh vertex,to representQuadrilateral grid U_R,k'ith' mesh vertex, U_R,k'is the kth' quadrilateral mesh in { R (x, y) },to representQuadrilateral grid U_R,k' as the 1 st mesh vertex,to representQuadrilateral grid U_R,k' as the 2 nd mesh vertex,to representQuadrilateral grid U_R,k' as the top-right mesh vertex of the 3 rd mesh vertex,to representQuadrilateral grid U_R,k' as the lower right mesh vertex of the 4 th mesh vertex,to representAndthe euclidean distance between them,to representTarget quadrilateral meshThe ith' mesh vertex of (a),is U_R,k' a corresponding target quadrilateral mesh,representing characteristic pointsTarget quadrilateral meshAs the 1 st mesh vertex,to representTarget quadrilateral meshAs the 2 nd mesh vertex,to representTarget quadrilateral meshAs the top-right mesh vertex of the 3 rd mesh vertex,representing characteristic pointsTarget quadrilateral meshAs the 4 th mesh vertex.

Calculating the total background holding energy of the target quadrilateral grids corresponding to all quadrilateral grids in the { L (x, y) } and the { R (x, y) } which fall in the background area, and recording the total background holding energy as E according to all the feature points in the { L (x, y) } and the corresponding target feature points, and all the feature points in the { R (x, y) } and the corresponding target feature points_back(ii) a The background area is an area except the rectangular area where the important content selected by the user is located.

In this embodiment, step E of step three_backThe calculation process of (2) is as follows:

wherein,representing a set of sequence numbers formed by renumbering all feature points falling in the background area in { L (x, y) } and { R (x, y) },represents the g-th feature point falling within the background region in { L (x, y) },represents the g-th feature point falling within the background region in { R (x, y) },to representThe corresponding target feature points are set to be in a same shape,to representCorresponding target characteristic points, the symbol "| | |" is a Euclidean distance-calculating symbol, i' is more than or equal to 1 and less than or equal to 4, k is more than or equal to 1 and less than or equal to M,to representQuadrilateral grid U_L,k"ith' mesh vertex, U_L,k"is the kth" quadrilateral mesh in { L (x, y) },to representQuadrilateral grid U_L,k"left upper grid vertex as the 1 st grid vertex,to representQuadrilateral grid U_L,k"as the 2 nd mesh vertex,to representQuadrilateral grid U_L,k"top right mesh vertex as the 3 rd mesh vertex,to representQuadrilateral grid U_L,k"as the lower right mesh vertex of the 4 th mesh vertex,to representAndthe euclidean distance between them,to representTarget quadrilateral netGrid (C)The ith' mesh vertex of (a),is U_L,k"the corresponding target quadrilateral mesh,to representTarget quadrilateral meshAs the 1 st mesh vertex,to representTarget quadrilateral meshAs the 2 nd mesh vertex,to representTarget quadrilateral meshAs the top-right mesh vertex of the 3 rd mesh vertex,to representTarget quadrilateral meshAs the 4 th mesh vertex,to representQuadrilateral grid U_R,k"ith' mesh vertex, U_R,k"is the kth" quadrilateral mesh in { R (x, y) },to representQuadrilateral grid U_R,k"left upper grid vertex as the 1 st grid vertex,to representQuadrilateral grid U_R,k"as the 2 nd mesh vertex,to representQuadrilateral grid U_R,k"top right mesh vertex as the 3 rd mesh vertex,to representQuadrilateral grid U_R,k"as the lower right mesh vertex of the 4 th mesh vertex,to representAndthe euclidean distance between them,to representTarget quadrilateral meshThe ith' mesh vertex of (a),is U_R,k"the corresponding target quadrilateral mesh,to representTarget quadrilateral meshAs the 1 st mesh vertex,representing characteristic pointsTarget quadrilateral meshAs the 2 nd mesh vertex,representing characteristic pointsTarget quadrilateral meshAs the top-right mesh vertex of the 3 rd mesh vertex,representing characteristic pointsTarget quadrilateral meshAs the 4 th mesh vertex.

Step four: according to E_edge、E_object、E_depthAnd E_backCalculating the total energy of the target quadrilateral grids corresponding to all the quadrilateral grids in the { L (x, y) } and the { R (x, y) }, and recording the total energy as E_total，E_total＝E_edge+λ_object×E_object+λ_depth×E_depth+λ_back×E_back(ii) a Then solving by least squares optimizationObtaining a set formed by optimal target quadrilateral grids corresponding to all quadrilateral grids in the { L (x, y) } and optimal target quadrilateral grids corresponding to all quadrilateral grids in the { R (x, y) }Set of best target quadrilateral meshes, correspondence being notedAnd then according toCalculating the optimal similarity transformation matrix of the optimal target quadrilateral grids corresponding to each quadrilateral grid in the { L (x, y) }, and converting U into U_L,kCorresponding optimal target quadrilateral meshIs recorded as the optimal similarity transformation matrix And according toCalculating the optimal similarity transformation matrix of the optimal target quadrilateral grids corresponding to each quadrilateral grid in the { R (x, y) }, and converting U into U_R,kCorresponding optimal target quadrilateral meshIs recorded as the optimal similarity transformation matrix Wherein λ is_objectIs E_objectWeighting ofParameter, λ_depthIs E_depthA weighting parameter of_backIs E_backIn this embodiment, take λ as the weighting parameter of_object＝4、λ_depth＝8、λ_backMin () is the take minimum function, 4,represents a set of target quadrilateral meshes corresponding to all quadrilateral meshes in the { L (x, y) },represents a set of target quadrilateral meshes corresponding to all quadrilateral meshes in R (x, y),represents U_L,kThe corresponding optimal target quadrilateral mesh is selected from the set of target quadrilateral meshes,described by its set of 4 mesh vertices above left, below left, above right and below right, corresponding representation1 st mesh vertex, 2 nd mesh vertex, 3 rd mesh vertex, 4 th mesh vertex,represents U_R,kThe corresponding optimal target quadrilateral mesh is selected from the set of target quadrilateral meshes,described by its set of 4 mesh vertices above left, below left, above right and below right, corresponding representation1 st mesh vertex, 2 nd mesh vertex, 3 rd mesh vertex, 4 th mesh vertex of (a)_L,k)^TIs A_L,kTranspose of (A) ((A)_L,k)^TA_L,k)^-1Is (A)_L,k)^TA_L,kThe inverse of (a) is,andcorresponding representationA horizontal coordinate position and a vertical coordinate position of,andcorresponding representationA horizontal coordinate position and a vertical coordinate position of,andcorresponding representationA horizontal coordinate position and a vertical coordinate position of,andcorresponding representation(ii) a horizontal coordinate position and a vertical coordinate position of (A)_R,k)^TIs A_R,kTranspose of (A) ((A)_R,k)^TA_R,k)^-1Is (A)_R,k)^TA_R,kThe inverse of (a) is,andcorresponding representationA horizontal coordinate position and a vertical coordinate position of,andcorresponding representationA horizontal coordinate position and a vertical coordinate position of,andcorresponding representationA horizontal coordinate position and a vertical coordinate position of,andcorresponding representationA horizontal coordinate position and a vertical coordinate position.

Step five: according to the optimal similarity transformation matrix of the optimal target quadrilateral grid corresponding to each quadrilateral grid in the { L (x, y) }, calculating the horizontal coordinate position and the vertical coordinate position of each pixel point in each quadrilateral grid in the { L (x, y) } after the optimal similarity transformation rectangular transformation, and converting the U into the U-shaped coordinate position_L,kThe position of the middle horizontal coordinate is x'_L,kAnd vertical coordinate position y'_L,kThe correspondence of the horizontal coordinate position and the vertical coordinate position of the pixel point after the optimal similarity transformation matrix is recorded asAnd then, according to the horizontal coordinate position and the vertical coordinate position of each pixel point in each quadrilateral grid in the { L (x, y) } after the optimal similarity transformation rectangular transformation, a left viewpoint image after depth adjustment is obtained and recorded as a left viewpoint image after the depth adjustmentWherein x is not less than 1'_L,k≤W，1≤y'_L,k≤H，X 'is more than or equal to 1 and less than or equal to W', y 'is more than or equal to 1 and less than or equal to H, W' represents the width of the three-dimensional image after the depth adjustment, H is the height of the three-dimensional image after the depth adjustment,to representAnd the pixel value of the pixel point with the middle coordinate position of (x ', y').

Similarly, according to the optimal similarity transformation matrix of the optimal target quadrilateral grid corresponding to each quadrilateral grid in the { R (x, y) }, calculating the horizontal coordinate position and the vertical coordinate position of each pixel point in each quadrilateral grid in the { R (x, y) } after the optimal similarity transformation rectangular transformation, and converting the U into the U shape_R,kThe position of the middle horizontal coordinate is x'_R,kAnd vertical coordinate position y'_R,kThe correspondence of the horizontal coordinate position and the vertical coordinate position of the pixel point after the optimal similarity transformation matrix is recorded asAnd then, according to the horizontal coordinate position and the vertical coordinate position of each pixel point in each quadrilateral grid in the { R (x, y) } after the optimal similarity transformation rectangular transformation, a right viewpoint image after depth adjustment is obtained and recorded as a right viewpoint imageWherein x is not less than 1'_R,k≤W，1≤y'_R,k≤H，1≤x'≤W'，1≤y'≤H，To representAnd the pixel value of the pixel point with the middle coordinate position of (x ', y').

To further illustrate the feasibility and effectiveness of the method of the present invention, the method of the present invention was tested.

Next, the subject depth adjustment experiment was performed on four stereoscopic images, Image1, Image2, Image3, and Image4, by the method of the present invention. FIG. 2a shows a "red/green" map of the original stereo Image of "Image 1", and FIG. 2b shows a "red/green" map of "Image 1" after object depth adjustment; FIG. 3a shows a "red/green" map of the original stereo Image of "Image 2", and FIG. 3b shows a "red/green" map of "Image 2" after object depth adjustment; FIG. 4a shows a "red/green" map of the original stereo Image of "Image 3", and FIG. 4b shows a "red/green" map of "Image 3" after object depth adjustment; fig. 5a shows a "red/green" map of the original stereoscopic Image of "Image 4", and fig. 5b shows a "red/green" map of "Image 4" after object depth adjustment. As can be seen from fig. 2a to 5b, the depth-adjusted stereoscopic image obtained by the method of the present invention can better retain the object shape, and the size of the important object can be increased according to the selection of the user.

Claims (1)

1. A method for adjusting depth of a stereoscopic image based on an object is characterized by comprising the following steps:

the method comprises the following steps: the left viewpoint image, the right viewpoint image, and the left parallax image of the stereoscopic image having the width W and the height H to be processed are correspondingly denoted as { L (x, y) }, { R (x, y) }, and { d_L(x, y) }; wherein x is more than or equal to 1 and less than or equal to W, y is more than or equal to 1 and less than or equal to H, W and H can be evenly divided by 8, L (x, y) represents the pixel value of the pixel point with the coordinate position (x, y) in { L (x, y) }, R (x, y) represents the pixel value of the pixel point with the coordinate position (x, y) in { R (x, y) },d_L(x, y) represents { d }_LThe coordinate position in (x, y) is the pixel value of the pixel point of (x, y);

step two: dividing { L (x, y) } intoFour-sided grids of 8 × 8 size and not overlapping each other, and the kth four-sided grid in { L (x, y) } is marked as U_L,kAnd the set of all quadrilateral meshes in { L (x, y) } is denoted as V_L，V_L＝{U_L,kL 1 is more than or equal to k and less than or equal to M }; then according to all quadrilateral meshes in { L (x, y) } and { d_L(x, y) }, acquiring all non-overlapping quadrilateral grids with the size of 8 multiplied by 8 in the { R (x, y) }, and marking the kth quadrilateral grid in the { R (x, y) }asU_R,kAnd the set of all quadrilateral meshes in { R (x, y) } is denoted as V_R，V_R＝{U_R,kL 1 is more than or equal to k and less than or equal to M }; wherein k is a positive integer, k is not less than 1 and not more than M, M represents the total number of quadrilateral meshes contained in { L (x, y) } and also represents the total number of quadrilateral meshes contained in { R (x, y) },U_L,kdescribed by its set of 4 mesh vertices above left, below left, above right and below right, corresponds to and represents U_L,kA left upper grid vertex as a1 st grid vertex, a left lower grid vertex as a2 nd grid vertex, a right upper grid vertex as a 3 rd grid vertex, a right lower grid vertex as a 4 th grid vertex,to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that,U_R,kdescribed by its set of 4 mesh vertices above left, below left, above right and below right, corresponds to and represents U_R,kA left upper grid vertex as a1 st grid vertex, a left lower grid vertex as a2 nd grid vertex, a right upper grid vertex as a 3 rd grid vertex, a right lower grid vertex as a 4 th grid vertex,to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that,represents { d_L(x, y) } coordinate position ofThe pixel value of the pixel point of (a),to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, represents { d_L(x, y) } coordinate position ofThe pixel value of the pixel point of (a),to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, represents { d_L(x, y) } coordinate position ofThe pixel value of the pixel point of (a),to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, represents { d_L(x, y) } coordinate position ofThe pixel value of the pixel point of (1);

extracting all feature points in the { L (x, y) } by adopting scale-invariant feature conversion; then, the q-th feature point in { L (x, y) } is noted as Then according to each feature point in { L (x, y) } and { d }_L(x, y) }, acquiring feature points in the { R (x, y) } which are matched with each feature point in the { L (x, y) }, and matching the feature points in the { R (x, y) } with the feature points in the { L (x, y) }The matched feature points are recorded as Wherein Q is a positive integer, 1. ltoreq. q.ltoreq.Q, Q represents the total number of feature points in { L (x, y) }, and also represents the total number of feature points in { R (x, y) },to representThe position of the abscissa of the bar (a),to representThe position of the ordinate of (a) is,to representThe position of the abscissa of the bar (a),represents { d_L(x, y) } coordinate position ofThe pixel value of the pixel point of (a),to representThe position of the ordinate of (a) is,

step three: calculating the total edge retention energy of the target quadrilateral meshes corresponding to all quadrilateral meshes in the { L (x, y) } and the { R (x, y) } according to a matrix formed by the edges of all grid vertexes of each quadrilateral mesh in the { L (x, y) } and a matrix formed by the edges of all grid vertexes of the target quadrilateral mesh corresponding to each quadrilateral mesh in the { L (x, y) }, a matrix formed by the edges of all grid vertexes of each quadrilateral mesh in the { R (x, y) } and a matrix formed by the edges of all grid vertexes of the target quadrilateral mesh corresponding to each quadrilateral mesh in the { R (x, y) }, and recording the total edge retention energy as E_edge；

According to the important content selected by the user, calculating the total object control energy of target quadrilateral grids corresponding to all quadrilateral grids in the rectangular area where the important content selected by the user in { L (x, y) } and { R (x, y) } is located, and marking the total object control energy as E_object；

Calculating { L (x, y) } and { R (x, y) } according to all feature points in the { L (x, y) } and { R (x, y) } which fall in the rectangular area where the important content selected by the user is located{ R (x, y) } total depth control energy of target quadrilateral meshes corresponding to all quadrilateral meshes in the rectangular area where the important content selected by the user is located, and recorded as E_depth；

Calculating the total background holding energy of the target quadrilateral grids corresponding to all quadrilateral grids in the { L (x, y) } and the { R (x, y) } which fall in the background area, and recording the total background holding energy as E according to all the feature points in the { L (x, y) } and the corresponding target feature points, and all the feature points in the { R (x, y) } and the corresponding target feature points_back(ii) a The background area is an area except a rectangular area where the important content selected by the user is located;

step four: according to E_edge、E_object、E_depthAnd E_backCalculating the total energy of the target quadrilateral grids corresponding to all the quadrilateral grids in the { L (x, y) } and the { R (x, y) }, and recording the total energy as E_total，E_total＝E_edge+λ_object×E_object+λ_depth×E_depth+λ_back×E_back(ii) a Then solving by least squares optimizationObtaining a set formed by the optimal target quadrilateral grids corresponding to all quadrilateral grids in the { L (x, y) } and a set formed by the optimal target quadrilateral grids corresponding to all quadrilateral grids in the { R (x, y) }, and recording the optimal target quadrilateral grids corresponding to all quadrilateral grids in the { L (x, y) }asthe optimal target quadrilateral grids corresponding to all quadrilateral grids in the { R (x, y) } correspondinglyAnd then according toCalculate each of { L (x, y) }The optimal similarity transformation matrix of the optimal target quadrilateral grid corresponding to each quadrilateral grid is obtained by transforming U_L,kCorresponding optimal target quadrilateral meshIs recorded as the optimal similarity transformation matrix And according toCalculating the optimal similarity transformation matrix of the optimal target quadrilateral grids corresponding to each quadrilateral grid in the { R (x, y) }, and converting U into U_R,kCorresponding optimal target quadrilateral meshIs recorded as the optimal similarity transformation matrix Wherein λ is_objectIs E_objectA weighting parameter of_depthIs E_depthA weighting parameter of_backIs E_backMin () is a function taking the minimum value,represents a set of target quadrilateral meshes corresponding to all quadrilateral meshes in the { L (x, y) },denotes ones in { R (x, y) }A set of target quadrilateral meshes corresponding to all quadrilateral meshes,represents U_L,kThe corresponding optimal target quadrilateral mesh is selected from the set of target quadrilateral meshes, corresponding representation1 st mesh vertex, 2 nd mesh vertex, 3 rd mesh vertex, 4 th mesh vertex,represents U_R,kThe corresponding optimal target quadrilateral mesh is selected from the set of target quadrilateral meshes, corresponding representation1 st mesh vertex, 2 nd mesh vertex, 3 rd mesh vertex, 4 th mesh vertex of (a)_L,k)^TIs A_L,kTranspose of (A) ((A)_L,k)^TA_L,k)^-1Is (A)_L,k)^TA_L,kThe inverse of (a) is,andcorresponding representationA horizontal coordinate position and a vertical coordinate position of,andcorresponding representationA horizontal coordinate position and a vertical coordinate position of,andcorresponding representationA horizontal coordinate position and a vertical coordinate position of,andcorresponding representation(ii) a horizontal coordinate position and a vertical coordinate position of (A)_R,k)^TIs A_R,kTranspose of (A) ((A)_R,k)^TA_R,k)^-1Is (A)_R,k)^TA_R,kThe inverse of (a) is,andcorresponding representationA horizontal coordinate position and a vertical coordinate position of,andcorresponding representationA horizontal coordinate position and a vertical coordinate position of,andcorresponding representationA horizontal coordinate position and a vertical coordinate position of,andcorresponding representationHorizontal coordinate position and vertical coordinate position of (a);

step five: according to { L (x, y)Calculating the horizontal coordinate position and the vertical coordinate position of each pixel point in each quadrilateral grid in the (L (x, y)) after the rectangular transformation of the optimal similarity, and converting the U into the optimal similarity of the optimal target quadrilateral grid corresponding to each quadrilateral grid in the (L, y) matrix_L,kThe position of the middle horizontal coordinate is x'_L,kAnd vertical coordinate position y'_L,kThe correspondence of the horizontal coordinate position and the vertical coordinate position of the pixel point after the optimal similarity transformation matrix is recorded asAnd then, according to the horizontal coordinate position and the vertical coordinate position of each pixel point in each quadrilateral grid in the { L (x, y) } after the optimal similarity transformation rectangular transformation, a left viewpoint image after depth adjustment is obtained and recorded as a left viewpoint image after the depth adjustmentWherein x is not less than 1'_L,k≤W，1≤y'_L,k≤H，X 'is more than or equal to 1 and less than or equal to W', y 'is more than or equal to 1 and less than or equal to H, W' represents the width of the three-dimensional image after the depth adjustment, H is the height of the three-dimensional image after the depth adjustment,to representThe pixel value of the pixel point with the middle coordinate position (x ', y');

also, an optimal target corresponding to each quadrilateral mesh in { R (x, y) } is setCalculating the horizontal coordinate position and the vertical coordinate position of each pixel point in each quadrilateral grid in the { R (x, y) } after the rectangular transformation of the optimal similarity transformation matrix, and converting U_R,kThe position of the middle horizontal coordinate is x'_R,kAnd vertical coordinate position y'_R,kThe correspondence of the horizontal coordinate position and the vertical coordinate position of the pixel point after the optimal similarity transformation matrix is recorded asAnd then, according to the horizontal coordinate position and the vertical coordinate position of each pixel point in each quadrilateral grid in the { R (x, y) } after the optimal similarity transformation rectangular transformation, a right viewpoint image after depth adjustment is obtained and recorded as a right viewpoint imageWherein x is not less than 1'_R,k≤W，1≤y'_R,k≤H，1≤x'≤W'，1≤y'≤H，To representThe pixel value of the pixel point with the middle coordinate position (x ', y');

e in the third step_edgeThe calculation process of (2) is as follows:

a1, calculating the edge holding energy of the target quadrilateral grids corresponding to all quadrilateral grids in the { L (x, y) }, and recording the edge holding energy as Wherein e is_L,kRepresents U_L,kIs used to form a matrix of edges of all mesh vertices,(e_L,k)^Tis e_L,kTranspose of (e) ((e)_L,k)^Te_L,k)^-1Is (e)_L,k)^Te_L,kThe inverse of (a) is,to representIs used to form a matrix of edges of all mesh vertices, represents U_L,kThe corresponding target quadrilateral mesh is then selected,described by its set of 4 mesh vertices above left, below left, above right and below right, corresponding representationA left upper grid vertex as a1 st grid vertex, a left lower grid vertex as a2 nd grid vertex, a right upper grid vertex as a 3 rd grid vertex, a right lower grid vertex as a 4 th grid vertex,to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that,the symbol "| | |" is a euclidean distance solving symbol;

similarly, the edge holding energy of the target quadrilateral mesh corresponding to all quadrilateral meshes in { R (x, y) } is calculated and recorded as Wherein e is_R,kRepresents U_R,kOf all mesh verticesThe number of the arrays is determined,(e_R,k)^Tis e_R,kTranspose of (e) ((e)_R,k)^Te_R,k)^-1Is (e)_R,k)^Te_R,kThe inverse of (a) is,to representIs used to form a matrix of edges of all mesh vertices, represents U_R,kThe corresponding target quadrilateral mesh is then selected,described by its set of 4 mesh vertices above left, below left, above right and below right, corresponding representationA left upper grid vertex as a1 st grid vertex, a left lower grid vertex as a2 nd grid vertex, a right upper grid vertex as a 3 rd grid vertex, a right lower grid vertex as a 4 th grid vertex,to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that, to be provided withHorizontal coordinate position ofAnd vertical coordinate positionTo be described, the method has the advantages that,

a2, according toAndcalculating the total edge-preserving energy E of the target quadrilateral meshes corresponding to all quadrilateral meshes in the { L (x, y) } and the { R (x, y) } according to the total edge-preserving energy E_edge，

E in the third step_depthThe calculation process of (2) is as follows:

b1, calculating comfort level maintenance energy of all feature points in the rectangular area where the important content selected by the user is located in the { L (x, y) }, and recording the comfort level maintenance energy as E_comfort， Wherein,representing a set formed by the serial numbers of all the characteristic points which fall in the rectangular area of the important content selected by the user after renumbering in the { L (x, y) },indicates the depth value of the p-th feature point in the L (x, y) within the rectangular area where the important content selected by the user is located,exp () represents an exponential function with the natural base e as the base, the symbol "|" is an absolute value symbol, Z_maxDenotes the maximum depth value of { L (x, y) }, Z_minDenotes the minimum depth value of { L (x, y) }, CVZ_minA minimum comfortable viewing zone range is indicated,H_erepresenting a horizontal baseline distance between left and right viewpoints of the stereoscopic image to be processed, D representing a viewing distance between the left and right viewpoints of the stereoscopic image to be processed and the display, W_dIndicating the horizontal width, R, of the display_dIndicating the horizontal resolution of the display, η₁Indicating minimum comfortable viewing angle, CVZ_maxIndicating the maximum comfortable viewing zone range,η₂indicating the maximum comfortable viewing angle of view,the parallax value representing the p-th feature point in { L (x, y) } that falls within the rectangular region in which the important content selected by the user is located also represents { d }_LThe coordinate position in (x, y) is the pixel value of the pixel point of the p-th characteristic point in the rectangular area where the important content selected by the user is located,to representThe target depth value of (2);

b2, calculating the feature retention energy of all feature points in the rectangular area where the important content selected by the user is located in the { L (x, y) }, and recording the feature retention energy as E_feature，Wherein,representing a set formed by the serial numbers of all feature points in the eight neighborhood range adjacent to the p-th feature point in the rectangular region in which the important content selected by the user is located in the { L (x, y) } region,indicating the depth value of the p' th characteristic point in the eight neighborhood range adjacent to the p-th characteristic point in the rectangular area in which the important content selected by the user is located in the L (x, y),to representThe target depth value of (2);

b3 by solving for min: (E_comfort+λ_feature×E_feature) Obtaining the optimal target depth value set of all feature points in the rectangular region where the important content selected by the user falls in { L (x, y) } and recording the optimal target depth value set as Is composed ofThe optimal target depth value; then according toAcquiring the optimal target disparity value set of all feature points in the rectangular region where the important content selected by the user is located in the { L (x, y) } and recording the optimal target disparity value set as Is composed ofThe optimal target disparity value of (a) is,wherein min () is a function of taking the minimum value, λ_featureIs E_featureThe weighting parameter of (2);

b4, according toCalculating the total of target quadrilateral grids corresponding to all quadrilateral grids in the rectangular area where the important content selected by the user is located in the { L (x, y) } and the { R (x, y) }Depth control energy E_depth， Wherein the symbol "| | |" is a euclidean distance-solving symbol,indicating the p-th feature point in the { L (x, y) } that falls within the rectangular area where the important content selected by the user is located,represents the p-th feature point in the R (x, y) within the rectangular area where the important content selected by the user is located,to representThe corresponding target feature points are set to be in a same shape,to representThe corresponding target feature points are set to be in a same shape,to representThe position of the abscissa of the bar (a),to representThe position of the abscissa of the bar (a),to representThe position of the abscissa of the bar (a),to representThe horizontal coordinate position of the X-ray detector is that i 'is more than or equal to 1 and less than or equal to 4, k' is more than or equal to 1 and less than or equal to M,to representQuadrilateral grid U_L,k'The ith' mesh vertex of (1), U_L,k'Is the k' th quadrilateral mesh in { L (x, y) },to representQuadrilateral grid U_L,k'As the 1 st mesh vertex,to representQuadrilateral grid U_L,k'As a firstThe lower left mesh vertex of the 2 mesh vertices,to representQuadrilateral grid U_L,k'As the top-right mesh vertex of the 3 rd mesh vertex,to representQuadrilateral grid U_L,k'As the 4 th mesh vertex,to representAndthe euclidean distance between them,to representTarget quadrilateral meshThe ith' mesh vertex of (a),is U_L,k'The corresponding target quadrilateral mesh is then selected,to representTarget quadrilateral meshAs the 1 st mesh vertex,to representTarget quadrilateral meshAs the 2 nd mesh vertex,to representTarget quadrilateral meshAs the top-right mesh vertex of the 3 rd mesh vertex,to representTarget quadrilateral meshAs the 4 th mesh vertex,to representQuadrilateral grid U_R,k'The ith' mesh vertex of (1), U_R,k'Is the k' th quadrilateral mesh in { R (x, y) },to representQuadrilateral grid U_R,k'As the 1 st mesh vertex,to representQuadrilateral grid U_R,k'As the 2 nd mesh vertex,to representQuadrilateral grid U_R,k'As the top-right mesh vertex of the 3 rd mesh vertex,to representQuadrilateral grid U_R,k'As the 4 th mesh vertex,to representAndthe euclidean distance between them,to representTarget quadrilateral meshThe ith' mesh vertex of (a),is U_R,k'The corresponding target quadrilateral mesh is then selected,representing characteristic pointsTarget quadrilateral meshAs the 1 st mesh vertex,to representTarget quadrilateral meshAs the 2 nd mesh vertex,to representTarget quadrilateral meshAs the top-right mesh vertex of the 3 rd mesh vertex,representing characteristic pointsTarget quadrilateral meshAs the 4 th mesh vertex, a lower right mesh vertex;

e in the third step_backThe calculation process of (2) is as follows: wherein,indicating that the { L (x, y) } and { R (x, y) } fall in the background regionA set of renumbered sequence numbers for all feature points in the domain,represents the g-th feature point falling within the background region in { L (x, y) },represents the g-th feature point falling within the background region in { R (x, y) },to representThe corresponding target feature points are set to be in a same shape,to representCorresponding target characteristic points, the symbol "| | |" is a Euclidean distance-calculating symbol, i' is more than or equal to 1 and less than or equal to 4, k is more than or equal to 1 and less than or equal to M,to representQuadrilateral grid U_L,k”The ith' mesh vertex of (1), U_L,k”Is the k-th quadrilateral mesh in { L (x, y) },to representQuadrilateral grid U_L,k”As the 1 st mesh vertexThe left-hand upper grid vertex of (a),to representQuadrilateral grid U_L,k”As the 2 nd mesh vertex,to representQuadrilateral grid U_L,k”As the top-right mesh vertex of the 3 rd mesh vertex,to representQuadrilateral grid U_L,k”As the 4 th mesh vertex,to representAndthe euclidean distance between them,to representTarget quadrilateral meshThe ith' mesh vertex of (a),is U_L,k”The corresponding target quadrilateral mesh is then selected,to representTarget quadrilateral meshAs the 1 st mesh vertex,to representTarget quadrilateral meshAs the 2 nd mesh vertex,to representTarget quadrilateral meshAs the top-right mesh vertex of the 3 rd mesh vertex,to representTarget quadrilateral meshAs the 4 th mesh vertex,to representQuadrilateral grid U_R,k”The ith' mesh vertex of (1), U_R,k”Is the kth quadrilateral mesh in { R (x, y) },to representQuadrilateral grid U_R,k”As the 1 st mesh vertex,to representQuadrilateral grid U_R,k”As the 2 nd mesh vertex,to representQuadrilateral grid U_R,k”As the top-right mesh vertex of the 3 rd mesh vertex,to representQuadrilateral grid U_R,k”As the 4 th mesh vertex,to representAndthe euclidean distance between them,to representTarget quadrilateral meshThe ith' mesh vertex of (a),is U_R,k”The corresponding target quadrilateral mesh is then selected,to representTarget quadrangle in whichGrid meshAs the 1 st mesh vertex,representing characteristic pointsTarget quadrilateral meshAs the 2 nd mesh vertex,representing characteristic pointsTarget quadrilateral meshAs the top-right mesh vertex of the 3 rd mesh vertex,representing characteristic pointsTarget quadrilateral meshAs the 4 th mesh vertex, a lower right mesh vertex;

e in the third step_objectThe calculation process of (2) is as follows:

wherein,a range of a rectangular area in which important contents selected by the user are indicated,denotes a horizontal coordinate position of a mesh vertex of { L (x, y) } which is jth in the horizontal direction and ith in the vertical direction,denotes a horizontal coordinate position of a mesh vertex of { L (x, y) } which is j +1 th in the horizontal direction and i-th in the vertical direction,denotes a horizontal coordinate position of a mesh vertex in the target quadrangular mesh, of a mesh vertex which is jth in the horizontal direction and ith in the vertical direction in { L (x, y) },denotes the horizontal coordinate position of the mesh vertex in the target quadrangular mesh of the mesh vertex of j +1 th in the horizontal direction and i th in the vertical direction in { L (x, y) },denotes a horizontal coordinate position of a mesh vertex of { R (x, y) } which is jth in the horizontal direction and ith in the vertical direction,denotes a horizontal coordinate position of a mesh vertex of { R (x, y) } which is j +1 th in the horizontal direction and i-th in the vertical direction,denotes a horizontal coordinate position of a mesh vertex in the target quadrangular mesh of a mesh vertex jth in the horizontal direction and ith in the vertical direction in { R (x, y) },denotes the horizontal coordinate position, s, of the mesh vertex in the target quadrilateral mesh of the mesh vertex in { R (x, y) } which is j +1 th in the horizontal direction and i-th in the vertical direction_xA horizontal scaling factor representing user-specified important content.