CN107845061A

CN107845061A - Image processing method, device and terminal

Info

Publication number: CN107845061A
Application number: CN201711107117.XA
Authority: CN
Inventors: 袁雨龙; 黄文辉
Original assignee: Storm Group Ltd By Share Ltd
Current assignee: Storm Group Ltd By Share Ltd
Priority date: 2017-11-10
Filing date: 2017-11-10
Publication date: 2018-03-27

Abstract

The invention discloses a kind of image processing method, device and terminal.Methods described includes：Determine the pending partial sphere figure on spherical panoramic image；The corresponding surface pending partial sphere figure being mapped in default polyhedral structure, obtains the plane regional area in the corresponding surface；Default processing is carried out to the plane regional area；Plane regional area after processing is mapped to the region on the spherical panoramic image corresponding to the pending partial sphere figure.Pass through technical scheme, the spherical diagram of pending part in spherical panoramic image can be mapped to plan, after handling plan, the plan after processing is remapped to respective regions on sphere, realizes the processing to spherical panoramic image upper part spherical diagram picture.

Description

Image processing method and device and terminal

Technical Field

The present invention relates to the field of image processing technologies, and in particular, to an image processing method, an image processing apparatus, and a terminal.

Background

At present, with the rapid development of internet VR (Virtual Reality) videos, there are increasing demands for panoramic videos, and sometimes a part of contents of a spherical panorama needs to be processed, and a processing algorithm in the related art is often based on a plane rather than a spherical curvature, for example, an existing popular AR (Augmented Reality) tracking technology is to find an image to be processed in a planar picture and then perform image replacement and other processing, so how to process a part of images on the spherical panorama by using a processing algorithm in the related art becomes an urgent problem to be solved.

Disclosure of Invention

In view of this, the technical problem to be solved by the present invention is to provide an image processing method, an apparatus and a terminal, which can map a spherical image of a portion to be processed in a spherical panorama into a planar image, process the planar image, and then re-map the processed planar image into a corresponding area on the spherical surface, so as to implement processing of a portion of the spherical image in the spherical panorama.

The invention provides an image processing method, which comprises the following steps:

determining a local spherical image to be processed on the spherical panoramic image;

mapping the local spherical graph to be processed to a corresponding surface in a preset polyhedral structure to obtain a plane local area in the corresponding surface;

presetting the plane local area;

and mapping the processed plane local area to an area corresponding to the local spherical image to be processed on the spherical panoramic image.

In one embodiment, the mapping the local spherical map to be processed to a corresponding face in a preset polyhedral structure to obtain a planar local area in the corresponding face includes:

acquiring the latitude and longitude range of the local spherical image to be processed;

according to the longitude and latitude range, determining the corresponding surface corresponding to the longitude and latitude range and a mapping candidate area in the corresponding surface from the polyhedral structure;

and mapping the local spherical map to be processed to the mapping candidate region to obtain the plane local region.

In one embodiment, the determining the mapping candidate region in the corresponding plane according to the latitude and longitude range includes:

determining the field angle of the plane local area according to the latitude and longitude range;

and selecting the mapping candidate region from the corresponding surface according to the opening angle.

In one embodiment, the mapping the to-be-processed local spherical map to the mapping candidate region to obtain the planar local region includes:

determining plane coordinate values of each point on the mapping candidate area;

determining the three-dimensional coordinate value of each point in the space coordinate system of the spherical panoramic image according to the plane coordinate value of each point;

acquiring longitude and latitude values of corresponding points of the points on the mapping candidate area on the spherical panoramic image according to the three-dimensional coordinate values of the points in the space;

and drawing a plane image corresponding to the local spherical image to be processed on the mapping candidate area according to the plane coordinate value of each point on the mapping candidate area and the longitude and latitude value of the corresponding point on the spherical panoramic image so as to obtain the plane local area.

In one embodiment, the method further comprises:

and normalizing the plane coordinate values of all points on the mapping candidate area to obtain the normalized plane coordinate values of all points on the mapping candidate area.

In one embodiment, the determining the three-dimensional coordinate values of the points in the spatial coordinate system of the spherical panorama according to the plane coordinate values of the points includes:

determining a preset rotation angle of the mapping candidate area according to the latitude and longitude range;

and when the initial rotation angle of the mapping candidate region is not matched with the preset rotation angle, respectively determining the three-dimensional coordinate values of each point on the mapping candidate region before and after rotation based on the plane coordinate values of each point after the mapping candidate region is rotated.

In one embodiment, the drawing, according to the plane coordinate values of the points on the mapping candidate region and the longitude and latitude values of the corresponding points on the spherical panorama, a plane image corresponding to the local spherical map to be processed on the mapping candidate region to obtain the plane local region includes:

determining the plane coordinate value of each point on the mapping candidate area and the longitude and latitude value of the corresponding point of the spherical panorama as a vertex coordinate and a texture coordinate in opengl respectively;

and drawing the plane image on the mapping candidate region according to the vertex coordinate and the texture coordinate in the opengl so as to obtain the plane local region.

In one embodiment, the method further comprises:

when the processed plane local area is mapped to an area corresponding to the local spherical image to be processed on the spherical panoramic image, the plane coordinate values of all points on the plane local area and the longitude and latitude values of corresponding points of the spherical panoramic image are respectively determined as texture coordinates and vertex coordinates in opengl.

The present invention also provides an image processing apparatus comprising:

the first determining module is used for determining a local spherical image to be processed on the spherical panoramic image;

the first mapping module is used for mapping the local spherical graph to be processed to a corresponding surface in a preset polyhedral structure to obtain a plane local area in the corresponding surface;

the first processing module is used for carrying out preset processing on the plane local area;

and the second mapping module is used for mapping the processed plane local area to an area corresponding to the local spherical image to be processed on the spherical panoramic image.

In one embodiment, the first mapping module comprises:

the acquisition submodule is used for acquiring the latitude and longitude range of the local spherical image to be processed;

the determining submodule is used for determining the corresponding surface corresponding to the latitude and longitude range and the mapping candidate area in the corresponding surface from the polyhedral structure according to the latitude and longitude range;

and the mapping submodule is used for mapping the local spherical map to be processed to the mapping candidate region to obtain the plane local region.

In one embodiment, the determining sub-module includes:

the first determining unit is used for determining the opening angle of the plane local area according to the latitude and longitude range;

and the selection unit is used for selecting the mapping candidate region from the corresponding surface according to the opening angle.

In one embodiment, the mapping submodule includes:

a second determining unit configured to determine plane coordinate values of points on the mapping candidate region;

the third determining unit is used for determining the three-dimensional coordinate value of each point in the space coordinate system of the spherical panoramic image according to the plane coordinate value of each point;

the acquisition unit is used for acquiring the longitude and latitude values of corresponding points of the points on the mapping candidate area on the spherical panoramic image according to the three-dimensional coordinate values of the points in the space;

and the drawing unit is used for drawing a plane image corresponding to the local spherical map to be processed on the mapping candidate area according to the plane coordinate value of each point on the mapping candidate area and the longitude and latitude value of the corresponding point on the spherical panoramic map so as to obtain the plane local area.

In one embodiment, the apparatus further comprises:

the second processing module is used for normalizing the plane coordinate values of all points on the mapping candidate area to obtain the normalized plane coordinate values of all points on the mapping candidate area;

in one embodiment, the third determination unit is configured to:

In one embodiment, the rendering unit is to:

In one embodiment, the apparatus further comprises:

and the second determining module is used for respectively determining the plane coordinate value of each point on the plane local area and the longitude and latitude value of the corresponding point of the spherical panoramic image as the texture coordinate and the vertex coordinate in opengl when the processed plane local area is mapped to the area corresponding to the local spherical image to be processed on the spherical panoramic image.

The present invention also provides a terminal, including: the image processing apparatus according to any one of the above aspects.

The technical scheme provided by the embodiment of the invention can have the following beneficial effects:

the processing of the partial spherical image on the spherical panoramic image can be realized by mapping the spherical image of the part to be processed in the spherical panoramic image into a plan image, processing the plan image, and then remapping the processed plan image into the corresponding area on the spherical surface.

Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

The technical solution of the present invention is further described in detail by the accompanying drawings and embodiments.

Drawings

The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention and not to limit the invention. In the drawings:

FIG. 1 is a flow diagram illustrating an image processing method according to an exemplary embodiment.

FIG. 2 is a flow diagram illustrating another image processing method according to an exemplary embodiment.

FIG. 3 is a flow chart illustrating yet another image processing method according to an exemplary embodiment.

Fig. 4A is a schematic diagram illustrating a spherical panorama created with an inscribed hexahedron according to an exemplary embodiment.

Fig. 4B is a diagram illustrating selection of a mapping candidate region from the ABCD plane, according to an exemplary embodiment.

Fig. 4C is a schematic diagram illustrating a relative relationship between the local spherical map N to be processed in the spherical panorama and the spatial orientation of the mapping candidate region according to an exemplary embodiment.

Fig. 4D is a diagram illustrating a rotated mapping candidate region according to an exemplary embodiment.

Fig. 5 is a block diagram illustrating an image processing apparatus according to an exemplary embodiment.

Fig. 6 is a block diagram illustrating another image processing apparatus according to an exemplary embodiment.

Fig. 7 is a block diagram illustrating yet another image processing apparatus according to an exemplary embodiment.

Fig. 8 is a block diagram illustrating still another image processing apparatus according to an exemplary embodiment.

Detailed Description

Various exemplary embodiments of the present invention will now be described in detail with reference to fig. 1 to 7. It should be noted that: the relative arrangement of the components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.

The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail but are intended to be part of the specification where appropriate.

It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, further discussion thereof is not required in subsequent figures.

Embodiments of the invention are operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well known computing systems, environments, and/or configurations that may be suitable for use with the computer system/server include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, hand-held or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, networked personal computers, minicomputer systems, mainframe computer systems, distributed cloud computing environments that include any of the above, and the like.

The computer system/server may be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The computer system/server may be practiced in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media including memory storage devices.

As shown in fig. 1, the image processing method in the embodiment of the present invention may include: step S101 to step S104, wherein,

in step S101, a local spherical image to be processed on the spherical panorama is determined;

in step S102, mapping the local spherical map to be processed to a corresponding surface in a preset polyhedral structure, and obtaining a planar local region in the corresponding surface;

the polyhedron structure is an inscribed polyhedron in the spherical panorama, and the corresponding surface is one of a plurality of surfaces of the polyhedron corresponding to the local spherical map to be processed, and for convenience of calculation, the polyhedron structure can be a hexahedron, namely a cube.

And a planar image corresponding to the local spherical image to be processed is mapped on the planar local area, and the size of the planar local area is smaller than or equal to that of the corresponding surface.

In step S103, performing preset processing on the planar local area;

the preset processing may be any processing, such as partial/total replacement of the plane image mapped on the plane partial area, AR processing, and the like, such as replacement of the plane image mapped on the plane partial area into a logo or an advertisement of an arbitrary shape such as a quadrangle.

In step S104, the processed planar local area is mapped to an area corresponding to the local spherical image to be processed on the spherical panorama.

After determining the local spherical image to be processed on the spherical panoramic image, mapping the local spherical image to be processed to the corresponding surface in the preset polyhedral structure to obtain the local planar area in the corresponding surface, wherein the image mapped on the local planar area is the planar image corresponding to the local spherical image to be processed, i.e. mapping the local spherical image to be processed into the planar image, so that by pre-processing the local planar area, the image can be processed by using a processing algorithm based on the planar image in the related art, and then the local planar area after processing is re-mapped to the area corresponding to the local spherical image to be processed on the spherical panoramic image, i.e. obtaining the processed spherical panoramic image, therefore, the invention can map the spherical image of the part to be processed in the spherical panoramic image into the planar image and process the planar image, and remapping the processed plan map to a corresponding area on the spherical surface, so as to realize the processing of partial spherical surface images on the spherical panoramic image.

As shown in fig. 2, in an embodiment, in step S102 shown in fig. 1, that is, mapping the local spherical map to be processed to corresponding faces in a preset polyhedral structure, and obtaining a planar local area in the corresponding faces, includes:

in step S201, a latitude and longitude range of the local spherical map to be processed is obtained;

the latitude and longitude range is a rough latitude and longitude range, that is, an approximate angle range of the local spherical image to be processed relative to the three-dimensional coordinate system where the spherical panoramic image is located, and is used for representing an approximate position/orientation of the local spherical image to be processed in the spherical panoramic image.

In step S202, according to the latitude and longitude range, a corresponding surface corresponding to the latitude and longitude range and a mapping candidate area in the corresponding surface are determined from the polyhedral structure;

the corresponding face is the face most likely to cover the plane image corresponding to the local spherical map to be processed in the plurality of faces of the polyhedron.

In step S203, the local spherical map to be processed is mapped to the mapping candidate region, and a planar local region is obtained.

When the planar local area is obtained, the latitude and longitude range of the local spherical map to be processed can be obtained, then a corresponding surface corresponding to the latitude and longitude range is selected from a plurality of surfaces of the polyhedral structure based on the latitude and longitude range, and the local spherical map to be processed is mapped to the corresponding surface, namely the spherical map is mapped to the planar map, so that the planar local area is obtained.

In one embodiment, determining the mapping candidate region in the corresponding plane according to the latitude and longitude range includes:

the field angle is a first included angle formed by a connecting line of the midpoint of the upper edge and the lower edge of the plane local area and the origin (or referred to as the sphere center) of a space coordinate system where the spherical panorama is located; and/or the field angle is a second included angle formed by a connecting line of a midpoint of the left side and the right side of the plane local area and an origin (or referred to as a sphere center) of a space coordinate system where the spherical panorama is located, and for convenience of processing, the first included angle and the second included angle may be the same, that is, the plane local area may be a square.

Selecting a mapping candidate region from the corresponding surface according to the opening angle;

since the planar image corresponding to the local spherical map to be processed may be relatively small, and may not completely occupy the corresponding surface, and the size of the opening angle may determine the specific size of the planar image, a small region, i.e. the mapping candidate region, may be selected from the corresponding surfaces according to the opening angle, and then the local spherical map to be processed may be mapped to the mapping candidate region, i.e. the planar local region mapped with the planar image corresponding to the local spherical map to be processed is obtained, for example, when the polyhedron structure is a hexahedron, after 1 square face is selected from six faces of the hexahedron as the corresponding surface, if the opening angle size is 45 °, a small square region with a side length half of that of the square is selected from the square with the center point of the square as the center, and then the local spherical map to be processed is mapped to the small square region, i.e. a planar local area is obtained.

As shown in fig. 3, in an embodiment, in step S203 shown in fig. 1, that is, mapping the local spherical map to be processed to the mapping candidate region to obtain the planar local region, may be performed as:

in step S301, a plane coordinate value of each point on the mapping candidate region is determined;

the plane coordinate value is a coordinate value of each point on the mapping candidate area relative to the two-dimensional coordinate system after the mapping candidate area or the corresponding surface establishes the two-dimensional coordinate system, and if the mapping candidate area/the corresponding surface is a square, the two adjacent sides of the square are respectively taken as a horizontal axis and a vertical axis to establish the two-dimensional coordinate system;

the selection of each point on the mapping candidate region may be determined according to the resolution of the planar image desired to be obtained, for example, if the resolution is 480 × 480, each point on the mapping candidate region is 480 points in the transverse direction, 480 points in the longitudinal direction, and then the coordinate values of each point of the 480 × 480 points with respect to the two-dimensional coordinate system are obtained.

In step S302, determining three-dimensional coordinate values of the points in a spatial coordinate system where the spherical panorama is located according to the plane coordinate values of the points;

the origin of the space coordinate system is the sphere center of the sphere where the spherical panorama is located, and the X, Y, Z axes of the space coordinate system can be freely arranged, such as horizontally towards the right, forwards and vertically upwards along the radius of the sphere, respectively.

On the other hand, since the two-dimensional coordinate system in which the plane coordinate values are relative to each other is associated with the space coordinate system, the three-dimensional coordinate value of each point in the space coordinate system can be determined from the plane coordinate value of each point, and for example, if the corresponding plane is a plane perpendicular to XOZ (where O is the center of sphere) and parallel to XOY, since the Z-coordinates of each point on the mapping candidate region are all the same, e.g., a two-dimensional coordinate system is established with the horizontal and vertical sides of the corresponding plane as the x-axis and the y-axis, respectively (assuming that the directions of the x-axis and the y-axis of the two-dimensional coordinate system are the same and parallel to each other as the directions of the x-axis and the y-axis of the three-dimensional coordinate system), and after X, Y coordinates (i.e., (x, y)) of each point on the mapping candidate region are obtained, the three-.

In addition, even if the mapping candidate region is not parallel or perpendicular to the plane formed by the adjacent coordinate axes, the plane equation of the plane can be calculated according to each end point on the mapping candidate region, then the plane is projected, the two-dimensional coordinates of each point on the projection plane are calculated, and finally the three-dimensional coordinates of each point on the mapping candidate region can be obtained based on the plane equation and the two-dimensional coordinates of each point on the projection plane.

In step S303, according to the three-dimensional coordinate values of the points in the space, obtaining longitude and latitude values of corresponding points of the points on the mapping candidate region on the spherical panorama;

the corresponding points of each point on the mapping candidate area on the spherical panorama are the intersection points with the spherical panorama when the rays (the spherical center is the end point of the rays) determined by each point on the mapping candidate area and the spherical center pass through the spherical panorama, and the intersection points are also the points on the spherical map to be processed.

In step S304, a planar image corresponding to the local spherical map to be processed is drawn on the mapping candidate region according to the planar coordinate values of the points on the mapping candidate region and the longitude and latitude values of the corresponding points on the spherical panoramic map, so as to obtain a planar local region.

According to the three-dimensional coordinate value of each point in the space and the spherical radius value corresponding to the sphere where the spherical panorama is located, the longitude and latitude values of corresponding points of each point on the mapping candidate area on the spherical panorama can be obtained, and then according to the plane coordinate value of each point on the mapping candidate area and the longitude and latitude value of the corresponding point on the spherical panorama, a plane image corresponding to the local spherical map to be processed can be drawn on the mapping candidate area to obtain a plane local area, so that subsequent processing can be conveniently carried out on the plane image.

In one embodiment, the method further comprises: and normalizing the plane coordinate values of all points on the mapping candidate area to obtain the normalized plane coordinate values of all points on the mapping candidate area.

For convenience of processing, the plane coordinate values of the points on the mapping candidate region can be normalized to obtain the plane coordinate values of the points on the normalized mapping candidate region, so that the three-dimensional coordinate values of the points in the space coordinate system where the spherical panorama is located can be calculated by using the plane coordinate values of the points on the normalized mapping candidate region.

In one embodiment, the step S302 in fig. 3, namely determining the three-dimensional coordinate values of the points in the space coordinate system where the spherical panorama is located according to the plane coordinate values of the points, may be performed as follows:

when the initial rotation angle of the mapping candidate region is not matched with the preset rotation angle, the mapping candidate region is rotated (so that the current rotation angle of the rotated mapping candidate region is matched with the preset rotation angle), and then three-dimensional coordinate values of each point on the mapping candidate region before and after rotation are respectively determined based on the plane coordinate values of each point.

When the initial rotation angle of the mapping candidate region is not matched with the preset rotation angle, it is indicated that the mapping candidate region is offset from the local spherical map to be processed, at this time, corresponding points of some points on the mapping candidate region on the spherical panoramic map may not be points on the local spherical map to be processed, that is, the mapping candidate region cannot completely cover a planar image corresponding to the local spherical map to be processed, so that the mapping candidate region can be rotated to match the current rotation angle of the rotated mapping candidate region with the preset rotation angle, and three-dimensional coordinate values of the mapping candidate region before and after the rotation are respectively determined based on planar coordinate values of the points on the mapping candidate region.

For example: when the initial rotation angle of the mapping candidate region is not matched with the preset rotation angle, the mapping candidate region can be horizontally and/or vertically rotated, the plane coordinate values of all points on the mapping candidate region are mapped, the three-dimensional coordinate values of all points on the mapping candidate region before rotation are determined, and then the three-dimensional coordinate values of all points on the mapping candidate region after rotation are calculated according to the three-dimensional coordinate values of all points on the mapping candidate region before rotation.

The initial rotation angle of the mapping candidate region is not matched with the preset rotation angle, which means that the initial rotation angle is not equal to the preset rotation angle, or the angle difference between the initial rotation angle and the preset rotation angle is greater than the preset angle difference.

And then, the rotation angle (including the initial rotation angle and the preset rotation angle) is an included angle between the mapping candidate region and a plane formed by two coordinate axes specified in the space coordinate system, and the rotation angle includes a vertical rotation angle and a horizontal rotation angle.

In one embodiment of the present invention,

drawing a plane image corresponding to the local spherical image to be processed on the mapping candidate area according to the plane coordinate value of each point on the mapping candidate area and the longitude and latitude value of the corresponding point on the spherical panoramic image to obtain a plane local area, wherein the plane local area comprises the following steps:

drawing a plane image on the mapping candidate area according to the vertex coordinate and the texture coordinate in opengl to obtain a plane local area.

When drawing a plane image according to the plane coordinate values of each point on the mapping candidate area and the longitude and latitude values of the corresponding point on the spherical panorama, the plane coordinate values of each point on the mapping candidate area and the longitude and latitude values of the corresponding point on the spherical panorama can be respectively determined as the vertex coordinate and the texture coordinate in opengl, and then the pixel value on the texture coordinate is rendered to the vertex coordinate, so that the plane image mapped by the spherical image of a part to be processed can be obtained through opengl (Open Graphics Library, which is a professional graphical program interface defining a cross-programming language and cross-platform programming interface specification).

In one embodiment, the method further comprises:

when the processed plane local area is mapped to an area corresponding to a local spherical image to be processed on the spherical panoramic image, the plane coordinate values of all points on the plane local area and the longitude and latitude values of corresponding points of the spherical panoramic image are respectively determined as texture coordinates and vertex coordinates in opengl.

When the plane local area is remapped to the spherical surface, the plane coordinate value of each point on the plane local area and the longitude and latitude value of the corresponding point of the spherical panoramic image can be respectively determined as the texture coordinate and the vertex coordinate in opengl when opengl is used for drawing, then the new pixel value on the texture coordinate after the preset processing is rendered to the vertex coordinate, the mapping of the image can be completed, and the processed plane image is reapplied to the spherical panoramic image.

Further, it will be appreciated by those skilled in the art that the above embodiments may be combined with each other, and if a mapping candidate region is selected from the corresponding plane according to the opening angle.

Examples of the invention will now be illustrated:

constructing a polyhedral structure in the spherical panoramic image;

if the local spherical map P to be processed approximately corresponds to the M surface in the polyhedral structure according to the approximate longitude and latitude range of the local spherical map P to be processed on the spherical panoramic map, namely the M surface is the corresponding surface, the local spherical map P to be processed is mapped onto the M surface, namely the plane local area mapped with the plane image corresponding to the local spherical map to be processed is obtained, and then the processed plane local area is mapped to the area corresponding to the local spherical map to be processed on the spherical panoramic map after the plane local area is correspondingly processed.

In the above embodiment, when the local spherical map P to be processed is mapped onto the M plane to obtain the local planar local area, the opening angle of the local planar area may be determined according to the approximate latitude and longitude range, and then a corresponding small area is selected from the corresponding plane according to the opening angle as a mapping candidate area, and then if the initial rotation angle of the mapping candidate area does not match the preset rotation angle, that is, the mapping candidate area cannot cover the planar image corresponding to the local spherical map to be processed, the mapping candidate area is vertically or horizontally rotated, three-dimensional coordinates of points on the mapping candidate area in the three-dimensional coordinate system before and after the rotation are determined, and based on the spherical radius, the latitude and longitude values of the intersection point of the ray between each point on the mapping candidate area and the spherical center (where the spherical center is the end point of the ray) on the spherical surface (where the spherical center is the corresponding point of each point on the spherical panoramic map candidate area) are determined, then, according to the plane coordinate values of the points on the mapping candidate region and the longitude and latitude values of the corresponding points of the points on the mapping candidate region on the spherical panorama, a plane image is drawn on the mapping candidate region, and then the plane local region (i.e. the mapping candidate region on which the plane image is drawn) is obtained.

Embodiments of the present invention will be described in further detail below with reference to fig. 4A to 4D:

the invention maps a part of spherical images to be processed on the spherical panoramic image into a plan image, and the plan image is processed and then attached back to the original spherical panoramic image. Wherein: in the embodiment, s- > cfov is the opening angle, s- > rotate _ v is the vertical rotation angle, and s- > rotate _ h is the horizontal rotation angle; wherein,

s- > cfov is an included angle formed by the midpoint of the upper edge and the lower edge of the mapping candidate region and the sphere center;

s- > rotate _ v refers to an angle at which the mapping candidate region should be rotated along the X axis shown in fig. 4A to 4D, and is also an angle difference between the mapping candidate region and the XOY plane before and after the rotation along the X axis, the vertical rotation angle is equal to an angle difference between the mapping candidate region and the XOY plane, which should reach a preset vertical angle, which is determined according to a approximate latitude range of the map N, and an initial vertical angle of an included angle between the mapping candidate region and the XOY plane;

s- > rotate _ h refers to an angle at which the mapping candidate region should be rotated along the Y axis shown in fig. 4A to 4D, and may also be an angle difference between the mapping candidate region and the YOZ plane before and after the rotation along the Y axis, and the vertical rotation angle is equal to an angle difference between the mapping candidate region and the YOZ plane which should reach the preset horizontal angle and an initial horizontal angle of an included angle between the mapping candidate region and the YOZ plane, and the preset vertical angle is determined according to an approximate longitude range of the map N. The specific scheme is as follows:

the first step is as follows: as shown in fig. 4A, an inscribed regular hexahedron is built inside the spherical panorama, the direction of its X, Y, Z axis and the inscribed regular hexahedron are as shown in fig. 4A, and a diagram N in fig. 4A represents a partial spherical diagram to be processed on the spherical panorama.

The second step is as follows: according to the approximate latitude and longitude range of the diagram N, one of the six surfaces of the hexahedron corresponding to the diagram N can be determined, as shown in FIG. 4A, and the ABCD plane in the hexahedron corresponding to the diagram N is determined as the corresponding surface of the diagram N.

The third step: as can be seen from fig. 4A, the size of the map N is smaller than that of the ABCD, and therefore, the mapping candidate area, i.e., the shaded area in fig. 4B, can be determined from the ABCD plane to be a small square area according to the approximate opening angle of the map N (as shown in fig. 4B, the opening angle is the included angle formed by the connecting lines of the upper middle point E and the lower middle point F of the shaded portion and the center O, and of course, in order to facilitate the selection of the area, in this embodiment, the included angle formed by the connecting lines of the left middle point and the right middle point of the shaded portion and the center O is also the same, but is not shown in fig. 4B).

The fourth step: the process of normalizing the horizontal and vertical coordinates, i.e., X and Y, of each point on the mapping candidate region, which is the shadow region shown in fig. 4B (of course, the horizontal axis of the corresponding two-dimensional coordinate system is the coordinate value of the plane of each point on the mapping candidate region, the horizontal direction of the CD side is the same as and parallel to the X-axis direction in the three-dimensional coordinate system shown in fig. 4B, the horizontal direction of the BC side is the same as and parallel to the Y-axis direction in the three-dimensional coordinate system shown in fig. 4B, and the vertical axis is not shown in fig. 4B), is as follows:

ix＝(x-0.5)*((float)s->cfov/90.0)+0.5；

iy＝(y-0.5)*((float)s->cfov/90.0)+0.5；

x＝ix；

y＝iy；

the maximum angle of the opening angle is 90 degrees, and the narrowing is performed by taking the center of the ABCD surface as the center.

The fifth step: the process of determining the three-dimensional coordinate values of the points x, y on the mapping candidate area in the three-dimensional coordinate system XYZ is as follows (where, f in the following is float, which indicates a floating-point type data type):

sin_y＝sin(0.0f)；

cos_y＝cos(0.0f)；

sin_p＝sin(s->rotate_v*M_PI/180.0f)；

cos_p＝cos(s->rotate_v*M_PI/180.0f)；

tx＝qx*cos_y-qy*sin_y*sin_p+qz*sin_y*cos_p；

ty＝qy*cos_p+qz*sin_p；

tz＝qx*(-sin_y)-qy*cos_y*sin_p+qz*cos_y*cos_p；

d＝sqrtf(tx*tx+ty*ty+tz*tz)；

qx, qy, qz is the three-dimensional coordinate of the point on the mapping candidate area in the three-dimensional coordinate system XYZ before the mapping candidate area is vertically rotated, tx, ty, tz is the three-dimensional coordinate after the mapping candidate area is vertically rotated, d is the distance from each point on the mapping surface to the center of the sphere (i.e. the radius of the sphere) after the vertical rotation, and M _ PI is the circumferential ratio PI;

in addition, in this step, since the ABCD plane is perpendicular to the XOZ plane and the Z coordinates of all points thereon are the same, when qx, qy, and qz are calculated from X and Y in the fourth step, since the two-dimensional coordinate system in which the ABCD plane is located and the X axis and the Y axis in the XYZ three-dimensional coordinate system are parallel, qx, qy, and qz, which is the perpendicular distance between the ABCD plane and the XOY plane, can be determined from the perpendicular distance between the ABCD plane and the XOY plane after X and Y in the fourth step are calculated.

A sixth step: obtaining the longitude and latitude values of corresponding points of each point on the mapping candidate area on the spherical panorama, namely the coordinates of the corresponding points on the spherical panorama:

outX＝-atan2f(-tx/d,tz/d)/(M_PI*1.0f)+0.5f；

outY＝asinf(-ty/d)/M_PI+0.5f；

outX+＝(s->rotate_h/360.0f)；

thus, given a certain point (x, y) on the mapping candidate region, the coordinates (outX, outY) of the corresponding point on the spherical panorama are obtained, and the rotation effects in the fifth step and the sixth step are shown in fig. 4D, where fig. 4C shows the relative relationship between the mapping candidate region before rotation and the spatial orientation of the map N, and as can be seen from fig. 4C, if the map N is mapped to the mapping candidate region before rotation (i.e., the shadow region), i.e., the initial rotation angle of the mapping candidate region and the preset rotation angle, at this time, the mapping candidate region before rotation cannot completely cover the plane image corresponding to the map N, i.e., the mapping candidate region before rotation and the corresponding plane image are not parallel to each other but inclined to each other, and thus, the mapping candidate region can be rotated vertically and/or horizontally according to the above steps to cover the plane image corresponding to the map N, and the rotated mapping candidate region is shown in fig. 4D.

A seventh step of: using opengl to set x, y as vertex coordinates and outX, outY as texture coordinates, we get the planar image of the part to be processed (i.e. drawing the planar image corresponding to the map N on the mapping candidate area after rotation as shown in fig. 4D can get the planar local area).

An eighth step: the planar image is processed, wherein various processing may be performed, such as AR, post-processing, and so on.

A ninth step: remapping the processed planar image into a spherical panorama, specifically: when opengl is used for drawing, x and y are set as texture coordinates, and outX and outY are set as vertex coordinates.

As shown in fig. 5, the present invention also provides an image processing apparatus comprising:

a first determining module 501 configured to determine a local spherical map to be processed on the spherical panorama;

a first mapping module 502, configured to map the local spherical map to be processed to a corresponding surface in a preset polyhedral structure, to obtain a planar local area in the corresponding surface;

a first processing module 503 configured to perform preset processing on the planar local area;

the second mapping module 504 is configured to map the processed planar local area to an area corresponding to a local spherical map to be processed on the spherical panorama.

As shown in fig. 6, in one embodiment, the first mapping module 502 may include:

the acquisition submodule 5021 is configured to acquire a longitude and latitude range of the local spherical image to be processed;

the determining submodule 5022 is configured to determine a corresponding surface corresponding to the latitude and longitude range and a mapping candidate area in the corresponding surface from the polyhedral structure according to the latitude and longitude range;

the mapping submodule 5023 is configured to map the local spherical map to be processed to the mapping candidate region, and obtain a planar local region.

As shown in fig. 7, in one embodiment, the determination submodule 5022 may comprise:

a first determining unit 50221, configured to determine the opening angle of the plane local area according to the latitude and longitude range;

a selecting unit 50222 configured to select a mapping candidate region from the corresponding plane according to the opening angle.

As shown in fig. 8, in one embodiment, the mapping submodule 5023 includes:

a second determining unit 50231 configured to determine plane coordinate values of each point on the mapping candidate region;

a third determining unit 50232, configured to determine three-dimensional coordinate values of the points in a space coordinate system where the spherical panorama is located according to the plane coordinate values of the points;

an obtaining unit 50233, configured to obtain the longitude and latitude values of corresponding points of each point on the mapping candidate area on the spherical panorama according to the three-dimensional coordinate values of each point in the space;

a drawing unit 50234 configured to draw a plane image corresponding to the local spherical map to be processed on the mapping candidate area according to the plane coordinate values of the points on the mapping candidate area and the longitude and latitude values of the corresponding points on the spherical panorama to obtain a plane local area.

In one embodiment, the apparatus further comprises:

the second processing module is configured to normalize the plane coordinate values of the points on the mapping candidate area to obtain the normalized plane coordinate values of the points on the mapping candidate area;

in one embodiment, the third determination unit is configured to:

and when the initial rotation angle of the mapping candidate region is not matched with the preset rotation angle, rotating the mapping candidate region, and respectively determining the three-dimensional coordinate values of each point on the mapping candidate region before and after rotation based on the plane coordinate values of each point.

In one embodiment, the rendering unit is configured to:

In one embodiment, the apparatus further comprises:

and the second determining module is configured to respectively determine the plane coordinate value of each point on the plane local area and the latitude and longitude value of the corresponding point of the spherical panorama as the texture coordinate and the vertex coordinate in opengl when the processed plane local area is mapped to the area corresponding to the local spherical map to be processed on the spherical panorama.

In addition, in the present specification, the embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same or similar parts among the embodiments are referred to each other. For the device embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and for the relevant points, reference may be made to the partial description of the method embodiment.

The method and apparatus of the present invention may be implemented in a number of ways. For example, the methods and systems of the present invention may be implemented in software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order for the steps of the method is for illustrative purposes only, and the steps of the method of the present invention are not limited to the order specifically described above unless specifically indicated otherwise. Furthermore, in some embodiments, the present invention may also be embodied as a program recorded in a recording medium, the program including machine-readable instructions for implementing a method according to the present invention. Thus, the present invention also covers a recording medium storing a program for executing the method according to the present invention.

The description of the present invention has been presented for purposes of illustration and description, and is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to practitioners skilled in this art. The embodiment was chosen and described in order to best explain the principles of the invention and the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.

Claims

1. An image processing method, comprising:

presetting the plane local area;

2. The method of claim 1,

the mapping the local spherical graph to be processed to a corresponding surface in a preset polyhedral structure to obtain a plane local area in the corresponding surface includes:

3. The method of claim 2,

determining a mapping candidate area in the corresponding plane according to the latitude and longitude range comprises the following steps:

4. The method according to claim 2 or 3,

the mapping the to-be-processed local spherical map to the mapping candidate region to obtain the plane local region includes:

5. The method of claim 4, further comprising:

6. The method of claim 4,

the determining the three-dimensional coordinate value of each point in the space coordinate system of the spherical panoramic image according to the plane coordinate value of each point comprises:

7. The method of claim 4,

the drawing a plane image corresponding to the local spherical map to be processed on the mapping candidate region according to the plane coordinate values of the points on the mapping candidate region and the longitude and latitude values of the corresponding points on the spherical panoramic map to obtain the plane local region includes:

8. The method of claim 7, further comprising:

9. An image processing apparatus characterized by comprising:

10. The apparatus of claim 9,

the first mapping module comprises:

11. The apparatus of claim 10,

the determination sub-module includes:

12. The apparatus of claim 10 or 11,

the mapping submodule includes:

13. The apparatus of claim 12, further comprising:

and the second processing module is used for normalizing the plane coordinate values of the points on the mapping candidate area to obtain the normalized plane coordinate values of the points on the mapping candidate area.

14. The apparatus of claim 12,

the third determination unit is configured to:

15. The apparatus of claim 12,

the drawing unit is used for:

16. The apparatus of claim 15, further comprising:

17. A terminal, comprising: the image processing apparatus according to any one of claims 9 to 16.