CN106688016A - Method and terminal for handling video stream - Google Patents
Method and terminal for handling video stream Download PDFInfo
- Publication number
- CN106688016A CN106688016A CN201680002250.6A CN201680002250A CN106688016A CN 106688016 A CN106688016 A CN 106688016A CN 201680002250 A CN201680002250 A CN 201680002250A CN 106688016 A CN106688016 A CN 106688016A
- Authority
- CN
- China
- Prior art keywords
- gpu
- image data
- described image
- characteristic point
- video
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
- G06T11/001—Texturing; Colouring; Generation of texture or colour
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
Abstract
The present disclosure provides a method (300), a corresponding terminal (100), a computer program, and a computer program product for processing a video stream using a graphics processing unit GPU (104) at a processor (102) of a terminal (100). The method comprises the following steps: (S310) image data of at least one image in the video stream is obtained; (S320) at least one feature point of the image data is determined; (S330) the GPU (104) is instructed to subjecting the image data to image processing operation.
Description
Copyright notice
This patent document disclosure includes material protected by copyright.The copyright is all for copyright holder.Copyright
Owner does not oppose that anyone replicates the patent document or special in the presence of the proce's-verbal of Patent＆Trademark Office and archives
Profit is disclosed.
Technical field
This disclosure relates to image processing field, relates more specifically to process video flowing using GPU (GPU)
Method and terminal.
Background technology
With becoming increasingly popular for terminal, especially movable termination (for example, smart mobile phone, panel computer etc.), it is
Become the indispensable part in production, life of people.Terminal provides abundant various functions, including is for example directed to
The recording of image/video, treatment and live etc..One of critical function as video processing applications is to provide the user to repair
Figure function (or more generically, the U.S. thin face function of face).User can be modified by the original image that camera is collected
Process to realize making customer satisfaction system image/video effect.
The content of the invention
According to the first aspect of the disclosure, it is proposed that a kind of that GPU (GPU) is used at the processor of terminal
Method to process video flowing.The method includes：Obtain the view data of at least two field picture in the video flowing；Determine institute
State at least one of view data characteristic point；And indicate the GPU to be based at least one characteristic point to described image
Data carry out image procossing.
In certain embodiments, the video flowing is the video flowing with 4K or higher resolution.In certain embodiments,
An at least two field picture include face, and at least one characteristic point include it is following at least one：Face, eyes, nose
Son, mouth, ear, hair or eyebrow.In certain embodiments, described image treatment include it is following at least one：Contour revising,
Color correct, brightness correction and Fuzzy Processing.In certain embodiments, methods described also includes：For by image procossing
Described image data, indicate that the GPU performs video record operation, video display operation, Object tracking is operated or video is straight
Broadcast at least one in operating.In certain embodiments, the GPU is indicated to perform video record operation, video display operation, right
At least one during image tracing is operated or net cast is operated includes：Indicate the GPU executed in parallel video record operation, regard
At least two in frequency display operation, Object tracking operation or net cast operation.In certain embodiments, in the GPU bases
Before at least one characteristic point carries out image procossing to described image data, coordinate change is carried out to described image data
Change, allow it to carry out image procossing by the GPU.In certain embodiments, it is based on described at least one spy in the GPU
Levy after a little described image data are carried out with image procossing, if the GPU is to be carried out at image for the video flowing first
Reason, then create new texture, otherwise in the case of the ratio of width to height of described image data is immovable, only updates existing texture.
In certain embodiments, if the GPU is not to carry out image procossing and described image data for the video flowing first
The ratio of width to height there occurs change, then create new texture.In certain embodiments, the change of the ratio of width to height of described image data be by
What change camera and/or the rotation terminal caused.
According to the second aspect of the disclosure, it is proposed that one kind processes the end of video flowing using GPU (GPU)
End.The terminal includes：Image data acquisition apparatus, the view data for obtaining at least two field picture in the video flowing；
Characteristic point determination device, for determining at least one of described image data characteristic point；And image procossing instruction device, use
Image procossing is carried out to described image data based at least one characteristic point in the GPU is indicated.
In certain embodiments, the video flowing is the video flowing with 4K or higher resolution.In certain embodiments,
An at least two field picture include face, and at least one characteristic point include it is following at least one：Face, eyes, nose
Son, mouth, ear, hair or eyebrow.In certain embodiments, described image treatment include it is following at least one：Contour revising,
Color correct, brightness correction and Fuzzy Processing.In certain embodiments, the terminal also includes：Operation instruction device, uses
Video record operation, video display operation, right is performed in the GPU for the described image data by image procossing, is indicated
Image tracing operate or net cast operation at least one.In certain embodiments, the operation instruction device is additionally operable to：
In indicating GPU executed in parallel video record operation, video display operation, Object tracking operation or net cast to operate
At least two.In certain embodiments, image is carried out to described image data based at least one characteristic point in the GPU
Before treatment, coordinate transform is carried out to described image data, allow it to carry out image procossing by the GPU.In some realities
In applying example, after the GPU carries out image procossing based at least one characteristic point to described image data, if described
GPU is to carry out image procossing for the video flowing first, then new texture is created, otherwise in the ratio of width to height of described image data
In the case of immovable, existing texture is only updated.In certain embodiments, if the GPU is not to be regarded for described first
Frequency stream carries out image procossing and the ratio of width to height of described image data there occurs change, then create new texture.In some embodiments
In, the change of the ratio of width to height of described image data is caused by change camera and/or the rotation terminal.
According to the third aspect of the disclosure, it is proposed that a kind of terminal for processing video flowing.The terminal includes：Treatment
Device；GPU (GPU)；Memory, store instruction, the instruction causes the treatment when by the computing device
Device：Obtain the view data of at least two field picture in the video flowing；Determine at least one of described image data feature
Point；And the instruction GPU carries out image procossing based at least one characteristic point to described image data.
In certain embodiments, the video flowing is the video flowing with 4K or higher resolution.In certain embodiments,
An at least two field picture include face, and at least one characteristic point include it is following at least one：Face, eyes, nose
Son, mouth, ear, hair or eyebrow.In certain embodiments, described image treatment include it is following at least one：Contour revising,
Color correct, brightness correction and Fuzzy Processing.In certain embodiments, the instruction is gone back when by the computing device
So that the processor：For the described image data by image procossing, the GPU is indicated to perform video record operation, regard
Frequently at least one during operation, Object tracking are operated or net cast is operated is shown.In certain embodiments, the instruction exists
During by the computing device, the processor is also caused：For the described image data by image procossing, indicate described
At least two in the operation of GPU executed in parallel video record, video display operation, Object tracking operation or net cast operation.
In certain embodiments, before the GPU carries out image procossing based at least one characteristic point to described image data,
Coordinate transform is carried out to described image data, allows it to carry out image procossing by the GPU.In certain embodiments, exist
The GPU is carried out after image procossing based at least one characteristic point to described image data, if the GPU is first
Image procossing is carried out for the video flowing, then creates new texture, it is otherwise immovable in the ratio of width to height of described image data
In the case of, only update existing texture.In certain embodiments, if the GPU is not to be carried out for the video flowing first
The ratio of width to height of image procossing and described image data there occurs change, then create new texture.In certain embodiments, the figure
As the change of the ratio of width to height of data is caused by change camera and/or the rotation terminal.
According to the fourth aspect of the disclosure, it is proposed that a kind of computer program, caused when by computing device described
Reason device performs the method according to any one of the first aspect of the disclosure.
A kind of the 5th aspect according to the disclosure, it is proposed that computer program product, including according to the four directions of the disclosure
Computer program described in face.
Brief description of the drawings
In order to be more fully understood from the embodiment of the present disclosure and its advantage, referring now to the following description with reference to accompanying drawing, its
In：
Fig. 1 shows the hardware architecture diagram of the terminal according to the embodiment of the present disclosure.
In the terminal that Fig. 2 shows shown in Fig. 1 between each component data/commands interaction schematic diagram.
Fig. 3 is showed and regarded according to being processed using GPU for execution at the primary processor of terminal of the embodiment of the present disclosure
The flow chart of the method for frequency stream.
Fig. 4 shows the illustrative functions frame of the terminal for performing the method shown in Fig. 3 according to the embodiment of the present disclosure
Structure block diagram.
Specific embodiment
According to reference to accompanying drawing to the described in detail below of disclosure exemplary embodiment, the other side of the disclosure, advantage
Be will become obvious for those skilled in the art with prominent features.
In the disclosure, term " including " and " containing " and its derivative mean including and it is unrestricted；Term "or" is bag
Containing property, mean and/or.
In this manual, following is explanation for describing the various embodiments of disclosure principle, should not be with any
Mode is construed to limit scope of disclosure.Referring to the drawings described below is used to help comprehensive understanding by claim and its equivalent
The exemplary embodiment of the disclosure that thing is limited.It is described below to help understand including various details, but these details should
Think what is be merely exemplary.Therefore, it will be appreciated by those of ordinary skill in the art that not departing from the scope of the present disclosure and spirit
In the case of, embodiment described herein can be made various changes and modifications.Additionally, for clarity and brevity,
Eliminate the description of known function and structure.Additionally, running through accompanying drawing, same reference numerals are used for same or analogous function and behaviour
Make.
Before formally each embodiment of the disclosure is introduced, the various terms that general description first may be used herein.
1080P：It is the format standard of the digital TV in high resolution that American film Television Engineer association (SMPTE) formulates,
Effective resolution is：1920×1080.It is a kind of display format that 1920 × 1080 resolution ratio is issued in progressive scan.
4K：4K resolution ratio is simultaneously not specific to certain special value, and it refers to horizontal direction and there are about 4000 pixels or so,
There is trickle gap according to different application fields.The trend of 4K resolution ratio development is 4096 × 2160 pixel resolution,
It is 4 times of 2K projectors and HDTV resolution ratio, belongs to ultra high-definition resolution ratio.Than the 4K films of theory as usual, its film point
Resolution 4096 × 2160, is determined by the imaging format of 4K video cameras.And the TV domain discussed in people, due to screen 16: 9
Into main flow, thus TV screen 4K refer to 3840 × 2160 physical resolution, equivalent to 1920 × 1080 (i.e.,
1080P) laterally and it is vertical on respectively turned over some, be still 16: 9 specifications of standard.In the context of the disclosure, such as without spy
Do not indicate, the 4K being previously mentioned includes various known or actual 4K resolution ratio, including but not limited to foregoing two kinds of 4K are differentiated
Rate.Additionally, similarly there is also defined the standards such as 2K, 8K at present.
Object tracking：Refer to when a certain object is appeared in the visual field/picture of camera, can be by Object identifying
Algorithm determines the presence of the object, and indicates the camera to go with clapping the determination object, until object clapping beyond camera
Untill taking the photograph scope.Object tracking is widely used in the fields such as video monitoring, unmanned plane driving.
Present inventor notices：The associative operation of the thin face function of U.S. face of existing video U.S. face software is in
Performed on Central Processing Unit (CPU), its video resolution for being at most only capable of supporting 1080P.Even if being adopted in terms of flow for displaying
Performed with GPU, it cannot also realize repairing figure treatment in real time for 4K or more the video of resolution ratio.Additionally, it is more
The concurrently execution of following various operations cannot be supported：Roller blind, take pictures, record a video, it is live, display and/or Object tracking.More typically
Ground, disclosure herein people notice：In existing terminal and the powerful data localized abilities of GPU are underused, and
So that CPU has born excessive image processing work, the problem that cannot make full use of existing hardware resource is caused.
In the present embodiment, the terminal can be movable termination, such as mobile phone, notebook computer, hand-held filming apparatus；
Or desktop computer etc..
Generally, the embodiment of the present disclosure is proposed and a kind of is realized carrying out the picture frame in video flowing using GPU at a high speed
The scheme for the treatment of.In this scenario, coordinate rotation, U.S. face are carried out to the data in picture frame by GPU, and in the auxiliary of CPU
Under carry out thin face, the high speed processing of the facial image for fine definition (for example, 4K or higher resolution) can be obviously improved.
Additionally, being solved in 4K or higher resolution by using the replacement of the data texturing on GPU and CPU, to resource rational management
Under conditions of be able to can also be executed concurrently after the thin face of U.S. face display, record a video, take pictures, live and/or Object tracking
(Tracking) function such as.
First, the terminal for carrying out video flow processing according to the embodiment of the present disclosure is described in detail by Fig. 1 is combined
Framework.Fig. 1 shows the hardware schematic of the terminal 100 for carrying out video flow processing according to the embodiment of the present disclosure.Such as Fig. 1
Shown, terminal 100 can include that primary processor (is alternatively referred to as CPU or CPU or host sometimes below
Processor) 102, GPU (being otherwise referred to as GPU below) 104 and display 106.
First, it is noted that arrive：Although primary processor 102 and GPU 104 are shown as into single two in Fig. 1
Module, but actually disclosure not limited to this.In certain embodiments, GPU 104 can be as primary processor
The graphics processing core embedded in 102.For example, in the mobile phone Mate8 that Huawei releases, its primary processor kylin 950 includes 8
Individual general purpose processor core (the 4 Cortex A72 and 4 Cortex A53 that are released by ARM companies) and 1 GPU core
(the Mali T880 released by ARM companies).Therefore, in such an example, although primary processor 102 and GPU
104 are actually a physical hardware, but logically can still be regarded as two single logic modules.In fact, right
For programmer, 8 core CPU being made up of 8 general purpose processor cores still need to call the figure of such as OpenGL etc
Routine interface could command the GPU to carry out graphics calculations work.Therefore, it can primary processor 102 and graphics process logically
Unit 104 is distinguish between, but this does not represent the necessarily different physical hardware of both of which.
Used as the processor dedicated for graphics process, it calculates core to GPU 104 compared with CPU 102 with more
The heart and therefore have more powerful computation capability, be also more suitable for image procossing.In fact, in terms of image procossing,
As explained below, for identical image procossing, GPU 104 can have the meter than at least 100 times high of CPU 102
Calculate speed.However, as it was noted above, in existing image processing process, not utilizing GPU 104 this powerful completely
Data localized ability, cannot the larger image of real-time processing resolution ratio and/or video (example so as to result in existing terminal
Such as, image and/or video with 4K or higher resolution).
Additionally, the display 106 of terminal 100 can be connected with GPU 104, show various with the control of GPU 104
Image, word etc..Display 106 can be including but not limited to：CRT (cathode-ray tube) display, LCD (liquid crystal) displays
Device, LED (light emitting diode) display, OLED (Organic Light Emitting Diode) display etc..Additionally, being shown on display 106
Content can be rotated under the control of GPU 104, with for example adapt to terminal 100 current device orient.For example, seeing
When seeing video, terminal 100 can be changed into transversal orientation from portrait orientation, to enable that video is displayed in full screen, occupy whole
Individual screen, rather than show on a portion of a screen.In addition, it is noted that display 106 is not the necessary component of terminal 100.Thing
In reality, terminal 100 with external-connection displayer or otherwise to remote display streaming display picture, and can show without built-in
Show device 106.
Communication unit 108 may be such that the module or unit that terminal 100 can be communicated with external equipment.Communication unit
Unit 108 can be Landline communication unit or wireless communication unit or the combination of both.When communication unit 108 is wire communication list
When first, it can be including but not limited to：USB module, the modules of IEEE 1394, ethernet module, DSL (DSL or
More generally xDSL) modem, serial port module etc..When communication unit 108 is wireless communication unit, it can include
Such as (but not limited to)：Various 2G modules (for example, GSM/GPRS modules etc.), 3G module (such as WCDMA modules, CDMA2000
Module, TD-SCDMA modules etc.), 4G modules (for example, TD-LTE modules etc.), the various 5G moulds that are developing and will occur
The telecommunication module such as block.Additionally, communication unit 108 can also include close range wireless communication module, for example (but do not limit
In)：Wi-Fi module, bluetooth module, NFC module, RFID module, infrared module etc..In fact, communication unit 108 is not limited to
Any module is stated, and can be any module for enabling primary processor 102 and PERCOM peripheral communication, it might even be possible to be have to share
The module of internal memory form so that primary processor 102 can use the mode of " write-in/read " and the shared number of external equipment/processor
According to.Similarly, it is noted that communication unit 108 is nor the necessary component of terminal 100.In fact, terminal 100 can be with external communication
Unit with other external device communications, and without built-in communication unit 108.
Terminal 100 can also include one or more cameras 110.In the example illustrated in figure 1, terminal 100 includes
One camera 110-1 and second camera 110-2 (hereinafter, being referred to as camera 110).The two cameras can have
There are different technical parameters.For example, the first camera 110-1 may be located at the back of terminal 100, can simultaneously check bat
Take the photograph reference object in the case of picture.Second camera 110-2 may be located at the front of terminal 100, be carried out with to user itself
Autodyne.Additionally, in further embodiments, the first camera 110-1 can rotate with and meanwhile realize the first camera 110-1 and
The shoot function of second camera 110-2.In further embodiments, the first camera 110-1 even can include two mirrors
Head, to realize 3D shootings, obtain apparent image etc..The two cameras 110-1 and 110-2 can have different resolutions
Rate, zooming range/fixed focal length, shutter speed, aperture, the depth of field, speed (ISO) etc..Additionally, the two cameras 110 with
The connected mode of primary processor 102 is also not necessarily limited to the mode shown in Fig. 1, and can be different from primary processor 102 respectively pipe
Pin is connected, or is all connected in certain data/controlling bus, disclosure not limited to this.Similarly, it is noted that camera 110-
1 and 110-2 is nor the necessary component of terminal 100.Although second camera 110-2 is shown as into dotted line frame, to show that it is
Optional camera, but this does not imply that camera 1 is essential camera.In fact, terminal 100 can with external camera with
Image and/or video are obtained, and without built-in camera 110-1 and 110-2.Additionally, terminal 100 even can be by communication unit
Unit 108 obtains image/video stream or obtains from memory 112 image/video stream, and without any built-in/outside shooting
Head.
Terminal 100 can also include one or more memories 112.Memory 112 can be volatile memory or non-
Volatile memory.For example, used as volatile memory, memory 112 can be including but not limited to：Random access memory
(RAM), dynamic ram (DRAM), static state RAM (SRAM), synchronous dram (SDRAM), cache, register etc..For example, conduct
Nonvolatile memory, memory 112 can also be including but not limited to：Disposable programmable read only memory (OTPROM),
Erasable programmable ROM (EPROM), electrically erasable ROM (EEPROM), mask ROM, flash rom, flash memory, hard disk drive
Dynamic device, solid-state drive etc..Additionally, memory 112 can also include such as (but not limited to) high density flash memory (CF), safety
Numeral (SD), miniature SD, mini SD, very fast numeral (xD), multimedia card (MMC) or memory stick etc..Memory 112 can be with
Storage is for the instruction performed by primary processor 102 and/or GPU 104 and/or to process or processed data.In some implementations
In example, such as GPU 104 and 102 shared memories 112 of CPU, memory 112 can be existed with the mode of transmission pointer
Data, such as view data in pending and/or processed video flowing are transmitted between GPU104 and CPU 102.For example, such as
Shown in dotted arrow in Fig. 1 between GPU 104 and memory 112, it alternatively can physically be directly connected to storage
Device 112, so as to carry out shared memory in the way of the time-division with CPU 102.Additionally, GPU 104 can also be by primary processor 102
To access the instruction and/or data that are stored in memory 112.Disclosure not limited to this.
Terminal 100 can also include being referred to as the various other functional units of other modules 114, including but not limited to：
For example, power module, sensor assembly, input/output module (for example, keyboard, button etc., audio-frequency module), vibration module, plus
Close module etc..But whether the presence of which has no effect on the understanding and realization to the purport of the embodiment of the present disclosure, therefore this
Place eliminates detailed description thereof.
The data/commands being described in detail in next with reference to Fig. 2 in the terminal 100 shown in Fig. 1 between each component are handed over
Mutually, the interaction enables terminal 100 using its GPU 104 to process video flowing.
In the embodiment shown in Figure 2, at step S201, the video stream data as captured by camera 110 can lead to
Cross various modes to be transmitted to primary processor 102, GPU 104 and/or memory 112, for example, by data/address bus directly to each
Component is transmitted, or is transferred to CPU 102, and is forwarded from CPU 102 to other each components.Please note：In certain embodiments,
The video stream data can not be transmitted to memory 112 and/or CPU 102, and only be obtained by GPU 104.
Next, each two field picture number of the video stream data collected to camera 110 in step S202, GPU 104
Rotated according to optional coordinate is performed, be the number that can be correctly processed by GPU 104 by the image data transformation that camera is photographed
According to form (for example, under transforming to the coordinate system of GPU 104).Then, in step S203, GPU 104 is from the video stream data
Each frame raw image data is extracted, and is sent to primary processor 102 and/or memory 112 in step S204.
After the raw image data is received, primary processor 102 can be held in step S205 to the initial data
Row recognition of face and/or human face characteristic point determine algorithm, whether to determine in each two field picture comprising face, and if comprising
Face, it is determined that the position of each face feature (for example, face, eyes, nose, mouth, ear, hair or eyebrow etc.) in face
Put, profile, color, one or more in brightness etc..Then, primary processor 102 can be in step S206 by features described above
Data is activation is to GPU 104.Recognition of face/the characteristic point determines that the reason for why algorithm performs on CPU 102 is GPU
104 itself possess more control unit (for example, branch prediction, caching etc.) unlike CPU 102, therefore this kind of performing
During algorithm, GPU 104 is generally and efficient not as CPU 102.So, the algorithm can mainly be held by primary processor 102
OK, rather than allowing it to be performed on GPU 104.However, in further embodiments, it is also possible to allow GPU 104 to perform these
Operation.Additionally, the algorithm is not the emphasis paid close attention to herein, and has various algorithms for recognition of face, therefore herein not
It is described in detail again, but this has no effect on those skilled in the art and realizes the embodiment of the present disclosure.Although additionally, making herein
Feature with characteristic point come representative on the face, but it is exactly single point or pixel in image not represent it, and can be to represent tool
There is the region of definite shape and area, such as eye feature point can include shape roughly the same with eye portion in image and face
Long-pending block of pixels.
Next, in step S207 and S208, each characteristic point that GPU 104 can find according to CPU 102 is held
Row image procossing (for example, repairing graphic operation).Image procossing can include it is following at least one：Contour revising is (for example, thin face, eye
Portion amplifies etc.), color correct (for example, the colour of skin, eye color, lip color correct etc.), brightness correction is (for example, facial polishing
Deng) and Fuzzy Processing (for example, Gaussian Blur, or be commonly called as being " mill skin ").Additionally, such as thin face, U.S. face (whole structure
Including for example grinding skin, whitening etc., can specifically include that for example bilateral filtering, rim detection, sharpening, the colour of skin are adjusted, and/or it
One or more) etc the order for repairing graphic operation in his site color/brightness regulation etc. has no a fixed pattern.Although in figure
In 2 illustrated embodiments, thin face operation S207 is first carried out, has then performed U.S. face operation S208, but execution can also have been overturned,
Or even executed in parallel.Additionally, in certain embodiments, at least partly U.S.'s face operation S208 can not need depositing for characteristic point
, and therefore itself or one part (that is, before step S204) can even be carried out before characteristic point confirmation.
Additionally, present inventor has found：In a mobile device, as long as display in a second 30 has the figure of 4K resolution ratio
As just having no interim card sense.According to the embodiment of the present disclosure, the time about 7.5ms that U.S. face is processed using GPU 104 (and is used
CPU 102 at least want more than 750ms performing same operation), time of the thin face for the treatment of is about 12ms, and (including CPU 102 determines
The time of characteristic point about 9.5ms and GPU 104 perform the time about 2.5ms of thin face operation).It is thus possible to realize every frame 4K figures
As treatment only needs about 19.5ms, so that video flowing does not have interim card sense.
Then, in step S209, in the case of the view data of a certain video flowings of first treated of GPU 104, GPU 104
A new texture can be created with processed data or initial data, in the case where height (or resolution ratio) wide is constant,
As long as carrying out a secondary data later updates a data texturing, and need not re-create new texture.So as to reduce data resource
Establishment repeatedly, can further lift calculating performance.Additionally, in the situation of such as switching front camera and rear camera 110-1 and 110-2
Down or in the case of rotation terminal 100 (for example, longitudinal direction becomes horizontal or vice versa), resolution ratio (or width is high) would generally change
Become, and can now re-create new texture.In spite of when be likely to result in the zero point interim card of several seconds, but user is generally also
Will not notice that such interim card.After data texturing treatment is completed, GPU 104 can continue to be gone to be rendered into texture
On GPUBuffer.
In step S210, GPU 104 can be sent in the display number in step S209 creating or updating to display 106
According to enabling a user to see the video by repairing figure on display 106.Meanwhile, in step S210, GPU 104 is also
The display data can be sent to CPU 102 and/or memory 112, to allow that CPU 102 performs subsequent treatment, and made
Obtaining memory 112 can store the display data, so as to realize video record function.
Can respectively be generated for communication unit according to the display data in step S211 and/or S212, CPU 102
108 live data (for example, display data is encapsulated by the various forms of network streaming, is packed) and/or it is directed to
The trace command of camera 110, to realize net cast and/or Object tracking respectively.
So far, described in detail with reference to Fig. 1 and Fig. 2 and video flowing is processed using GPU 104 according to the embodiment of the present disclosure
Overall procedure.By using the method according to the embodiment of the present disclosure, terminal and/or computer program, user can be in operation
U.S. face thin face function is opened when mobile phone, and has no interim card sense in the state of 4K high definitions, moreover it is possible to while carrying out live, record
As, take pictures, show and the operation such as Rotation screen.So as to allow performance to reach most by way of the rational management of GPU/CPU
It is excellent.
The use GPU 104 of terminal 100 according to the embodiment of the present disclosure is described in detail below with reference to Fig. 3~4 come place
Manage the method for video flowing and the functional configuration of terminal 100.
Fig. 3 to be showed and using GPU 104 process video flowing according to what the embodiment of the present disclosure was performed in terminal 100
Method 300 flow chart.As shown in figure 3, method 300 can include step S310, S320 and S330.According to the disclosure, side
Some steps of method 300 can individually perform or combine execution, and can be performed with executed in parallel or order, it is not limited to scheme
Concrete operations shown in 3 are sequentially.In certain embodiments, method 300 can be as shown in Figure 1 terminal 100 and/or processor
102 perform.
Fig. 4 shows the functional block diagram of the exemplary terminal 100 according to the embodiment of the present disclosure.As shown in figure 4, terminal 100
Can include：Image data acquisition apparatus 150, characteristic point determination device 160 and image procossing instruction device 170.
Image data acquisition apparatus 150 can be used for obtaining the view data of at least two field picture in video flowing.Image
Data acquisition facility 150 can be the CPU (for example, CPU 102) of terminal 100, digital signal processor (DSP),
Microprocessor, microcontroller etc., it can be engaged with the camera 110 of such as terminal 100, in acquisition video flowing at least
The view data of one two field picture.Additionally, it can also be engaged with the communication unit 108 of terminal 100 and/or memory 112, obtain
Obtain at least view data of a two field picture and/or storage the regarding in local storage of the video flowing come from external equipment transmission
The view data of an at least two field picture of frequency stream.
Characteristic point determination device 160 is determined at least one of view data characteristic point.Characteristic point determines
Device 160 can also be CPU (for example, CPU 102), digital signal processor (DSP), the microprocessor of terminal 100
Device, microcontroller etc., it can be based on image characteristic point recognizer to determine position, the size of the characteristic point in image
Etc. parameter.For example, during image can be determined according to face characteristic identification parameter whether there is face, and face each face
Position, size, color of portion's organ etc..
Image procossing instruction device 170 can serve to indicate that GPU 104 based on determined by least one characteristic point to this
View data carries out image procossing.Image procossing instruction device 170 can also be the CPU (CPU of terminal 100
102), digital signal processor (DSP), microprocessor, microcontroller etc., it can be for example, by graphical programming interface (example
Such as, OpenGL, Direct3D etc.), GPU 104 is indicated according at least one detected characteristic point to perform at respective image
Reason.For example, the place that GPU 104 can be indicated to detect face acne in the picture performs Fuzzy Processing (for example, Gaussian Blur
Deng), the place that skin is detected in the picture performs the treatment such as brightness correction, color correct, and/or is detecting face contour
Place perform contour revising (for example, thin face etc.) treatment.
Additionally, terminal 100 can also include other functional units not shown in Fig. 4, such as operation instruction device.One
In a little embodiments, operation instruction device can be used for for the view data by image procossing, indicate GPU 104 to be regarded to perform
Frequency recording operation, video display operation, Object tracking operation or net cast operation at least one.Additionally, the operation refers to
Showing device can be also used for：Indicate GPU 104 executed in parallel video record operations, video display operation, Object tracking operate or
At least two in net cast operation.
Additionally, terminal 100 can also include the other functions unit not shown in Fig. 4, this is had no effect on yet with it
Art personnel understand embodiment of the present disclosure, therefore are omitted in fig. 4.For example, terminal 100 can also include with
The next item down or multiple function unit：Power supply, memory (for example, memory 112), data/address bus, antenna, radio receiving-transmitting unit
(for example, communication unit 108) etc..
Below with reference to Fig. 3 and Fig. 4, to the use GPU 104 for performing on the terminal 100 according to the embodiment of the present disclosure come
The method 300 and terminal 100 for processing video flowing are described in detail.
Method 300 starts from step S310, in step S310, can be by the image data acquisition apparatus 150 of terminal 100
Obtain the view data of at least two field picture in video flowing.
In step s 320, at least in the view data can be determined by the characteristic point determination device 160 of terminal 100
Individual characteristic point.
In step S330, GPU 104 can be indicated to be based at least one by the image procossing instruction device 170 of terminal 100
Individual characteristic point carries out image procossing to view data.
In certain embodiments, the video flowing can be the video flowing with 4K or higher resolution.In some embodiments
In, an at least two field picture can include face, and at least one characteristic point can include it is following at least one：Face, eye
Eyeball, nose, mouth, ear, hair or eyebrow.In certain embodiments, image procossing can include it is following at least one：Profile
Amendment, color correct, brightness correction and Fuzzy Processing.In certain embodiments, method 300 can also include：For process
The view data of image procossing, indicates GPU 104 to perform video record operation, video display operation, Object tracking operation or regard
Frequently at least one in live operation.In certain embodiments, GPU 104 is indicated to perform video record operation, video display behaviour
At least one in work, Object tracking operation or net cast operation can include：Indicate the record of the executed in parallel videos of GPU 104
At least two in system operation, video display operation, Object tracking operation or net cast operation.In certain embodiments, exist
Before GPU 104 carries out image procossing based at least one characteristic point to view data, coordinate change can be carried out to view data
Change, allow it to carry out image procossing by GPU 104.In certain embodiments, it is based at least one characteristic point in GPU 104
View data is carried out after image procossing, if GPU104 is to carry out image procossing for video flowing first, can created
New texture, otherwise in the case of the ratio of width to height of view data is immovable, can only update existing texture.In some implementations
In example, if GPU 104 is not the ratio of width to height for carrying out image procossing and view data for video flowing first there occurs change,
New texture can be created.In certain embodiments, the change of the ratio of width to height of view data can be by change camera and/or
Rotation terminal causes.
By using the method according to the embodiment of the present disclosure, terminal and/or computer program, user can be in operating handset
When open the U.S. thin face function of face, and have no interim card sense in the state of 4K high definitions, moreover it is possible to while carrying out live, video recording, clapping
According to, display and the operation such as Rotation screen.So as to allow performance to be optimal by way of the rational management of GPU/CPU.
It should be noted that the function of being realized by pure hardware, pure software and/or firmware is described herein as,
Can also be realized by modes such as the combinations of specialized hardware, common hardware and software.For example, being described as by specialized hardware
(for example, field programmable gate array (FPGA), application specific integrated circuit (ASIC) etc.), can be by common hardware come the function realized
The mode of the combination of (for example, CPU (CPU), digital signal processor (DSP)) and software realizes, otherwise also
So.Additionally, being for example described as the function by realizations such as bluetooth module, NFC chip/coils, it is also possible to by general processor (example
Such as, CPU, DSP etc.) come real with reference to the hardware such as analog to digital conversion circuit, amplifying circuit, antenna and bluetooth, NFC relevant treatments software
Existing, vice versa.
Although the certain exemplary embodiments with reference to the disclosure have shown and described the disclosure, art technology
Personnel it should be understood that in the case of the spirit and scope of the present disclosure limited without departing substantially from appended claims and its equivalent,
The various changes in form and details can be carried out to the disclosure.Therefore, the scope of the present disclosure should not necessarily be limited by above-described embodiment,
But not only should be determined by appended claims, also it is defined by the equivalent of appended claims.
Claims (31)
1. a kind of method for processing video flowing using GPU GPU at the processor of terminal, including：
Obtain the view data of at least two field picture in the video flowing；
Determine at least one of described image data characteristic point；And
Indicate the GPU carries out image procossing based at least one characteristic point to described image data.
2. method according to claim 1, wherein, the video flowing is the video flowing with 4K or higher resolution.
3. method according to claim 1, wherein, an at least two field picture includes face, and described at least one
Characteristic point include it is following at least one：Face, eyes, nose, mouth, ear, hair or eyebrow.
4. method according to claim 1, wherein, described image treatment include it is following at least one：Contour revising, color
Amendment, brightness correction and Fuzzy Processing.
5. method according to claim 1, also includes：For the described image data by image procossing, indicate described
GPU performs at least one during video record operation, video display operation, Object tracking operation or net cast are operated.
6. method according to claim 5, wherein, indicate the GPU perform video record operation, video display operation,
At least one during Object tracking is operated or net cast is operated includes：Indicate the GPU executed in parallel video record to operate,
At least two in video display operation, Object tracking operation or net cast operation.
7. method according to claim 1, wherein, at least one characteristic point is based on to described image in the GPU
Before data carry out image procossing, coordinate transform is carried out to described image data, allow it to carry out image by the GPU
Reason.
8. method according to claim 1, wherein, at least one characteristic point is based on to described image in the GPU
Data are carried out after image procossing, if the GPU is to carry out image procossing for the video flowing first, create new line
Reason, otherwise in the case of the ratio of width to height of described image data is immovable, only updates existing texture.
9. method according to claim 8, wherein, if the GPU is not to carry out image for the video flowing first
The ratio of width to height for the treatment of and described image data there occurs change, then create new texture.
10. method according to claim 8, wherein, the change of the ratio of width to height of described image data is by change camera
And/or rotate what the terminal caused.
A kind of 11. terminals that video flowing is processed using GPU GPU, including：
Image data acquisition apparatus, the view data for obtaining at least two field picture in the video flowing；
Characteristic point determination device, for determining at least one of described image data characteristic point；And
Image procossing instruction device, for indicating the GPU to be carried out to described image data based at least one characteristic point
Image procossing.
12. terminals according to claim 11, wherein, the video flowing is the video flowing with 4K or higher resolution.
13. terminals according to claim 11, wherein, an at least two field picture includes face, and described at least one
Individual characteristic point include it is following at least one：Face, eyes, nose, mouth, ear, hair or eyebrow.
14. terminals according to claim 11, wherein, described image treatment include it is following at least one：Contour revising, face
Colour correction, brightness correction and Fuzzy Processing.
15. terminals according to claim 11, also include：Operation instruction device, for for the institute by image procossing
View data is stated, indicates the GPU to perform video record operation, video display operation, Object tracking operation or net cast behaviour
At least one in work.
16. terminals according to claim 15, wherein, the operation instruction device is additionally operable to：The GPU is indicated to hold parallel
At least two in the operation of row video record, video display operation, Object tracking operation or net cast operation.
17. terminals according to claim 11, wherein, at least one characteristic point is based on to the figure in the GPU
Before carrying out image procossing as data, coordinate transform is carried out to described image data, allow it to carry out image by the GPU
Treatment.
18. terminals according to claim 11, wherein, at least one characteristic point is based on to the figure in the GPU
After carrying out image procossing as data, if the GPU is to carry out image procossing for the video flowing first, create new
Texture, otherwise in the case of the ratio of width to height of described image data is immovable, only updates existing texture.
19. terminals according to claim 18, wherein, if the GPU is not to carry out figure for the video flowing first
As the ratio of width to height for the treatment of and described image data there occurs change, then new texture is created.
20. terminals according to claim 18, wherein, the change of the ratio of width to height of described image data is by change camera
And/or rotate what the terminal caused.
A kind of 21. terminals for processing video flowing, including：
Processor；
GPU GPU；
Memory, store instruction, the instruction causes the processor when by the computing device：
Obtain the view data of at least two field picture in the video flowing；
Determine at least one of described image data characteristic point；And
Indicate the GPU carries out image procossing based at least one characteristic point to described image data.
22. terminals according to claim 21, wherein, the video flowing is the video flowing with 4K or higher resolution.
23. terminals according to claim 21, wherein, an at least two field picture includes face, and described at least one
Individual characteristic point include it is following at least one：Face, eyes, nose, mouth, ear, hair or eyebrow.
24. terminals according to claim 21, wherein, described image treatment include it is following at least one：Contour revising, face
Colour correction, brightness correction and Fuzzy Processing.
25. terminals according to claim 21, wherein, the instruction causes described when by the computing device, also
Processor：For the described image data by image procossing, the GPU is indicated to perform video record operation, video display behaviour
Make, Object tracking operation or net cast operation at least one.
26. terminals according to claim 25, wherein, the instruction causes described when by the computing device, also
Processor：For the described image data by image procossing, the GPU executed in parallel video record operation, video is indicated to show
Show at least two in operation, Object tracking operation or net cast operation.
27. terminals according to claim 21, wherein, at least one characteristic point is based on to the figure in the GPU
Before carrying out image procossing as data, coordinate transform is carried out to described image data, allow it to carry out image by the GPU
Treatment.
28. terminals according to claim 21, wherein, at least one characteristic point is based on to the figure in the GPU
After carrying out image procossing as data, if the GPU is to carry out image procossing for the video flowing first, create new
Texture, otherwise in the case of the ratio of width to height of described image data is immovable, only updates existing texture.
29. terminals according to claim 28, wherein, if the GPU is not to carry out figure for the video flowing first
As the ratio of width to height for the treatment of and described image data there occurs change, then new texture is created.
30. terminals according to claim 28, wherein, the change of the ratio of width to height of described image data is by change camera
And/or rotate what the terminal caused.
31. terminal according to any one of claim 21-30, wherein, the terminal is movable termination.
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/CN2016/107993 WO2018098677A1 (en) | 2016-11-30 | 2016-11-30 | Video stream processing method and terminal |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106688016A true CN106688016A (en) | 2017-05-17 |
Family
ID=58849586
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201680002250.6A Pending CN106688016A (en) | 2016-11-30 | 2016-11-30 | Method and terminal for handling video stream |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN106688016A (en) |
| WO (1) | WO2018098677A1 (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107578371A (en) * | 2017-09-29 | 2018-01-12 | 北京金山安全软件有限公司 | Image processing method and device, electronic equipment and medium |
| CN108495043A (en) * | 2018-04-28 | 2018-09-04 | Oppo广东移动通信有限公司 | Image processing method and relevant apparatus |
| CN108600771A (en) * | 2018-05-15 | 2018-09-28 | 东北农业大学 | Recorded broadcast workstation system and operating method |
| CN109089043A (en) * | 2018-08-30 | 2018-12-25 | Oppo广东移动通信有限公司 | Shoot image pre-processing method, device, storage medium and mobile terminal |
| CN110730335A (en) * | 2019-11-14 | 2020-01-24 | 深圳市高巨创新科技开发有限公司 | Unmanned aerial vehicle video real-time preview method and system |
| CN111164569A (en) * | 2017-08-02 | 2020-05-15 | 深圳传音通讯有限公司 | Filter switching method and filter switching system based on intelligent terminal |
| CN111182350A (en) * | 2019-12-31 | 2020-05-19 | 广州华多网络科技有限公司 | Image processing method, image processing device, terminal equipment and storage medium |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117331512B (en) * | 2023-12-01 | 2024-04-12 | 芯动微电子科技(武汉)有限公司 | Data compression and processing method for executing write operation on GPU (graphics processing unit) nuclear memory |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110085139A1 (en) * | 2009-10-08 | 2011-04-14 | Tobii Technology Ab | Eye-tracking using a gpu |
| CN102572391A (en) * | 2011-12-09 | 2012-07-11 | 深圳市万兴软件有限公司 | Method and device for genius-based processing of video frame of camera |
| CN103685926A (en) * | 2012-09-21 | 2014-03-26 | 宏达国际电子股份有限公司 | Method for image processing of face regions and electronic device by using the same |
| CN104392409A (en) * | 2014-12-01 | 2015-03-04 | 厦门美图之家科技有限公司 | Acceleration method for beautifying image |
| CN105872447A (en) * | 2016-05-26 | 2016-08-17 | 努比亚技术有限公司 | Video image processing device and method |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104331858B (en) * | 2014-11-24 | 2018-02-16 | 厦门美图之家科技有限公司 | Accelerated method a kind of while that image procossing is carried out using CPU and GPU |
| CN106127673B (en) * | 2016-07-19 | 2019-02-12 | 腾讯科技（深圳）有限公司 | A kind of method for processing video frequency, device and computer equipment |
-
2016
- 2016-11-30 CN CN201680002250.6A patent/CN106688016A/en active Pending
- 2016-11-30 WO PCT/CN2016/107993 patent/WO2018098677A1/en active Application Filing
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110085139A1 (en) * | 2009-10-08 | 2011-04-14 | Tobii Technology Ab | Eye-tracking using a gpu |
| CN102572391A (en) * | 2011-12-09 | 2012-07-11 | 深圳市万兴软件有限公司 | Method and device for genius-based processing of video frame of camera |
| CN103685926A (en) * | 2012-09-21 | 2014-03-26 | 宏达国际电子股份有限公司 | Method for image processing of face regions and electronic device by using the same |
| CN104392409A (en) * | 2014-12-01 | 2015-03-04 | 厦门美图之家科技有限公司 | Acceleration method for beautifying image |
| CN105872447A (en) * | 2016-05-26 | 2016-08-17 | 努比亚技术有限公司 | Video image processing device and method |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111164569A (en) * | 2017-08-02 | 2020-05-15 | 深圳传音通讯有限公司 | Filter switching method and filter switching system based on intelligent terminal |
| CN111164569B (en) * | 2017-08-02 | 2020-11-10 | 深圳传音通讯有限公司 | Filter switching method and filter switching system based on intelligent terminal |
| CN107578371A (en) * | 2017-09-29 | 2018-01-12 | 北京金山安全软件有限公司 | Image processing method and device, electronic equipment and medium |
| CN108495043A (en) * | 2018-04-28 | 2018-09-04 | Oppo广东移动通信有限公司 | Image processing method and relevant apparatus |
| CN108495043B (en) * | 2018-04-28 | 2020-08-07 | Oppo广东移动通信有限公司 | Image data processing method and related device |
| CN108600771A (en) * | 2018-05-15 | 2018-09-28 | 东北农业大学 | Recorded broadcast workstation system and operating method |
| CN109089043A (en) * | 2018-08-30 | 2018-12-25 | Oppo广东移动通信有限公司 | Shoot image pre-processing method, device, storage medium and mobile terminal |
| CN109089043B (en) * | 2018-08-30 | 2021-07-30 | Oppo广东移动通信有限公司 | Shot image preprocessing method and device, storage medium and mobile terminal |
| CN110730335A (en) * | 2019-11-14 | 2020-01-24 | 深圳市高巨创新科技开发有限公司 | Unmanned aerial vehicle video real-time preview method and system |
| CN111182350A (en) * | 2019-12-31 | 2020-05-19 | 广州华多网络科技有限公司 | Image processing method, image processing device, terminal equipment and storage medium |
| CN111182350B (en) * | 2019-12-31 | 2022-07-26 | 广州方硅信息技术有限公司 | Image processing method, device, terminal equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2018098677A1 (en) | 2018-06-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106688016A (en) | Method and terminal for handling video stream | |
| CN106331850B (en) | Browser live broadcast client, browser live broadcast system and browser live broadcast method | |
| US11263723B2 (en) | Image warping method and device | |
| US20190213786A1 (en) | Three dimensional content generating apparatus and three dimensional content generating method thereof | |
| US20210158021A1 (en) | Method for processing images and electronic device | |
| US11823327B2 (en) | Method for rendering relighted 3D portrait of person and computing device for the same | |
| WO2020108291A1 (en) | Face beautification method and apparatus, and computer device and storage medium | |
| JP6357589B2 (en) | Image display method, apparatus, program, and recording medium | |
| US20230262320A1 (en) | Shooting method, shooting apparatus, and electronic device | |
| US11194389B2 (en) | Foveated rendering of graphics content using a rendering command and subsequently received eye position data | |
| KR102474088B1 (en) | Method and device for compositing an image | |
| JP2017528215A (en) | Method and system for mediated reality welding | |
| CN107610044A (en) | Image processing method, computer-readable recording medium and virtual reality helmet | |
| WO2014030405A1 (en) | Display device, display method, television receiver, and display control device | |
| CN109639956A (en) | Twin-lens image processing apparatus and method | |
| CN110807769B (en) | Image display control method and device | |
| CN112788254A (en) | Camera image matting method, device, equipment and storage medium | |
| WO2022258013A1 (en) | Image processing method and apparatus, electronic device and readable storage medium | |
| WO2022108067A1 (en) | Method for rendering relighted 3d portrait of person and computing device for the same | |
| CN116055895A (en) | Image processing method and related device | |
| US11636708B2 (en) | Face detection in spherical images | |
| CN113228095A (en) | Image processing method, display processing apparatus, and computer-readable storage medium | |
| TWI630579B (en) | Electronic apparatus, computer readable recording medium storing program and facial image displaying method | |
| WO2023197284A1 (en) | Saliency-based adaptive color enhancement | |
| WO2024088322A1 (en) | Video display method and apparatus, video display device, and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170517 |
|
| RJ01 | Rejection of invention patent application after publication |