CN106846350B - One kind is based on RGB-D camera and stereosonic visually impaired people's barrier early warning system and method - Google Patents
One kind is based on RGB-D camera and stereosonic visually impaired people's barrier early warning system and method Download PDFInfo
- Publication number
- CN106846350B CN106846350B CN201611049463.2A CN201611049463A CN106846350B CN 106846350 B CN106846350 B CN 106846350B CN 201611049463 A CN201611049463 A CN 201611049463A CN 106846350 B CN106846350 B CN 106846350B
- Authority
- CN
- China
- Prior art keywords
- color
- depth
- infrared
- image
- camera
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 230000001771 impaired effect Effects 0.000 title claims abstract description 40
- 230000004888 barrier function Effects 0.000 title claims abstract description 39
- 238000000034 method Methods 0.000 title claims abstract description 28
- 210000000988 bone and bone Anatomy 0.000 claims abstract description 18
- 230000003068 static effect Effects 0.000 claims abstract description 11
- 230000035939 shock Effects 0.000 claims description 12
- 239000003086 colorant Substances 0.000 claims description 8
- 230000000007 visual effect Effects 0.000 claims description 5
- 230000003287 optical effect Effects 0.000 claims description 3
- 230000006870 function Effects 0.000 description 6
- 230000003993 interaction Effects 0.000 description 5
- 241001166076 Diapheromera femorata Species 0.000 description 4
- 239000011664 nicotinic acid Substances 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000002452 interceptive effect Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000002513 implantation Methods 0.000 description 2
- 238000001931 thermography Methods 0.000 description 2
- 239000002023 wood Substances 0.000 description 2
- 201000004569 Blindness Diseases 0.000 description 1
- 241001465754 Metazoa Species 0.000 description 1
- 208000028389 Nerve injury Diseases 0.000 description 1
- 201000007737 Retinal degeneration Diseases 0.000 description 1
- 206010047571 Visual impairment Diseases 0.000 description 1
- 206010064930 age-related macular degeneration Diseases 0.000 description 1
- 230000003796 beauty Effects 0.000 description 1
- 238000005266 casting Methods 0.000 description 1
- 230000019771 cognition Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 239000000428 dust Substances 0.000 description 1
- 210000005069 ears Anatomy 0.000 description 1
- 230000005611 electricity Effects 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 239000004615 ingredient Substances 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000008764 nerve damage Effects 0.000 description 1
- 230000007935 neutral effect Effects 0.000 description 1
- 210000001328 optic nerve Anatomy 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R3/00—Circuits for transducers, loudspeakers or microphones
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- G10H1/00—Details of electrophonic musical instruments
- G10H1/0091—Means for obtaining special acoustic effects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/204—Image signal generators using stereoscopic image cameras
- H04N13/25—Image signal generators using stereoscopic image cameras using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10048—Infrared image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20021—Dividing image into blocks, subimages or windows
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Acoustics & Sound (AREA)
- Signal Processing (AREA)
- Multimedia (AREA)
- Controls And Circuits For Display Device (AREA)
Abstract
The invention discloses one kind based on RGB-D camera and stereosonic visually impaired people's barrier early warning system and method.This method projects sightless near-infrared static speckles using infrared projector, image is acquired using two infrared cameras and a RGB camera, compact processor handles the image of acquisition, it calculates and obtains depth image, and piecemeal is carried out to depth image, the depth information that piecemeal is obtained is converted into stereo signal, visually impaired people finally is passed to using bone conduction earphone to assist, and can meet the requirement of visually impaired people's barrier early warning well.
Description
Technical field
The invention belongs to visually impaired people's ancillary technique, binocular vision technology, three-dimensional environment cognition technology, stereo interactive skills
Art field.The present invention includes a kind of based on RGB-D camera and stereosonic visually impaired people's barrier method for early warning, is related to a kind of benefit
Sightless near-infrared static speckles are projected with using infrared projector, are acquired using two infrared cameras and a RGB camera
Image, compact processor handle the image of acquisition, obtain depth image, and carry out piecemeal to depth image, finally will
The depth information of piecemeal is converted into stereo signal, finally passes to the barrier that visually impaired people is assisted using bone conduction earphone
Hinder object method for early warning.
Background technique
It is counted according to world health organization, there are 2.85 hundred million dysopia personages in the whole world.Visually impaired people has lost normally
Vision, all highly difficult to the understanding of color, shape, distance, movement, their live in daily life, trip etc. all receives
Tremendous influence.
Traditional visually impaired people's auxiliary tool such as bland person walking stick, visually impaired people need to move repeatedly walking stick, can just know
It is the case where in front, both time-consuming or laborious.The detection range of bland person walking stick is limited, can only detect the barrier on foot side, also can not be anti-
Reflect distant place and aerial situation.Seeing-eye dog can provide help for visually impaired people, but the training of seeing-eye dog and maintenance costs are
High, average family is difficult to undertake.Some occasions, seeing-eye dog cannot accompany blind person's entrance, such as bus and railway station, because
The auxiliary of this seeing-eye dog is limited.Bionic eye can help visually impaired people's recovered part vision, but the implantation of bionic eye
It needs to perform the operation, expense is high.Bionic eye, which is only applicable to retinal pigment degeneration or senile macular degeneration, leads to the blind person of blindness.
The visually impaired people of optic nerve damage can not be by implantation bionic eye come recovered part vision.
The auxiliary tool visually impaired of electronic type mainly uses ultrasonic technology, laser ranging technique, binocular vision technology, laser
Speckle encoding technology, laser radar technique, millimetre-wave radar technology, thermal imaging, global positioning system (GPS).Based on super
The ranging range of technology of acoustic wave and laser ranging technique is limited, can only realize single-point ranging, the information content of acquisition is very little, and power consumption
More, equipment is heavy, can only realize warning function, is easy by environmental disturbances.Auxiliary based on binocular vision technology depends on environment
The abundant degree of middle characteristic point and texture, the scene failure single for some textures, such as indoor white wall, smooth earth etc..
Binocular vision technology can be by the deception of the special cases such as mirror-reflection, to cause to fail to judge or judge by accident.It is compiled based on laser speckle
The auxiliary of code technology is in outdoor failure, because actively the structure light of projection is flooded by sunlight, so that the speckle of coding can not be identified.
Laser speckle coding techniques is since by power limit, there are maximum distances, and the object more than maximum distance can not be to its ranging.
Supplementary costs based on laser radar technique is high, and usual sample rate is low, sensitive to dust, haze, rainwater, and can not obtain face
Color and texture information.Auxiliary resolution ratio based on millimetre-wave radar is low, and signal processing is difficult.Auxiliary based on thermal imaging
Resolution ratio is low, and calibration process is complicated, and can only detect the thermal objects such as humans and animals.Auxiliary precision based on GPS is low, has letter
Number loss, cannot use indoors, and the obstacle information of local dynamic station can not be obtained.
The interactive mode of traditional visually impaired people's auxiliary mainly has voice prompting, tactile vibration.Semanteme prompt usually casting barrier
Distance and the direction for hindering object, need the regular hour to play, and cause delay and accident risk, and transferable information content is few.
Tactile vibration is hardware by vibration waistband or vibration vest, and to shake the orientation to prompt barrier, shaking device can be with
It solves the problems, such as delay, but brings burden to visually impaired people, the wearing feeling of different people is by difference.
Summary of the invention
In view of the above-mentioned deficiencies in the prior art, it is an object of the present invention to provide a kind of based on RGB-D camera and stereosonic view
Hinder personage's barrier early warning system and method.
The purpose of the present invention is what is be achieved through the following technical solutions: a kind of based on RGB-D camera and stereosonic visually impaired
Personage's barrier early warning system, the system include an infrared projector, two identical infrared cameras, a colored phase
Machine, a usb hub, a compact processor, a bone conduction earphone module, two osteoacusis shock modules, an electricity
Pond module.Infrared projector, two infrared cameras, color camera are connected by usb hub with compact processor, battery module
It is connected with compact processor.Color camera and infrared projector are located between two infrared cameras.Two infrared cameras, colored phase
The optical axis of machine is parallel to each other.Compact processor controls the infrared projector sightless static near-infrared of three-dimensional scenic projection forwards
Speckle, two infrared cameras acquire two width infrared images of the three-dimensional scenic after projecting in real time, and color camera is adopted in real time
Collect the color image of three-dimensional scenic.Two width infrared images and a width color image are transmitted to compact processor by usb hub.It is small-sized
Processor handles two width infrared images, the width color image of acquisition, obtains the depth image of three-dimensional scenic.Small-sized place
It manages device and piecemeal is carried out to depth image, and convert stereo signal for the depth information after piecemeal, and be transmitted to bone conduction earphone
Module.Stereo signal is switched to osteoacusis vibration signal by bone conduction earphone module, is transmitted to two osteoacusis shock modules.Two
Osteoacusis shock module, transmitting osteoacusis vibration signal give user visually impaired, realize barrier early warning.
A kind of barrier method for early warning, comprising the following steps:
(1) a binocular camera calibration is carried out to two infrared cameras, obtains the focal length f of two infrared camerasIR, left infrared
Principle point location (the c of cameraIR-x,cIR-y), the parallax range B of two infrared camerasIR-IR。
(2) camera calibration is carried out to color camera, obtains the focal length f of color cameracolor, principle point location (cCOLOR-x,
cCOLOR-y)。
(3) a binocular camera calibration is carried out to the infrared camera of color camera and left side, obtains left infrared camera and coloured silk
The parallax range B of form and aspect machineIR-COLOR。
(4) infrared projector projects sightless static near-infrared speckle into three-dimensional scenic in real time.
Two infrared image IR of (5) two infrared camera acquisition three-dimensional scenicsleftAnd IRright。
(6) the color image Color of color camera three-dimensional scenic.
(7) usb hub is by two width infrared image IRleftAnd IRright, a width color image Color is transmitted to small-sized processing
Device.
(8) compact processor is to two infrared image IRleftAnd IRrightThe edge Sobel is extracted, two sides Sobel are obtained
Edge image SobelleftAnd Sobelright。
(9) with left Sobel edge image SobelleftOn the basis of, to two Sobel edge image SobelleftWith
SobelrightImages match image block based is carried out, a series of available point E={ e matched are obtained1,e2,e3,...,eM}。
In left Sobel edge image SobelleftIn, each available point is e=(u, v, d)T, u is abscissa pixel value, and v is vertical sits
Pixel value is marked, d is parallax value.
(10) on the basis of the available point E matched, every three available points may make up a disparity plane, i-th of parallax
The equation of plane is d=aiu+biv+ci, wherein ai, bi, ciFor the coefficient of i-th of disparity plane.
It (11), will not matched pixel (u', v', d') on the basis of these disparity planesTBeing converted to matched has
It imitates point (u, v, d)T；Specifically: the pixel (u', v', d')TDistance to the i-th disparity plane isIf energy function isWherein ε, σ are constant.
To the pixel, all parallax value d'={ d' of disparity search range are traversedmin,...,d'max, it finds out so that energy function
Energy (d') the smallest parallax value, as the parallax value d of the pixel.In addition, u=u', v=v'.
(12) all not matched pixels are traversed, the parallax value of each not matched pixel is obtained, are obtained with left red
Anaglyph Disparity on the basis of outer cameraleft。
(13) according to the focal length f of two infrared camerasIRWith parallax range BIR-IR, traverse anaglyph in every bit (u,
V, d), depth value isTherefore depth image DepthleftMiddle every bit corresponds to (u, v, depth), from
And obtain the depth image Depth on the basis of left infrared cameraleft。
(14) depth image Depth is utilizedleftWith color image Color, the focal length f of two infrared camerasIR, left infrared phase
Principle point location (the c of machineIR-x,cIR-y), the focal length f of color cameracolor, principle point location (cCOLOR-x,cCOLOR-y), and it is left infrared
The parallax range B of camera and color cameraIR-COLOR, depth image and color image can be aligned, obtain color camera view
The depth image Depth of fieldcolor。
(15) by the depth image Depth of color camera visual fieldcolorFrom left to right it is divided into K block, calculates each piece of depth map
As DepthKMean depth depthK.(value of K is generally between 2 to 10)
(16) K block depth image Depth is indicated with the (instrumental) ensemble of the musical instrument of different tone colorsK: the depth image of different masses
DepthKWith the instruments sound of different tone colors.The mean depth depth of the loudness Volume of musical instrument sound and each block of imageKAt
Inverse ratio, it may be assumed that barrier is closer, mean depth depthKSmaller, loudness Volume is bigger；Barrier is remoter, mean depth depthK
Bigger, loudness Volume is smaller.The musical instrument sound in each direction is stereo.The optional piano of musical instrument, violin, gong, trumpet, wood
Qin etc. has special tone color and melodious.
(17) stereo signal is transmitted to bone conduction earphone module by compact processor.
(18) stereo signal is switched to osteoacusis vibration signal by bone conduction earphone module.
(19) osteoacusis vibration signal is passed to user visually impaired by osteoacusis shock module.
This method is essentially consisted in compared to the advantage of previous visually impaired people's householder method:
1, ambient adaptability.Due to using infrared projector and two infrared cameras, this method indoors, outdoor environment all
Use can be compatible with.When indoors, the static near-infrared hot spot of infrared projector projection increases texture to three-dimensional scenic, favorably
In the depth image that acquisition is dense.At outdoor, the near-infrared part of sunlight and three-dimensional scenic are combined, and are conducive to obtain dense
Depth image.Dense depth image can guarantee the accuracy of piecemeal depth and the experience effect of auxiliary interaction.
2, applicability day and night.Due to using infrared projector and two infrared cameras, this method daytime, night all
Use can be compatible with.During the day, the near-infrared ingredient in the static near-infrared hot spot and daylight of infrared projector projection all may be used
To increase texture to three-dimensional scenic, be conducive to dense depth image.At night, the static near-infrared of infrared projector projection
Hot spot increases texture to nearby three-dimensional scenic, also the depth image of available nearby three-dimensional scenic.This method is in day and night
Reliable depth image can be obtained, to guarantee the accuracy of piecemeal depth and the experience effect of auxiliary interaction.
3, it can detect aerial barrier.Traditional bland person walking stick can only detect the road conditions on foot side, and it is empty cannot to look after front
In situation, the barrier that this method can be aerial with early warning, to prevent user visually impaired from colliding the objects such as aerial branch or door beam
Body.
4, ears are not take up.This method delivers a signal to user visually impaired using bone conduction earphone, does not interfere user to listen to outer
The sound on boundary.Visually impaired people is dependent on extraneous sound to carry out some interpretations, as judged road according to wagon flow sound mostly
Towards etc..
5, both hands are not take up.The auxiliary device of this method is wearable, and compact processor is portable, can be placed on pocket or small
Bao Li, both will not bring very big burden to visually impaired, take auxiliary tool with hand without visually impaired people.
6, user will not be bothered.The stereo interactive mode of this method will not give use visually impaired using melodious instruments sound
Family causes to bother, and allows user visually impaired to listen melodious music energy avoidance when in use current.
7, sufficient information content is fed back.Compared to the voice broadcast of semantic formula, stereosonic interaction feedback utilizes different loudness,
The musical instrument of different tone colors indicates the distance of barrier, can simultaneously, fully transmit the road conditions of front different directions, early warning barrier
Hinder the presence of object.
8, learnability and Comprehensible.Compared to the acoustic coding of complex form, the interaction of neutral body sound of the present invention is based on deep
The piecemeal of figure is spent, treated that depth information will not be very lengthy and jumbled for piecemeal, and user visually impaired can learn rapidly and understand stereo
The meaning of signal, and according to stereo signal avoiding obstacles.
9, it timely feeds back.Compared to the voice broadcast of semantic formula, stereosonic interaction feedback is timely, not postpone.
To visually impaired people can timely avoiding obstacles, ensure that the safety of this method.
10, there is good forewarning function to barrier nearby.When close to barrier, the mean depth of corresponding direction
It is small, therefore the loudness of corresponding musical instrument is big, user visually impaired can be helped to predict the approaching of barrier well, to avoid urgent
Barrier.
11, tiny barrier can be detected.When there is tiny barrier, the mean depth of corresponding direction is changed,
Therefore the loudness of corresponding musical instrument is big, user visually impaired can be helped to predict the presence of tiny barrier well, to avoid tiny
Barrier.
Detailed description of the invention
Fig. 1 is the module connection diagram of visually impaired people's barrier early warning system；
Fig. 2 is the structural schematic diagram of visually impaired people's barrier early warning system；
Fig. 3 is the depth image Depth of color image Color and color camera visual fieldcolor, depth image pseudo-colours table
Show, color is more blue, and depth is bigger, and color is redder, and depth is smaller.
Fig. 4 is K block depth image DepthK, each piece of color indicates each piece of depth image DepthKMean depth
depthK, mean depth is bigger, and color is more blue, and mean depth is closer, and color is redder.
Fig. 5, which is that musical instrument is stereo, instrumental ensembles the schematic diagram for indicating barrier.
Specific embodiment
As shown in Figure 1, a kind of be based on RGB-D camera and stereosonic visually impaired people's barrier early warning system, the system
Include an infrared projector, two identical infrared cameras, a color camera, a usb hub, a small-sized processing
Device, a bone conduction earphone module, two osteoacusis shock modules, a battery module.Infrared projector, two infrared phases
Machine, color camera are connected by usb hub with compact processor, and battery module is connected with compact processor.Color camera and
Infrared projector is located between two infrared cameras.Two infrared cameras, the optical axis of color camera are parallel to each other.Compact processor
Controlling infrared projector, the sightless static near-infrared speckle of three-dimensional scenic projection, two infrared cameras acquire in real time forwards
Two width infrared images of the three-dimensional scenic after projecting, color camera acquire the color image of three-dimensional scenic in real time.USB set line
Two width infrared images and a width color image are transmitted to compact processor by device.Compact processor to two width infrared images of acquisition,
One width color image is handled, and the depth image of three-dimensional scenic is obtained.Compact processor carries out piecemeal to depth image, and will
Depth information after piecemeal is converted into stereo signal, and is transmitted to bone conduction earphone module.Bone conduction earphone module will be stereo
Signal switchs to osteoacusis vibration signal, is transmitted to two osteoacusis shock modules.Two osteoacusis shock modules, transmitting osteoacusis shake
Dynamic signal gives user visually impaired, realizes barrier early warning.The system can be designed analogous to glasses described in Fig. 2, to reach beauty
See effect.
The method for early warning of above system the following steps are included:
(1) a binocular camera calibration is carried out to two infrared cameras, obtains the focal length f of two infrared camerasIR, left infrared
Principle point location (the c of cameraIR-x,cIR-y), the parallax range B of two infrared camerasIR-IR。
(2) camera calibration is carried out to color camera, obtains the focal length f of color cameracolor, principle point location (cCOLOR-x,
cCOLOR-y)。
(3) a binocular camera calibration is carried out to the infrared camera of color camera and left side, obtains left infrared camera and coloured silk
The parallax range B of form and aspect machineIR-COLOR。
(4) infrared projector projects sightless static near-infrared speckle into three-dimensional scenic in real time.
Two infrared image IR of (5) two infrared camera acquisition three-dimensional scenicsleftAnd IRright。
(6) the color image Color of color camera three-dimensional scenic.
(7) usb hub is by two width infrared image IRleftAnd IRright, a width color image Color is transmitted to small-sized processing
Device.
(8) compact processor is to two infrared image IRleftAnd IRrightThe edge Sobel is extracted, two sides Sobel are obtained
Edge image SobelleftAnd Sobelright。
(9) with left Sobel edge image SobelleftOn the basis of, to two Sobel edge image SobelleftWith
SobelrightImages match image block based is carried out, a series of available point E={ e matched are obtained1,e2,e3,...,eM}。
In left Sobel edge image SobelleftIn, each available point is e=(u, v, d)T, u is abscissa pixel value, and v is vertical sits
Pixel value is marked, d is parallax value.
(10) on the basis of the available point E matched, every three available points may make up a disparity plane, i-th of parallax
The equation of plane is d=aiu+biv+ci, wherein ai, bi, ciFor the coefficient of i-th of disparity plane.
It (11), will not matched pixel (u', v', d') on the basis of these disparity planesTBeing converted to matched has
It imitates point (u, v, d)T；Specifically: the pixel (u', v', d')TDistance to the i-th disparity plane isIf energy function isWherein ε, σ are constant.
To the pixel, all parallax value d'={ d' of disparity search range are traversedmin,...,d'max, it finds out so that energy function
Energy (d') the smallest parallax value, as the parallax value d of the pixel.In addition, u=u', v=v'.
(12) all not matched pixels are traversed, the parallax value of each not matched pixel is obtained, are obtained with left red
Anaglyph Disparity on the basis of outer cameraleft。
(13) according to the focal length f of two infrared camerasIRWith parallax range BIR-IR, traverse anaglyph in every bit (u,
V, d), depth value isTherefore depth image DepthleftMiddle every bit corresponds to (u, v, depth), from
And obtain the depth image Depth on the basis of left infrared cameraleft。
(14) depth image Depth is utilizedleftWith color image Color, the focal length f of two infrared camerasIR, left infrared phase
Principle point location (the c of machineIR-x,cIR-y), the focal length f of color cameracolor, principle point location (cCOLOR-x,cCOLOR-y), and it is left infrared
The parallax range B of camera and color cameraIR-COLOR, depth image and color image can be aligned, obtain color camera view
The depth image Depth of fieldcolor。
(15) by the depth image Depth of color camera visual fieldcolorFrom left to right it is divided into K block, calculates each piece of depth map
As DepthKMean depth depthK.(value of K is generally between 2 to 10)
(16) K block depth image Depth is indicated with the (instrumental) ensemble of the musical instrument of different tone colorsK: the depth image of different masses
DepthKWith the instruments sound of different tone colors.The mean depth depth of the loudness Volume of musical instrument sound and each block of imageKAt
Inverse ratio, it may be assumed that barrier is closer, mean depth depthKSmaller, loudness Volume is bigger；Barrier is remoter, mean depth depthK
Bigger, loudness Volume is smaller.The musical instrument sound in each direction is stereo.The optional piano of musical instrument, violin, gong, trumpet, wood
Qin etc. has special tone color and melodious.
(17) stereo signal is transmitted to bone conduction earphone module by compact processor.
(18) stereo signal is switched to osteoacusis vibration signal by bone conduction earphone module.
(19) osteoacusis vibration signal is passed to user visually impaired by osteoacusis shock module.
Claims (1)
1. one kind is based on RGB-D camera and stereosonic visually impaired people's barrier early warning system, which is characterized in that the system packet
Containing an infrared projector, two identical infrared cameras, a color camera, a usb hub, a small-sized processing
Device, a bone conduction earphone module, two osteoacusis shock modules, a battery module；Infrared projector, two infrared phases
Machine, color camera are connected by usb hub with compact processor, and battery module is connected with compact processor；Color camera and
Infrared projector is located between two infrared cameras；Two infrared cameras, the optical axis of color camera are parallel to each other；Compact processor
Controlling infrared projector, the sightless static near-infrared speckle of three-dimensional scenic projection, two infrared cameras acquire in real time forwards
Two width infrared images of the three-dimensional scenic after projecting, color camera acquire the color image of three-dimensional scenic in real time；USB set line
Two width infrared images and a width color image are transmitted to compact processor by device；Compact processor to two width infrared images of acquisition,
One width color image is handled, and the depth image of three-dimensional scenic is obtained；Compact processor carries out piecemeal to depth image, and will
Depth information after piecemeal is converted into stereo signal, and is transmitted to bone conduction earphone module；Bone conduction earphone module will be stereo
Signal switchs to osteoacusis vibration signal, is transmitted to two osteoacusis shock modules；Two osteoacusis shock modules, transmitting osteoacusis shake
Dynamic signal gives user visually impaired, realizes barrier early warning；The barrier method for early warning of the system, comprising the following steps:
(1) a binocular camera calibration is carried out to two infrared cameras, obtains the focal length f of two infrared camerasIR, left infrared camera
Principle point location (cIR-x,cIR-y), the parallax range B of two infrared camerasIR-IR；
(2) camera calibration is carried out to color camera, obtains the focal length f of color cameracolor, principle point location (cCOLOR-x,
cCOLOR-y)；
(3) a binocular camera calibration is carried out to the infrared camera of color camera and left side, obtains left infrared camera and colored phase
The parallax range B of machineIR-COLOR；
(4) infrared projector projects sightless static near-infrared speckle into three-dimensional scenic in real time；
Two infrared image IR of (5) two infrared camera acquisition three-dimensional scenicsleftAnd IRright；
(6) the color image Color of color camera acquisition three-dimensional scenic；
(7) usb hub is by two width infrared image IRleftAnd IRright, a width color image Color is transmitted to compact processor；
(8) compact processor is to two infrared image IRleftAnd IRrightThe edge Sobel is extracted, two Sobel edge graphs are obtained
As SobelleftAnd Sobelright；
(9) with left Sobel edge image SobelleftOn the basis of, to two Sobel edge image SobelleftAnd SobelrightInto
Row images match image block based, obtains a series of available point E={ e matched1,e2,e3,...,eM}；In left Sobel
Edge image SobelleftIn, each available point is e=(u, v, d)T, u is abscissa pixel value, and v is ordinate pixel value, d
For parallax value；
(10) on the basis of the available point E matched, every three available points may make up a disparity plane, i-th of disparity plane
Equation be d=aiu+biv+ci, wherein ai, bi, ciFor the coefficient of i-th of disparity plane；
It (11), will not matched pixel (u', v', d') on the basis of these disparity planesTBe converted to matched available point
(u,v,d)T；Specifically: the pixel (u', v', d')TDistance to the i-th disparity plane is
If energy function isWherein ε, σ are constant；To the pixel, traverses parallax and search
All parallax value d'={ d' of rope rangemin,...,d'max, it finds out so that energy function Energy (d') the smallest parallax value,
As the parallax value d of the pixel；In addition, u=u', v=v'；
(12) all not matched pixels are traversed, the parallax value of each not matched pixel is obtained, are obtained with left infrared phase
Anaglyph Disparity on the basis of machineleft；
(13) according to the focal length f of two infrared camerasIRWith parallax range BIR-IR, the every bit (u, v, d) in anaglyph is traversed,
Its depth value isTherefore depth image DepthleftMiddle every bit corresponds to (u, v, depth), to obtain
Obtain the depth image Depth on the basis of left infrared cameraleft；
(14) depth image Depth is utilizedleftWith color image Color, the focal length f of two infrared camerasIR, left infrared camera
Principle point location (cIR-x,cIR-y), the focal length f of color cameracolorWith principle point location (cCOLOR-x,cCOLOR-y) and left infrared camera
With the parallax range B of color cameraIR-COLOR, depth image and color image can be aligned, obtain color camera visual field
Depth image Depthcolor；
(15) by the depth image Depth of color camera visual fieldcolorFrom left to right it is divided into K block, calculates each piece of depth image
DepthKMean depth depthK；
(16) K block depth image Depth is indicated with the (instrumental) ensemble of the musical instrument of different tone colorsK: the depth image Depth of different massesKWith
The instruments sound of different tone colors；The mean depth depth of the loudness Volume of musical instrument sound and each block of imageKIt is inversely proportional, it may be assumed that
Barrier is closer, mean depth depthKSmaller, loudness Volume is bigger；Barrier is remoter, mean depth depthKIt is bigger, it rings
It is smaller to spend Volume；The musical instrument sound in each direction is stereo；
(17) stereo signal is transmitted to bone conduction earphone module by compact processor；
(18) stereo signal is switched to osteoacusis vibration signal by bone conduction earphone module；
(19) osteoacusis vibration signal is passed to user visually impaired by osteoacusis shock module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611049463.2A CN106846350B (en) | 2016-11-23 | 2016-11-23 | One kind is based on RGB-D camera and stereosonic visually impaired people's barrier early warning system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611049463.2A CN106846350B (en) | 2016-11-23 | 2016-11-23 | One kind is based on RGB-D camera and stereosonic visually impaired people's barrier early warning system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106846350A CN106846350A (en) | 2017-06-13 |
| CN106846350B true CN106846350B (en) | 2019-09-24 |
Family
ID=59145877
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611049463.2A Expired - Fee Related CN106846350B (en) | 2016-11-23 | 2016-11-23 | One kind is based on RGB-D camera and stereosonic visually impaired people's barrier early warning system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106846350B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107682689A (en) * | 2017-09-29 | 2018-02-09 | 舜宇光学（浙江）研究院有限公司 | 3 D scanning system based on structure light |
| CN108876836B (en) * | 2018-03-29 | 2021-08-27 | 北京旷视科技有限公司 | Depth estimation method, device and system and computer readable storage medium |
| CN108510538B (en) * | 2018-03-30 | 2020-01-17 | 百度在线网络技术（北京）有限公司 | Three-dimensional image synthesis method and device and computer-readable storage medium |
| CN108460368B (en) * | 2018-03-30 | 2021-07-09 | 百度在线网络技术（北京）有限公司 | Three-dimensional image synthesis method and device and computer-readable storage medium |
| CN113379817B (en) * | 2021-01-12 | 2022-11-04 | 四川深瑞视科技有限公司 | Depth information acquisition method, device and system based on speckles |
| CN114419697B (en) * | 2021-12-23 | 2022-12-02 | 北京深睿博联科技有限责任公司 | Vision-impaired person prompting method and device based on mechanical vibration |
| CN117765499B (en) * | 2023-12-30 | 2024-06-25 | 武汉奥思工业设计有限公司 | Intelligent decision method and system for auxiliary driving of vehicle |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104463880A (en) * | 2014-12-12 | 2015-03-25 | 中国科学院自动化研究所 | RGB-D image acquisition method |
| CN104616284A (en) * | 2014-12-09 | 2015-05-13 | 中国科学院上海技术物理研究所 | Pixel-level alignment algorithm for color images to depth images of color depth camera |
| CN104899869A (en) * | 2015-05-14 | 2015-09-09 | 浙江大学 | Plane and barrier detection method based on RGB-D camera and attitude sensor |
| CN204766392U (en) * | 2015-05-14 | 2015-11-18 | 广州龙天软件科技有限公司 | Lead blind information processing apparatus |
| CN105701811A (en) * | 2016-01-12 | 2016-06-22 | 浙江大学 | Sound coding interaction method based on RGB-IR camera |
| CN105787442A (en) * | 2016-02-19 | 2016-07-20 | 电子科技大学 | Visual interaction based wearable auxiliary system for people with visual impairment, and application method thereof |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9117274B2 (en) * | 2011-08-01 | 2015-08-25 | Fuji Xerox Co., Ltd. | System and method for interactive markerless paper documents in 3D space with mobile cameras and projectors |
-
2016
- 2016-11-23 CN CN201611049463.2A patent/CN106846350B/en not_active Expired - Fee Related
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104616284A (en) * | 2014-12-09 | 2015-05-13 | 中国科学院上海技术物理研究所 | Pixel-level alignment algorithm for color images to depth images of color depth camera |
| CN104463880A (en) * | 2014-12-12 | 2015-03-25 | 中国科学院自动化研究所 | RGB-D image acquisition method |
| CN104899869A (en) * | 2015-05-14 | 2015-09-09 | 浙江大学 | Plane and barrier detection method based on RGB-D camera and attitude sensor |
| CN204766392U (en) * | 2015-05-14 | 2015-11-18 | 广州龙天软件科技有限公司 | Lead blind information processing apparatus |
| CN105701811A (en) * | 2016-01-12 | 2016-06-22 | 浙江大学 | Sound coding interaction method based on RGB-IR camera |
| CN105787442A (en) * | 2016-02-19 | 2016-07-20 | 电子科技大学 | Visual interaction based wearable auxiliary system for people with visual impairment, and application method thereof |
Non-Patent Citations (1)
| Title |
|---|
| "A Ground and Obstacle Detection Algorithm for the Visually Impaired";Ruiqi Cheng 等;《ICBISP 2015》;20151119;第1-6页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106846350A (en) | 2017-06-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106846350B (en) | One kind is based on RGB-D camera and stereosonic visually impaired people's barrier early warning system and method | |
| CN106203390B (en) | A kind of intelligent blind auxiliary system | |
| CN106597690B (en) | One kind predicting glasses based on RGB-D camera and stereosonic visually impaired people's access | |
| CN106817577B (en) | One kind is based on RGB-D cameras and stereosonic visually impaired people's barrier early warning glasses | |
| CN107341789A (en) | One kind is based on RGB D cameras and stereosonic visually impaired people's path precognition system and method | |
| CN106859929B (en) | A kind of Multifunctional blind person guiding instrument based on binocular vision | |
| CN106214437A (en) | A kind of intelligent blind auxiliary eyeglasses | |
| CN106821692B (en) | One kind is based on RGB-D camera and stereosonic visually impaired people's stair detection system and method | |
| CN106843491A (en) | Smart machine and electronic equipment with augmented reality | |
| CN108245385A (en) | A kind of device for helping visually impaired people's trip | |
| Patel et al. | Multisensor-based object detection in indoor environment for visually impaired people | |
| Sharma et al. | International journal of engineering sciences & research technology a review on obstacle detection and vision | |
| CN105686936A (en) | Sound coding interaction system based on RGB-IR camera | |
| CN109106563A (en) | A kind of automation blind-guide device based on deep learning algorithm | |
| Hsieh et al. | Outdoor walking guide for the visually-impaired people based on semantic segmentation and depth map | |
| Hub et al. | Interactive tracking of movable objects for the blind on the basis of environment models and perception-oriented object recognition methods | |
| CN105701811B (en) | A kind of acoustic coding exchange method based on RGB-IR cameras | |
| Oliveira | The path force feedback belt | |
| CN108125776A (en) | A kind of glasses for guiding blind | |
| CN107049717B (en) | One kind is based on RGB-D camera and stereosonic visually impaired people's zebra stripes detection system and method | |
| Shaikh et al. | Smart Helmet for Visually Impaired | |
| Ghatwary et al. | Intelligent assistance system for visually impaired/blind people (ISVB) | |
| CN117323185A (en) | Blind person indoor navigation system and method based on computer vision and training method | |
| TW202108102A (en) | Intelligent assistant system for the visually impaired | |
| Kandoth et al. | Dhrishti: a visual aiding system for outdoor environment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 9, 181, 310000, Wuchang Road, Wuchang Street, Yuhang District, Zhejiang, Hangzhou, 202-7 Applicant after: HANGZHOU KR-VISION TECHNOLOGY CO.,LTD. Address before: Room 589, C building, No. 525 Xixi Road, Xihu District, Zhejiang, Hangzhou 310012, China Applicant before: HANGZHOU KR-VISION TECHNOLOGY CO.,LTD. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20200819 Address after: 030032 a243, 4th floor, No.4 building, incubation base, No.11 Kangshou street, Taiyuan City, Shanxi Province Patentee after: Shanxi shikrypton Technology Co.,Ltd. Address before: 9, 181, 310000, Wuchang Road, Wuchang Street, Yuhang District, Zhejiang, Hangzhou, 202-7 Patentee before: HANGZHOU KR-VISION TECHNOLOGY Co.,Ltd. |
|
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20190924 |