WO2015167411A1

WO2015167411A1 - Smart navigation system for brainwave controlled wheelchairs

Info

Publication number: WO2015167411A1
Application number: PCT/TR2015/000173
Authority: WO
Inventors: Lütfi MUTLU
Original assignee: Mutlu Lütfi
Priority date: 2014-04-29
Filing date: 2015-04-28
Publication date: 2015-11-05

Abstract

This invention relates to smart navigation system which consist of hardware and software to provide autonomous transportation of brainwave controlled wheelchairs which can be used by disabled people who can't use their hands and feet to indoor and outdoor targets which given by the user.

Description

DESCRIPTION

SMART NAVIGATION SYSTEM FOR BRAINWAVE CONTROLLED WHEELCHAIRS Technical Field

This invention relates to smart navigation system which consist of hardware and software to provide autonomous transportation of brainwave controlled wheelchairs which can be used by disabled people who can't use their hands and feet, paralytic, MS, ALS patients etc. to the indoor and/or outdoor given targets/coordinates which given by the user.

Prior Art

Mobile vehicle guidance systems are the systems that provide or assist user to move vehicle where he/she wants to move or autonomously guide/control vehicle movements. These systems can be fully autonomous, without user control or time to time can be semi autonomous systems which require user intervention in achieving target/targets given by user. In existing mobile vehicle guidance systems, vehicles often pass obstacles which are in front of it by circumnavigating or following lines/marks which have to follow. Vehicles with obstacle avoidance algorithms generally pass obstacles by moving autonomously according to position/s of the obstacle/s which is/are in the vehicle's angle of vision. Vehicles can handle a dead end road even if they pass obstacle/s or it can cause lost more time and/or energy while achieving the target/s. Guidance systems which have image processing property with camera/s need fast processors and may not react quickly to obstacle/s which come across.

In line/track following guidance systems various marks which can be detected by vehicle are placed in environment. Vehicle is guided autonomously by detecting these marks to achieve target which given by user and achieves its target/s. These guidance systems' conformance to the changes in environment can be hard and vehicle can't achieve to target/s given by user in case of detecting new obstacle/s on the way. Also, number of target places/points which can be given by user is fairly restricted in these type guidance systems.

Guidance of brainwave controlled wheelchairs basically made by processing of signals which received from users brain (focusing on a predefined thought by user) and according to this signal/thought, vehicle's right/left turnings and forward/back goings by assistance of electronic cards.

In existing brainwave controlled wheelchairs autonomous/smart guidance system are not available and user have to give commands consistently with brainwave etc. sensors to guide/control the vehicle until he/she achieves the place/s where he/she wants to go.

Object of the Invention

Today, there is no vehicle which can find its direction full/semi autonomously and can be navigated by assistance of a software running hardware with using the data which received from brainwave sensor and/or analyzing this data. According to changing environment conditions (static/dynamic obstacles), originally developed smart navigation system provide vehicle/user to achieve the place where desired by user by following the optimal path by consistently updating optimal path which provide to achieving the target/s which given by user until the vehicle achieves its target without doing any intervention (marking, installing a system, placement a sensor or camera etc.) to environments that vehicle will be used.

Also in this way disabled people who can't use their hands and feet, paralytic, MS, ALS patients' transportation is aimed by giving target/s by controlling mouse movements and clicking on the map of the vehicle with only brainwave etc. sensors. User can achieve the target point with giving only one mouse click (without touching anywhere and without any commands/guidance while the vehicle is moving).

Description of the figures

Smart navigation system and running algorithm which carry out for achieving the aim of the invention is shown with figures below:

Figure 1 is scheme of the smart navigation system;

Figure 2 is view of the map on users screen;

Figure 3 is view of the system's running algorithm.

Description of the references in the figures A: Smart navigation system

1 : Input device

2: Screen

3: Processing unit

4: Obstacle (24) detection sensor

5: Electronic control card

6: Electronic motor driver

7: Electric motors

8: Encoders

9: Inertial measurement unit

10: GPS

B: Map

11 : Starting point

12: Vehicle

13: Sub target point

14: Shortest path

15: Main target point

16: Learned area

17: Area border

18: Unlearned area

19: Dead end

20: Closed path

21 : Cell

22: Value

23: Map (B) border

24: Obstacle 25: Shortest path (14) estimation

26: Determined last shortest path (14) estimation

C: Smart navigation algorithm

27: Start

28: Updating the map (B, 28, 33)

29: Assigning main target point (15) by user

30: Sub target point (13) computation

31 : Achieving next sub target point (13)

32: Is sub target point (13) achieved?

33: Updating the map (B, 28, 33)

34: Are there any changes on the map (B)

35: Deletion of sub target point (13)

36: Is main target point (15) achieved? Disclosure of the Invention

Invention topic Smart Navigation System for Brainwave Controlled Wheelchairs in the most basic form includes; an input device (1 ) which helps to give main target point (15) where user wants to go, a screen (2) which includes a map (B) that shows available places and the vehicle's (12) movements, processing unit (3) which smart navigation algorithm (C) runs, obstacle (24) detection sensor (4) which used to detect walls and obstacles (24) around the vehicle (12), electronic control card (5), electronic motor driver (6), electric motors (7) used for movements of the vehicle (12), encoder/s (8), inertial measurement unit (9) and GPS (10) which used to measure vehicle's (12) position and orientation.

Input device (1 ) in smart navigation system (A) provides assigning main target point (15) by user (29) on the map (B) with a device like mouse, keyboard, brainwave (EEG) sensor that includes a gyroscope, accelerometer, key etc. which used to control mouse cursor movements and clicking operations on the screen (2) that connected to processing unit (3). Mouse cursor control with brainwave sensor can be done with thoughts, mimes or head movements. Screen (2) provides user to see interface program, map (B), vehicle's (12) position and mouse cursor. Processing unit (3) provides, receiving data which comes from input device (1 ) and obstacle (24) detection sensor (4), management of electronic control card (5), connection of screen (2) and running of smart navigation algorithm (C). Obstacle (24) detection sensor (4) provides distance measurement of obstacles (24) around the vehicle (12) with methods like radar, laser, ultrasonic, infrared etc. Electronic control card (5) can provide control of electronic motor driver (6), receiving inertial measurement unit (9) and GPS (10) data, wired or wireless communication with processing unit (3) and it includes a microprocessor which loaded with control software. Electronic motor driver (6) provides turning direction and speed control of electric motors (7) according to received data from electronic control card (5). Electric motors (7) provide vehicle's (12) movements according to signals received from electronic motor drivers (6) and vehicle (12) includes at least two electric motors (7). Encoders (8) provide calculation of vehicle's (12) going forward by measuring number of rotations of electric motors (7). Inertial measurement unit (9) provides orientation measurement and helps position control of the vehicle (12) with accelerometer, gyroscope and magnetometer in it. GPS (10) provides position data of the vehicle (12) in outdoor implementations.

Map (B) which displayed on screen (2) in the smart navigation system (A) is divided matrix shaped cells (21 ). And it shows areas which vehicle (12) can move and main target point (15) can be given on invisible cells (21 ) and areas that includes cells (21 ) which detected as obstacle (24) and has visible borders.

Starting point (11 ) on the map (B) shows the pixel which belongs to shortest path's (14)

155 (which calculated for achieving the vehicle (12) to the main target point (15)) starting point.

Vehicle (12) shows movement and position of the vehicle (12) which has smart navigation system (A). Sub target points (13) shows the points which the vehicle (12) has to follow in order to achieve the main target point (15). Shortest path (14) shows the estimated shortest path (14) which has to be followed to achieve the main target point (15) by the

160 vehicle (12) and it (14) is consistently updated until the vehicle (12) achieves the main target point (15). Main target point (15) shows the point (pixel) on the map (B) which expected to achieved by the vehicle (12). Learned area (16) shows the area which has been detected before by the vehicle (12) if there is obstacles (24) or not with obstacle (24) detection sensor (4). Area border (17) shows the border between learned area (16) and

165 unlearned area (18). Unlearned area (18) shows the area which hasn't been detected before by the vehicle (12) if there are obstacles (24) or not with obstacle (24) detection sensor (4) with different colour from learned area (16). Dead end (19) shows closed end ways which the vehicle (12) undesired to deviate while achieving the main target point (15). Closed path (20) shows the places which has been learned before by the vehicle (12)

170 if there is obstacles (24) or not with obstacle (24) detection sensor (4) but detected obstacles (24) on that area on last map (B) updating (28, 33) operation. Cell (21 ) means areas like square shape, changeable size parcelled matrix shaped areas on the map (B). Each cell (21 ) has a row-column number, value (22) and colour number. Value (22) means the numbers which each cell (21 ) has and updated when calculating shortest path (14).

175 Map (B) border (23) shows border of the area which main target point (15) can be given on. Obstacle (24) shows the areas which the vehicle (12) couldn't pass on and ignored on calculation of shortest path (14). Shortest path (14) estimations (25) show estimations made after first shortest path (14) estimation. Determined last shortest path (14) estimation (26) shows last estimation made when there is an opened or closed path (20) on shortest

180 path (14) between the vehicle (12) and the main target point (15) on the map (B).

Vehicle's (12) position, orientation, static and dynamic obstacles (24) in environment, vehicle's (12) field of view, learned areas (16) and unlearned areas (18) in different colours are shown and updated consistently on the map (B) in interface program which run by 185 processing unit (3).

Smart navigation system (A) which starts (27) to work with voltage, provides to receive data with processing unit (3) from obstacle (24) detection sensor (4) on the vehicle (12), shows calculated obstacles (24) on the screen (2) and updates the map (B, 28, 33). Main

190 target point (15) (which desired to be achieved by the vehicle) is assigned by user (29) from the screen (2) with an input device (1 ). Sub target points (13) on the shortest path (14) which provides to achieve targeted main target point (15) is computed (30). Next sub target point (13) is achieved (31 ) by vehicle (12) turning towards and going forward to next sub target point (13) on the shortest path (14). If the sub target point (13) haven't been

195 achieved (32) yet by the vehicle (12), map (B) is updated (28, 33) and if there is opened or closed paths (20) (If there is any change occurs on map (B, 34)) which effects to estimated shortest path (14) on the map (B) then sub target points (13) are computed (30). If no change occurs in map (B, 34) next sub target point (13) achieving (32) operation is continues. If sub target point (13) has been achieved (32) by the vehicle (12) then

200 achieved sub target point (13) will be deleted (35). If the main target point (15) which is targeted is achieved (36) by the vehicle (12), then new main target point (15) is expected to be given by the user (15).

Map (B) can be downloaded to the vehicle (12) in first use. If not downloaded, vehicle (12) 205 assumes there is no obstacles (24) in unlearned areas (18) after first main target point (15) is given and turns towards main target point (15) by calculating shortest path (14) to achieve main target point (15). Map (B) and shortest path (14) which will be followed is updated consistently while following the shortest path (14). Vehicle (12) can be walked around to learn and mapping the environment by remote control.

210

Clicked pixel on the map (B) belongs to which cell (21 ) is determined when main target point (15) is given by input device (1 ). Value (22) of this cell (21 ) will be "0" (zero). Then value (22) of eight cells (21 ) around this cell (21 ) is checked. If a value (22) haven't been given before to these cells (21 ) (or if not defined as obstacle (24)) value (22) " (one) will 215 be given. Value (22) "2" (two) is given to the cells (21 ) around the cell (21 ) whose value (22) is " (one) by repeating same operation. This operation is repeated until a value (22) has been given to the cell (21 ) which belongs to starting point (11 ) where the vehicle (12) exists. The cell (21 ) value (22) which starting point (11 ) belongs shows that the vehicle (12) how many cells (21 ) far away from main target point (15).

220

Shortest path (14), starting from starting point (11 ) towards main target point (15) is drawn like this. Cell (21 ) values (22) around the first cell (21 ) (which belongs to starting point (11 )) is checked respectively up, right, down, left, up-right, down-right, down-left and up-left sides. If a cell (21 ) with one more smaller value (22) than first cell (21 ) is detected, (without

225 checking other cells (21 ) around) a line is drawn first cell (21 ) to center of this cell (21 ).

After that, one more line is drawn from this cell (21 ) center to the last found cell (21 ) center by checking with same operation. This shortest path (14) drawing operation continues until it achieves to main target point (15) which has zero value (22). After these operations have been completed, shortest path (14) which provides to achieve main target point (15) from

230 starting point (11 ) was been determined and shown on the map (B).

Places (turning points) where the direction changes while drawing shortest path (14) are been determined and sub target points (1 3) are been placed on the map (B). Vehicle (12) autonomously and respectively achieves these sub target points (1 3) to achieve main 235 target point (15).

Angle and distance between sub target point (14) and vehicle's (12) existing position are been calculated. Vehicle (12) turns to this angle from its nearest side and turns until the angle difference is approximately ±2°. Vehicle (12) starts to go forward after it head 240 towards to sub target point (13). If the angle difference will be above ±2° again, it head towards again to sub target point (13) with turning operation. Vehicle (12) continues to going forward until the distance to sub target point (13) is over 0.5 meter. Map (B) updating (28, 33) with obstacle (24) detection sensor (4) continues during turning and go forwarding operations.

245

Shortest path (14) which will be followed is updated and new shortest path (14) estimations (25) are made if there is a closed path (20) on estimated shortest path (14) or if more shortest paths (14) have been discovered as result of updating map (B, 28, 33) operation. And, sub target points (13) are being updated until main target point (15) (which 250 given on starting) is achieved. The Application of the Invention to the Industry

Smart Navigation System for Brainwave Controlled Wheelchairs (A) can be used on 255 disabled vehicles (12) which provides disabled people who can't use their hands and feet, paralytic, MS, ALS patients etc. practical transportation on their own by only one clicking on start with controlling mouse moves and clicking operations by brainwave sensor.

Besides indoor usage, it can be used on guidance of unmanned patrol vehicles (12) ) in 260 places like factories, housing estates etc. with help of GPS (10), camera, temperature, gas, humidity, colour and sound sensors etc. (with smart path finding feature according to obstacles (24) and map (B) which changes every day)).

It can be used on autonomous guidance of unmanned material transportation vehicles (12) 265 in places like warehouse, harbour factory etc. between stated starting point (11 ) and main target point (15)

Further, it can be used on autonomous guidance of unmanned tractor, mine searching vehicle (12), military security and sabotage vehicles (12) on described areas on the map 270 (B).

Claims

1 . Smart navigation system (A) which consist of software and hardware for disabled vehicles (12) which can be used by disabled people who can't use their hands and feet, characterized by comprising: Input device (1 ) which consist of electroencephalography sensor, gyroscope, accelerometer, button or combination of these, screen (2) with a map (B) which shows available places that user can go, processing unit (3) where the smart navigation system algorithm (C) runs obstacle (24) detection sensor (4) which provides detection of obstacles (24) around the vehicle (12) with methods that use radar, laser, ultrasonic, infrared sensors or combination of these, electronic control card (12) with software which controls vehicle (12) movements and provides communication with processing unit (3), electronic motor driver (6), electric motors (7) which provide vehicle (12) movements and encoders (8), inertial measurement unit (9) and GPS (10) which provide to measure movements, position and orientation of the vehicle (12).

2. Input device (2) according to claim 1 characterized in that; electroencephalography sensor, gyroscope, accelerometer, button or combination of these which provides to assign main target point (15) on the map (B) of the screen (2) which connected to the processing unit (3) with thoughts, mimes, mouth, head movements or combination of these.

3. Processing unit (3) according to claim 1 characterized in that; processor which provides to run smart navigation system (A) algorithm (C) and interface program which provides to watching recorded videos, recording map (B) view, downloading and recording map (B) of the environment where the vehicle used or will be used, receiving data which comes from electronic control card (5), guidance of electronic control card (5), reading obstacle (24) detection sensor (4) data, showing map (B) which placed on the screen (2).

4. Obstacle (24) detection sensor (4) in the system according to claim 1 characterized in that; hardware (and software) which provides to measure distances of obstacles (24) around the vehicle (12) with methods that use radar, laser, ultrasonic, infrared or combination of these.

5. Electronic control card (5) in the system according to claim 1 characterized in that; hardware which provides control of electronic motor driver (6), reading data from inertial measurement unit (9) and GPS (10), communication with processing unit (3) and including microprocessor which has software that controls these.

6. Map (B) which placed on the screen (2) of the system according to claim 1 and which displayed on the interface program that run by processing unit (3) of the system according to claim 3 characterized in that; starting point (11 ) that shows where the vehicle (12) has started to move, the vehicle (12) with position and direction, sub target points (13) which help to achieve main target point (15), shortest path (14), main target point (15) which assigned by user, learned areas (16) where obstacle (24) detection is completed, unlearned areas (18) where the vehicle (12) haven't been before, area borders (17) between these areas, dead ends (19), closed paths (20), map (B) borders (23), obstacles (24), shortest path (14) estimations (24), last detected shortest path (14) estimation (26) and cells (21 ) which parcelled in matrix shape with values (22) of these cells (21 ). Smart navigation system (A) with hardware and software which provides autonomous transportation to assigned targets on disabled vehicles (12) for disabled people who can't use their hands and feet wherein running method comprising the following steps:

- Updating the map (B, 28, 33),

- Assigning main target point (15) by user (29),

- Computation of sub target points (13, 30),

- Achieving next sub target point (13, 31 ),

- Controlling vehicle (12) to achieving the sub target point (13, 32),

- Updating the map (B, 28,33),

- Controlling if there is any changing on the map (B, 34),

- Deleting the sub target point (13) which achieved by vehicle (12, 35),

- Controlling vehicle (12) to achieving the main target point (15, 36).