JP7323045B2 - Control device, control method and program - Google Patents

Description

The present invention relates to a technical field of a control device, a control method, and a recording medium that perform processing related to tasks to be performed by a robot.

A control method has been proposed in which, when a task to be performed by a robot is given, the robot is controlled to perform the task. For example, in Patent Document 1, when a robot having a hand grips a plurality of items and stores them in a container, a combination of the order in which the hand grips the items is determined, and based on the index calculated for each combination, the items are stored. A robotic controller for determining the order of items to be delivered is disclosed.

JP 2018-51684 A

When a robot performs a task, depending on the given task, it may be necessary to transfer objects to and from another device.

SUMMARY OF THE INVENTION One of the objects of the present invention is to provide a control device, a control method, and a recording medium capable of suitably generating a motion sequence of a robot in view of the above-described problems.

One aspect of a control device is a control device comprising: robot motion information indicating motion characteristics of a robot that executes a task; and peripheral devices that indicate motion characteristics of a peripheral device that transfers an object related to the task to the robot. motion information; motion sequence generation means for generating a motion sequence indicating motions to be performed by the robot and the peripheral device, respectively; integrated control means for adjusting operation timing of at least one of the robot and the peripheral device when the state does not match the expected state of the robot and the peripheral device when the robot and the peripheral device operate according to the operation sequence. have.

In one aspect of the control method, a computer generates robot motion information indicating motion characteristics of a robot that executes a task, and peripheral device motion information that indicates motion characteristics of a peripheral device that transfers an object related to the task to the robot. , a motion sequence indicating motions to be performed by each of the robot and the peripheral device is generated, and the states of the robot and the peripheral device measured during execution of the motion sequence are determined according to the motion sequence. If it does not match the expected state of the robot and the peripheral device when the robot and the peripheral device operate in the same manner, the operation timing of at least one of the robot and the peripheral device is adjusted.

One aspect of the program is based on robot operation information indicating movement characteristics of a robot that executes a task and peripheral device operation information indicating movement characteristics of a peripheral device that transfers an object related to the task to the robot, a motion sequence generating means for generating a motion sequence indicating motions to be performed by each of the robot and the peripheral device; A program that causes a computer to function as an integrated control means that adjusts the operation timing of at least one of the robot and the peripheral device when it does not match the expected state of the robot and the peripheral device when the robot and the peripheral device operate as expected.

According to the present invention, it is possible to suitably generate each operation sequence of a robot and a peripheral device that transfers an object between the robot and the robot.

4 shows the configuration of the robot control system. The hardware configuration of the control device is shown. An example of the data structure of application information is shown. It is an example of a functional block of a control device in a 1st embodiment. Fig. 3 shows an overhead view of a workspace; A display example of a task input screen is shown. 6 is an example of a flow chart showing control processing regarding the robot and peripheral devices in the first embodiment. It is an example of the schematic block diagram of the control apparatus in 2nd Embodiment. FIG. 10 is an example of a flowchart showing control processing regarding the robot and peripheral devices in the second embodiment; FIG. It is an example of the schematic block diagram of the control apparatus in 3rd Embodiment. FIG. 11 is an example of a flow chart showing control processing relating to a robot and peripheral devices in a third embodiment; FIG. FIG. 11 is a schematic configuration diagram of a control device in a fourth embodiment; FIG. 11 is an example of a flowchart showing control processing regarding a robot and peripheral devices in a fourth embodiment; FIG.

Embodiments of a control device, a control method, and a recording medium will be described below with reference to the drawings.

<First embodiment>
(1) System configuration
FIG. 1 shows the configuration of a robot control system 100 according to the first embodiment. The robot control system 100 mainly includes a control device 1 , an input device 2 , a display device 3 , a storage device 4 , a robot 5 , a measurement device 7 and peripheral devices 8 .

When a task to be executed by the robot 5 and the peripheral device 8 (also referred to as a "target task") is specified, the control device 1 determines the operation to be performed by the robot 5 and the peripheral device 8 at each time step. A sequence is generated and the motion sequence is supplied to the robot 5 and the peripheral device 8 . As will be described later, the operation sequence is composed of tasks or commands (also called "subtasks") obtained by decomposing the target task into units that can be received by the robot 5 and the peripheral device 8, and hereinafter also called "subtask sequences". call.

The control device 1 is electrically connected to the input device 2 , the display device 3 and the storage device 4 . For example, the control device 1 receives from the input device 2 an input signal “S1” for designating a target task. Further, the control device 1 transmits a display signal “S2” for displaying a task to be executed by the robot 5 to the display device 3 . Further, the control device 1 transmits a control signal “S4” regarding control of the robot 5 to the robot 5 and transmits a control signal “S5” regarding control of the peripheral device 8 to the peripheral device 8 . Furthermore, the control device 1 receives an output signal “S3” from the measuring device 7 .

The input device 2 is an interface that receives user input, and corresponds to, for example, a touch panel, buttons, keyboard, voice input device, and the like. The input device 2 supplies an input signal S1 generated based on the user's input to the control device 1 .

The display device 3 is, for example, a display, a projector, etc., and performs predetermined display based on the display signal S2 supplied from the control device 1 . As will be described later, for example, the display device 3 displays an input screen (also called a "task input screen") for designating information about the target task based on the display signal S2.

The storage device 4 has an application information storage unit 41 . The application information storage unit 41 stores application information required to generate a sequence of subtasks from a target task. Details of the application information will be described later with reference to FIG. The storage device 4 may be an external storage device such as a hard disk connected to or built into the control device 1, or may be a recording medium such as a flash memory. Also, the storage device 4 may be a server device that performs data communication with the control device 1 . In this case, the storage device 4 may be composed of a plurality of server devices.

The robot 5 operates based on the control signal S4 transmitted from the control device 1 . The robot 5 shown in FIG. 1 has, as an example, a robot controller 51 and a robot arm 52 capable of gripping an object. The robot arm 52 is a pick that moves an object 61 conveyed by the peripheral device 8 to a destination area 63 without interfering with an obstacle 62 that is an object other than the object 61 under the control of the robot control unit 51 . And place (processing to pick up and move) is performed. The robot control unit 51 performs motion control of the robot arm 52 based on the subtask sequence indicated by the control signal S4. The robot 5 may be a vertical articulated robot or any type of robot such as a horizontal articulated robot.

The measuring device 7 is a camera, a range sensor, a sonar, or one or more external sensors that are a combination of these for measuring the work space 6 in which the robot 5 and the peripheral devices 8 work as a measurement target range. The measuring device 7 supplies the generated output signal S3 to the control device 1 . The output signal S3 may be image data obtained by photographing the inside of the work space 6, or may be point cloud data indicating the position of an object in the work space 6. FIG.

The peripheral device 8 operates based on the control signal S5 transmitted from the control device 1 . The peripheral device 8 shown in FIG. 1 has, as an example, a peripheral device control unit 81 and a transport device 82 capable of transporting objects. The transport device 82 transports the object 61 and the obstacle 62 under the control of the peripheral device control section 81 . The peripheral device control unit 81 controls the operation of the carrier device 82 based on the subtask sequence indicated by the control signal S5. The peripheral device 8 may be an AGV (Automated Guided Vehicle) or a slider, or may be a conveyor capable of transporting in a predetermined direction or multiple directions. Also, the peripheral device 8 may be a device that moves a container or a shelf containing the object 61 .

Note that the configuration of the robot control system 100 shown in FIG. 1 is an example, and various modifications may be made to the configuration. For example, a plurality of robots 5 may exist. Also, the robot 5 may include two or more robot arms 52 . Even in these cases, the control device 1 generates a subtask sequence to be executed for each robot arm 52 based on the target task, and sends the control signal S4 indicating the subtask sequence to the robot having the target robot arm 52. 5. Similarly, there may be a plurality of peripheral devices 8 or carrier devices 82 . Also, the measuring device 7 may be a part of the robot 5 . Also, the robot control unit 51 may be configured separately from the robot 5 . In this case, the robot control section 51 may be incorporated into the control device 1 . Further, the input device 2 and the display device 3 may each be configured as the same device as the control device 1 (for example, a tablet-type terminal) by being incorporated in the control device 1 or the like. Also, the control device 1 may be composed of a plurality of devices. In this case, the plurality of devices that make up the control device 1 exchange information necessary for executing previously assigned processing among the plurality of devices. Also, the robot 5 or the peripheral device 8 may have at least part of the functions of the control device.

(2) Hardware configuration of control device
FIG. 2 shows the hardware configuration of the control device 1. As shown in FIG. The control device 1 includes a processor 11, a memory 12, and an interface 13 as hardware. Processor 11 , memory 12 and interface 13 are connected via data bus 19 .

The processor 11 performs predetermined processing by executing programs stored in the memory 12 . The processor 11 is a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit).

The memory 12 is composed of various memories such as RAM (Random Access Memory) and ROM (Read Only Memory). The memory 12 also stores a program for the control device 1 to execute a predetermined process. Also, the memory 12 is used as a work memory and temporarily stores information and the like acquired from the storage device 4 . Note that the memory 12 may function as the storage device 4 . Similarly, storage device 4 may function as memory 12 of control device 1 . Note that the program executed by the control device 1 may be stored in a recording medium other than the memory 12 .

The interface 13 is an interface for electrically connecting the control device 1 and other devices. For example, the interface 13 includes an interface for connecting the control device 1 and the input device 2, an interface for connecting the control device 1 and the display device 3, and an interface for connecting the control device 1 and the storage device 4. Contains interfaces. Further, the interface 13 includes an interface for connecting the control device 1 and the robot 5, an interface for connecting the control device 1 and the peripheral device 8, and an interface for connecting the control device 1 and the measuring device 7. including. These connections may be wired connections or wireless connections. For example, the interface for connecting the control device 1 and the storage device 4 may be a communication interface for transmitting and receiving data to and from the storage device 4 under the control of the processor 11 by wire or wirelessly. In another example, the control device 1 and the storage device 4 may be connected by a cable or the like. In this case, the interface 13 includes an interface conforming to USB (Universal Serial Bus), SATA (Serial AT Attachment), etc. for exchanging data with the storage device 4 .

Note that the hardware configuration of the control device 1 is not limited to the configuration shown in FIG. 2 . For example, the control device 1 may include at least one of the input device 2 , the display device 3 and the storage device 4 . Also, the control device 1 may be connected to or built in a sound output device such as a speaker. In these cases, the control device 1 may be a tablet terminal or the like in which the input function and the output function are integrated with the main body.

(3) Application information
Next, the data structure of application information stored in the application information storage unit 41 will be described.

FIG. 3 shows an example of the data structure of application information stored in the application information storage unit 41. As shown in FIG. As shown in FIG. 3, the application information storage unit 41 stores abstract state designation information I1, constraint information I2, motion limit information I3, subtask information I4, abstract model information I5, and object model information I6. include.

The abstract state designation information I1 is information that designates an abstract state that needs to be defined when generating a subtask sequence. This abstract state is an abstract state of an object in the work space 6, and is defined as a proposition used in a target logical formula to be described later. For example, the abstract state designation information I1 designates an abstract state that needs to be defined for each type of target task. Note that the target task may be, for example, various types of tasks such as pick-and-place, product assembly, and placement of ingredients in a container such as a lunch box.

Constraint information I2 is information indicating the constraint regarding the robot 5 and the peripheral device 8 when executing the target task. Constraint condition information I2 includes, for example, a constraint condition that the robot 5 (robot arm 52) must not come into contact with an obstacle when the target task is a pick-and-place task, and a constraint condition that the robot 5 (robot arm 52) must not come into contact with the peripheral device 8. It shows the constraint conditions such as Note that the constraint condition information I2 may be information in which a constraint condition suitable for each type of target task is recorded.

The motion limit information I3 indicates information about motion limits of the robot 5 and the peripheral device 8 controlled by the control device 1 . In the case of the robot 5, the motion limit information I3 includes the maximum value of velocity and acceleration of each motion such as reaching of the robot arm 52, and motion constraint information such as the movable angle. In addition, in the case of the peripheral device 8, the motion limit information I3 includes information on motion constraints such as the maximum values of the transport speed and acceleration of the transport device 82 and possible transport directions.

The subtask information I4 includes information on subtasks that the robot 5 can receive and information on subtasks that the peripheral device 8 can receive. For example, if the target task is pick and place, the subtask information I4 defines reaching, which is movement of the robot arm 52, and grasping, which is grasping by the robot arm 52, as subtasks of the robot 5. FIG. Further, in this case, the subtask information I4 defines the operation related to transportation of the object by the transportation device 82 as a subtask of the peripheral device 8 . If the user can select the type of target task, the subtask information I4 may indicate information on subtasks that can be used for each type of target task.

The abstract model information I5 is information about a model (also called an “abstract model”) that abstracts the dynamics of objects including the robot 5 and the peripheral device 8 in the workspace 6 . The abstract model information I5 has information on an abstract model suitable for each type of target task when the type of target task can be selected by the user. The abstract model is represented by a hybrid system abstracted from real dynamics. The abstract model information I5 includes information indicating a dynamics switching condition (also referred to as a "dynamics switching condition") in the hybrid system described above. The dynamics switching condition is a dynamics switching condition related to at least one of the robot 5 and the peripheral device 8 . In the example of FIG. 1, the following dynamics switching conditions (a) and (b) are given.
(a) "When the object 61 or the obstacle 62 is placed on the transport device 82, it moves in a certain direction based on the operating speed of the transport device 82."
(b) "When the robot arm 52 grabs the target object 61, the target object 61 moves based on the motion of the robot arm 52 regardless of whether it is placed on the transfer device 82 or not."

The object model information I6 is each object to be recognized from the output signal S3 generated by the measuring device 7 (in the example of FIG. 1, the robot arm 52, the target object 61, the obstacle 62, the destination area 63, the transport equipment 82, etc.). is information about the object model of The object model information I6 includes, for example, information necessary for the control device 1 to recognize the type, position, and/or orientation of each object described above, and CAD data for recognizing the three-dimensional shape of each object. 3D shape information. The former information includes the parameters of the reasoner obtained by learning a learning model in machine learning such as a neural network. For example, when an image is input, this inference unit is trained in advance so as to output the type, position, orientation, etc. of an object that is a subject in the image.

In this way, the application information includes robot motion information indicating motion characteristics such as the motion limits, subtasks, and abstract models of the robot 5, and peripheral device motion information indicating motion characteristics such as the motion limits, subtasks, and abstract models of the peripheral device 8. contains information and, respectively. In addition to the information described above, the application information storage unit 41 may store various information related to the process of generating the subtask sequence.

(4) Function block
FIG. 4 is an example of functional blocks of the control device 1 . The processor 11 of the control device 1 functionally includes a state measuring unit 30, an abstract state setting unit 31, a target logical expression generating unit 32, a time step logical expression generating unit 33, an abstract model generating unit 34, It has a control input generator 35 and a subtask sequence generator 36 . Note that FIG. 4 shows an example of data exchanged between blocks, but the invention is not limited to this.

The state measurement unit 30 generates information (also referred to as “state information Im”) indicating the state of the object in the work space 6 based on the output signal S3 supplied from the measuring device 7 . Specifically, when the state measurement unit 30 receives the output signal S3, the state measurement unit 30 refers to the object model information I6 and the like, and uses technology (image processing technology, image recognition technology, voice recognition technology, etc.) to recognize the environment in the work space 6. technology, technology using RFID (Radio Frequency Identifier), etc.), the output signal S3 is analyzed. Thereby, the state measurement unit 30 measures the number, position, orientation, and the like of each type of each object in the workspace 6 related to the execution of the target task, and generates the measurement result as the state information Im. For example, in the case of FIG. 1, the state information Im includes information such as the numbers, positions, and attitudes of the robot arms 52, the objects 61, the obstacles 62, and the transfer equipment 82, respectively. The state measurement unit 30 supplies the generated state information Im to the abstract state setting unit 31 and the abstract model generation unit 34 . In addition to the measurement result of the three-dimensional object, the state information Im includes the measurement result of the position and range of a planar object (area) related to the target task such as the destination area 63 in FIG. may be included.

The abstract state setting unit 31 sets the abstract state in the workspace 6 that needs to be considered when executing the target task based on the above-described state information Im and the abstract state specifying information I1 acquired from the application information storage unit 41. set. In this case, the abstract state setting unit 31 defines a proposition for expressing the abstract state with a logical expression for each abstract state. The abstract state setting unit 31 supplies information indicating the set abstract state (also referred to as “abstract state setting information IS”) to the target logical expression generating unit 32 .

When receiving an input signal S1 related to a target task from the input device 2, the target logical expression generation unit 32 converts the target task indicated by the input signal S1 into a time phase representing the final achievement state based on the abstract state setting information IS. Convert to a logical formula (also called "target logical formula Ltag"). In this case, the target logical expression generation unit 32 refers to the constraint information I2 from the application information storage unit 41 to add the constraint to be satisfied in executing the target task to the target logical expression Ltag. Then, the target logical expression generation unit 32 supplies the generated target logical expression Ltag to the time step logical expression generation unit 33 . In addition, the target logical expression generation unit 32 generates a display signal S2 for displaying a task input screen for receiving input regarding the target task, and supplies the display signal S2 to the display device 3 .

The time step logical expression generation unit 33 converts the target logical expression Ltag supplied from the target logical expression generation unit 32 into a logical expression representing the state at each time step (also referred to as "time step logical expression Lts"). do. Then, the time step logical expression generator 33 supplies the generated time step logical expression Lts to the control input generator 35 .

The abstract model generation unit 34 generates an abstract model “Σ” that abstracts the actual dynamics in the work space 6 based on the state information Im and the abstract model information I5 stored in the application information storage unit 41 . In this case, the abstract model generator 34 regards the target dynamics as a hybrid system in which continuous dynamics and discrete dynamics coexist, and generates an abstract model Σ based on the hybrid system. A method of generating the abstract model Σ will be described later. The abstract model generator 34 supplies the generated abstract model Σ to the control input generator 35 .

The control input generation unit 35 satisfies the time step logical expression Lts supplied from the time step logical expression generation unit 33 and the abstract model Σ supplied from the abstract model generation unit 34, and optimizes the evaluation function at each time step determine the control inputs to the robot 5 and the peripheral device 8 of . The control input generator 35 then supplies the subtask sequence generator 36 with information (also referred to as “control input information Ic”) indicating control inputs to the robot 5 and the peripheral device 8 at each time step.

The subtask sequence generation unit 36 generates subtask sequences for the robot 5 and the peripheral device 8 to operate based on the control input information Ic supplied from the control input generation unit 35 and the subtask information I4 stored in the application information storage unit 41. Generate. Then, the subtask sequence generator 36 supplies a control signal S4 indicating a subtask sequence to be executed by the robot 5 to the robot 5 via the interface 13, and a control signal S5 indicating a subtask sequence to be executed by the peripheral device 8 via the interface. 13 to the peripheral device 8. The control signals S4 and S5 each contain information indicating the execution order and execution timing of the subtasks forming the subtask sequence.

4, the state measuring unit 30, the abstract state setting unit 31, the target logical expression generating unit 32, the time step logical expression generating unit 33, the abstract model generating unit 34, the control input generating unit 35, and the subtask sequence generating unit 36 can be realized by the processor 11 executing a program, for example. More specifically, each component can be implemented by processor 11 executing a program stored in memory 12 or storage device 4 . Further, each component may be realized by recording necessary programs in an arbitrary non-volatile recording medium and installing them as necessary. It should be noted that each of these components is not limited to being realized by software based on a program, and may be realized by any combination of hardware, firmware, and software. Also, each of these components may be realized using a user-programmable integrated circuit such as an FPGA (field-programmable gate array) or a microcomputer. In this case, this integrated circuit may be used to implement a program composed of the above components. Thus, each component may be realized by hardware other than the processor. The above also applies to other embodiments described later.

(5) Processing details for each block
Next, details of processing for each functional block shown in FIG. 4 will be described using a specific example.

(5-1) State measuring unit and abstract state setting unit
The state measurement unit 30 generates state information Im indicating the state (type, position, etc.) of the object in the work space 6 based on the output signal S3 supplied from the measuring device 7 . The abstract state setting unit 31 sets the abstract state in the work space 6 based on the state information Im. In this case, the abstract state setting unit 31 refers to the abstract state designation information I1 and recognizes the abstract state to be set within the work space 6. FIG. Note that the abstract state to be set in the work space 6 differs depending on the type of target task. Therefore, when the abstract state specification information I1 specifies the abstract state to be set for each type of target task, the abstract state setting unit 31 specifies the abstract state corresponding to the target task specified by the input signal S1. The information I1 is referenced to recognize the abstract state to be set.

FIG. 5 shows a bird's-eye view of the work space 6. As shown in FIG. The work space 6 shown in FIG. and exist.

In this case, the state measuring unit 30 analyzes the output signal S3 received from the measuring device 7 using the object model information I6 and the like to determine the positions of the robot arm 52, the transfer device 82, the object 61, and the obstacle 62. and posture, and the position and range of the destination area 63 serving as the destination of the object 61 are recognized. In this case, the state measurement unit 30 may set the tip (robot hand) of the robot arm 52 as the position of the robot arm 52 for the sake of abstraction. The abstract state setting unit 31 then refers to the abstract state designation information I1 to determine the abstract state to be defined in the target task. In this case, the abstract state setting unit 31 determines a proposition indicating the abstract state based on the state information Im indicating the recognition result of the state measuring unit 30 and the abstract state specifying information I1. In the example of FIG. 5, the abstract state setting unit 31 attaches identification labels to the object 61, the obstacle 62, the robot arm 52, the conveying device 82, and the destination area 63 specified by the state information Im. Further, the abstract state setting unit 31, based on the abstract state designation information I1, sets the proposition that the object 61 or the obstacle 62 is placed on the conveying device 82, the destination point where the object 61 should be finally placed, and Each proposition used in the target logical formula is defined, such as the proposition that the robot arm 52 exists within a certain area G (see the dashed frame 63) and the proposition that the robot arm 52 is interfering with the obstacle 62. FIG.

In this way, the abstract state setting unit 31 refers to the abstract state designation information I1 to recognize the abstract state to be defined, and converts the proposition representing the abstract state into the number of objects 61, the number of robot arms 52, , according to the number of obstacles 62 and the like. Then, the abstract state setting unit 31 supplies the information indicating the proposition representing the abstract state to the target logical expression generating unit 32 as the abstract state setting information IS.

(5-2) Target logical expression generator
The target logical expression generator 32 receives input of an input signal S1 that specifies the type of target task and the final state of the object to be worked on by the robot. In addition, the target logical expression generation unit 32 transmits to the display device 3 a display signal S2 of a task input screen that accepts these inputs.

The target logical formula generator 32 converts the target task specified by the input signal S1 into a logical formula using temporal logic. The input signal S1 may be expressed using natural language. There are various techniques for converting a task expressed in natural language into a logical expression. For example, in the example of FIG. 5, the target logical expression generator 32 is given a target task indicating the proposition "g" that "the object 61 finally exists in the region G." In this case, the target logical expression generation unit 32 sets the target task as an operator “◇” corresponding to “eventually” in a linear logical expression (LTL) and a proposition “g ” to generate the logical expression “◇g”. Note that the target logical expression generation unit 32 can generate arbitrary temporal logic operators other than the operator “◇” (logical product “∧”, logical sum “∨”, negation “￢”, logical inclusion “⇒”, always "□", next "○", until "U", etc.) may be used to express a logical expression. In addition, the logical expression may be expressed using any temporal logic such as MTL (Metric Temporal Logic), STL (Signal Temporal Logic), or the like, without being limited to linear temporal logic.

Also, the target logical expression generation unit 32 acquires the constraint condition information I2 from the application information storage unit 41 . If the application information storage unit 41 stores the constraint condition information I2 for each type of task, the target logical expression generation unit 32 calculates the constraint condition corresponding to the type of target task specified by the input signal S1. Information I2 is acquired from the application information storage unit 41 .

Then, the target logical formula generating unit 32 generates the target logical formula Ltag by adding the constraint indicated by the obtained constraint condition information I2 to the logical formula indicating the target task. For example, constraints corresponding to pick-and-place are included in the constraint information I2, such as "the robot arm 52 never interferes with the obstacle 62" and "the robot arm 52 never interferes with the transfer device 82". If so, the target logical formula generator 32 converts these constraints into logical formulas. Then, the target logical expression Ltag is generated by adding the converted logical expression of these constraints to the logical expression corresponding to the target task.

Next, an input example regarding the target task on the task input screen will be described.

FIG. 6 shows a display example of the task input screen. The target logical expression generation unit 32 generates a display signal S2 and transmits the display signal S2 to the display device 3, thereby causing the display device 3 to display the task input screen shown in FIG. A task input screen shown in FIG. Accepts specification of an object to be regarded as

The target logical expression generation unit 32 receives an input designating the type of target task in the task type designation column 15 . Here, as an example, the task type specification field 15 is an input field in the form of a pull-down menu. The target logical expression generation unit 32 displays, for example, a list of acceptable target task type candidates in a selectable manner in the task type designation column 15 . Here, the task type designation field 15 designates pick and place as the type of the target task.

In addition, the target logical expression generation unit 32 displays an image of the interior of the work space 6 captured by the measuring device 7 in the image display field 16 . Then, the target logical expression generation unit 32 receives an input designating an object regarded as the target object 61 based on a touch panel operation or a mouse operation on the image display field 16 .

In this case, in the first example, when the task input screen is displayed, the target logical expression generation unit 32 generates an object region ( Areas 18b, 18c) are recognized here. Then, when one of the regions is selected by a touch operation or a click operation, the target logical expression generation unit 32 regards the object in the selected region (here, the region 18b) as the target object 61 . In the second example, when some pixels forming the target object 61 are specified by a touch operation or a click operation, the target logical expression generation unit 32 converts the area of the object including the specified pixels to the target object 61 considered to be the area of In the third example, the target logical expression generation unit 32 regards the object in the circled image area as the target object 61 when detecting the operation of encircling the image area of the object to be the target object 61 .

Further, the target logical expression generation unit 32 automatically recognizes that the area 18a is an area indicating the destination area 63 by image recognition processing. In this case, the destination area 63 may be provided with a marker or the like for facilitating the recognition of the destination area 63 by the target logical expression generation unit 32. may be stored in the storage device 4 as application information. Instead of the target logical expression generation unit 32 automatically recognizing the destination area 63, the destination area 63 may be recognized by receiving an input designating the destination area 63 in the same manner as the target object 61. . Further, in this case, the target logical expression generation unit 32 may recognize the destination area 63 by accepting input of the final destination of the object 61 by a drag-and-drop operation.

Further, when the same kind of object specification button 17 is selected, the target logical expression generation unit 32 regards an object of the same kind as the object in addition to the object specified on the image display field 16 as the target object 61. . In this case, the object regarded as the target object 61 may be an object existing outside the measurement range of the measuring device 7 at the time the task input screen is displayed. Thereby, the target logical expression generation unit 32 can efficiently allow the user to designate the object 61 . Note that when the type of object to be the target object 61 is predetermined, the control device 1 may recognize the target object 61 without receiving an input designating the target object 61 . In this case, information for identifying the object 61 is included in the object model information I6 and the like, and the state measurement unit 30 refers to the object model information I6 and the like to identify and state (position) the object 61. , posture, etc.).

Then, when detecting that the decision button 20 has been selected, the target logical expression generation unit 32 recognizes the target task, the object 61 and/or the destination based on the input signal S1 indicating the input content on the task input screen. The information in area 63 is used to generate the target logical expression Ltag. The target logical expression generation unit 32 may supply the abstract state setting unit 31 with information on the target task, the target object 61 and/or the destination area 63 recognized based on the input signal S1. In this case, the abstract state setting unit 31 sets propositions regarding the target task, the object 61 and/or the destination area 63 based on the supplied information.

(5-3) Time step logical expression generator
The time step logical expression generator 33 determines the number of time steps (also referred to as “target number of time steps”) for completing the target task, and determines the state at each time step such that the target number of time steps satisfies the target logical expression Ltag. Define a combination of propositions representing Since there are usually a plurality of such combinations, the time step logical expression generation unit 33 generates a logical expression combining these combinations by logical sum as the time step logical expression Lts. The above combinations are candidates for logical expressions representing sequences of actions to be instructed to the robot 5, and are hereinafter also referred to as "candidates φ". Therefore, the time step logical expression generation unit 33 determines the logical sum of the generated candidates φ as the time step logical expression Lts. In this case, the time step logical expression Lts is true when at least one of the generated candidates φ is true.

Preferably, the time step logical expression generation unit 33 refers to the operation limit information I3 for the generated candidates to determine the feasibility thereof, and excludes the candidates φ that are determined to be unrealizable. For example, based on the motion limit information I3, the time step logical expression generation unit 33 determines the distance that the robot arm 52's hand (robot hand) can move per time step and the transfer device 82 moves the object 61 per time step. Recognize the distance that can be moved. Further, the time step logical expression generator 33 recognizes the distance between the object 61 to be moved and the robot hand based on the position vectors of the object 61 and the robot hand indicated by the state information Im. Then, the time step logical expression generator 33 determines feasibility based on these distances.

In this way, the time step logical expression generation unit 33 refers to the operation limit information I3 and excludes unrealizable candidates from the time step logical expression Lts, thereby suitably reducing the processing load of the subsequent processing unit. be able to.

Next, a supplementary explanation will be given on how to set the target number of time steps.

The timestep logical expression generation unit 33 determines the target number of timesteps based on, for example, the expected time of work designated by user input. In this case, the time step logical expression generation unit 33 calculates the target number of time steps from the above-described estimated time based on the information on the time width per time step stored in the memory 12 or storage device 4 . In another example, the time step logical expression generation unit 33 stores in the memory 12 or the storage device 4 in advance information that associates the target number of time steps suitable for each type of target task, and refers to the information. Thus, the target number of time steps is determined according to the type of target task to be executed.

Preferably, the timestep logical expression generator 33 sets the target number of timesteps to a predetermined initial value. Then, the time step logical expression generator 33 gradually increases the target number of time steps until the time step logical expression Lts that allows the control input generator 35 to determine the control input is generated. In this case, the time step logical expression generation unit 33 sets the target number of time steps to a predetermined number when the optimization process performed by the control input generation unit 35 fails to lead to the optimum solution. Add by a number (an integer of 1 or more).

At this time, the time step logical expression generation unit 33 may set the initial value of the target number of time steps to a value smaller than the number of time steps corresponding to the working time of the target task expected by the user. Thereby, the time step logical expression generation unit 33 preferably suppresses setting an unnecessarily large target number of time steps.

(5-4) Abstract model generator
The abstract model generator 34 generates an abstract model Σ based on the state information Im and the abstract model information I5. Here, information required for generating the abstract model Σ is recorded in the abstract model information I5. For example, if the target task is pick-and-place, the position and number of objects, the position of the destination area where the objects are placed, the number of robot arms 52, the position and transfer speed (and transfer direction) of transfer device 82, etc. is recorded in the abstract model information I5. This abstract model may be represented by a difference equation showing the relationship between the state of objects in workspace 6 at timestep 'k' and the state of objects in workspace 6 at timestep 'k+1'. . At this time, variables such as a position vector indicating the position of the object and a position vector indicating the position of the hand of the robot arm 52 are given to the difference equation.

Then, the abstract model generation unit 34 generates the position and number of objects indicated by the state information Im, the position of the area where the objects are placed, and the position of the robot 5 for the general-purpose abstract model recorded in the abstract model information I5. An abstract model Σ is generated by reflecting the number of machines.

Here, when the robot 5 is working on the target task, the dynamics in the work space 6 are frequently switched. Therefore, the abstract model stored in the abstract model information I5 is a model that abstractly expresses this switching of dynamics using logic variables. Therefore, the abstract model generation unit 34 abstractly expresses an event (behavior) in which dynamics are switched in an abstract model using logical variables, so that dynamics switching can be suitably expressed by the abstract model.

For example, in the workspace 6 shown in FIGS. 1 and 5, the following dynamics switching conditions exist.
(a) "When the object 61 or the obstacle 62 is placed on the transport device 82, it moves in a certain direction based on the operating speed of the transport device 82."
(b) "When the robot arm 52 grabs the target object 61, the target object 61 moves based on the motion of the robot arm 52 regardless of whether it is placed on the transfer device 82 or not."
Therefore, in this case, the abstract model generation unit 34 abstractly expresses the motion of placing the object 61 or the obstacle 62 on the carrier device 82 in the abstract model using logical variables, and the robot arm 52 is placed on the object 61 The action of grabbing is abstractly expressed in the abstract model using logical variables.

In this way, the abstract model generation unit 34 refers to the abstract model information I5, and the hybrid system in which the switching of dynamics is represented by a logical variable that is a discrete value, and the movement of an object is represented by a continuous value. An abstract model Σ, which is an abstracted model of dynamics, is set. Here, when the abstract model Σ is represented by a difference equation showing the relationship between the states of the object in the workspace 6 at time steps “k” and “k+1”, the difference equation represents the position It includes vectors and the like, variables (parameters) representing control inputs to the robot 5 and control inputs to the peripheral device 8, and logical variables representing dynamics switching.

Also, the abstract model Σ represents not the detailed dynamics of the entire robot 5 and the peripheral device 8 but the abstracted dynamics. For example, the abstract model Σ may represent only the dynamics of the robot hand, which is the hand of the robot 5 that actually grips the object, for the robot 5 . As another example, in the abstract model Σ, the dynamics of the peripheral device 8 may represent that the position of an object placed on the transport device 82 changes according to the control input of the peripheral device 8 . As a result, the calculation amount of optimization processing by the control input generation unit 35 can be suitably reduced.

Note that the abstract model generation unit 34 may generate a model of a hybrid system combining a mixed logical dynamic (MLD) system or a petri net, an automaton, or the like.

(5-5) Control input generator
Based on the time step logical expression Lts supplied from the time step logical expression generating unit 33 and the abstract model Σ supplied from the abstract model generating unit 34, the control input generating unit 35 generates the optimal robot 5 for each time step. and the control inputs to the peripheral device 8 for each time step. In this case, the control input generator 35 defines an evaluation function for the target task, and solves an optimization problem of minimizing the evaluation function with the abstract model Σ and the time step logical expression Lts as constraints.

The evaluation function is determined in advance for each type of target task, and stored in the memory 12 or the storage device 4, for example. The evaluation function may be designed to minimize the energy expended by the robot 5 or may be designed to minimize the energy expended by the robot 5 and the peripheral device 8 . In the example shown in FIGS. 1 and 5, for example, the control input generator 35 generates the distance “d _k ” between the target object 61 and the destination region 63 , the control input “u _kr ” to the robot 5 , the transport device 82 The evaluation function is determined so that the control input "u _kp " for is minimized. Specifically, the control input generation unit 35 generates, for example, the square of the norm of the distance _dk , the square of the norm of the control input u _kr , and the square of the norm of the control input u _kp in all time steps. Define the sum as the evaluation function. In this case, the control input generator 35 may multiply the term of the distance _dk , the term of the control input _ukr , and the term of the control input _ukp by a predetermined weighting factor. The control input u _kr and the control input u _kp may be velocity or acceleration.

Then, the control input generation unit 35 solves a constrained mixed integer optimization problem with the abstract model Σ and the time step logical expression Lts (that is, the logical sum of the candidates φ _i ) as constraints for the set evaluation function. Here, the control input generation unit 35 may reduce the amount of calculation by approximating the logical variables to continuous values to form a continuous relaxation problem. It should be noted that when STL is adopted instead of linear logic equations (LTL), it is possible to describe the problem as a nonlinear optimization problem. Thereby, the control input generator 35 calculates the control input _ukr for the robot 5 and the control input _ukp for the transfer device 82 respectively.

Further, when the target number of time steps is long (for example, when it is larger than a predetermined threshold value), the control input generation unit 35 sets the number of time steps used for optimization to a value smaller than the target number of time steps (for example, the threshold value described above). You may In this case, the control input generator 35 sequentially determines the control input u _kr and the control input u _kp by solving the above optimization problem every time a predetermined number of time steps elapses.

Preferably, the control input generation unit 35 solves the above-described optimization problem for each predetermined event corresponding to an intermediate state with respect to the target task achievement state, and determines the control input u _kr and the control input u _kp to be used. may decide. In this case, the control input generator 35 sets the number of time steps until the occurrence of the next event to the number of time steps used for optimization. The event described above is, for example, an event in which the dynamics in the workspace 6 are switched. For example, when the target task is pick-and-place, events such as the robot 5 picking up an object, finishing carrying one of a plurality of objects to be carried by the robot 5 to a destination point, etc. are defined as events. be done. The event is predetermined, for example, for each type of target task, and information specifying the event for each type of target task is stored in the storage device 4 .

Also according to this aspect, the number of time steps used for optimization can be reduced, and the amount of calculation of the optimization problem can be suitably reduced.

(5-6) Subtask sequence generator
The subtask sequence generation unit 36 generates a subtask sequence based on the control input information Ic supplied from the control input generation unit 35 and the subtask information I4 stored in the application information storage unit 41 . In this case, the subtask sequence generator 36 recognizes the subtasks that the robot 5 can receive and the subtasks that the peripheral device 8 can receive by referring to the subtask information I4. Then, the subtask sequence generator 36 converts the control input to the robot 5 at each time step indicated by the control input information Ic into a subtask for the robot 5, and the subtask to the peripheral device 8 at each time step indicated by the control input information Ic. Transform control inputs into peripheral 8 subtasks.

For example, the subtask information I4 includes a function indicating two subtasks of moving the robot hand (reaching) and grasping the robot hand (grasping) as subtasks that the robot 5 can accept when the target task is pick-and-place. is defined. In this case, the function "Move" representing reaching has, for example, the initial state of the robot 5 before executing the function, the final state of the robot 5 after executing the function, and the required time required to execute the function as arguments. is a function. Also, the function "Grasp" representing the grasping has, for example, the state of the robot 5 before execution of the function, the state of the object to be grasped before execution of the function, and logic variables indicating switching of dynamics as arguments. is a function. Here, the function "Grasp" represents the action of grasping an object when the logical variable is "1", and the action of releasing the object when the logical variable is "0". In this case, the subtask sequence generation unit 36 determines the function "Move" based on the trajectory of the robot hand determined by the control input at each time step indicated by the control input information Ic, and determines the function "Grasp" based on the control input information Ic. It is determined based on the transition of logic variables for each time step shown.

The subtask sequence generator 36 then generates a subtask sequence for the robot 5 composed of the function “Move” and the function “Grasp” and supplies the robot 5 with a control signal S4 indicating the subtask sequence. For example, if the target task is "finally, the object 61 exists in the area G", the subtask sequence generator 36 instructs the robot arm 52 to perform the function "Move", the function "Grasp", the function "Move", Create a subtask sequence for the function "Grasp". In this case, the robot arm 52 moves to the position of the target object 61 on the transfer device 82 by the first function "Move", grips the target object 61 by the first function "Grasp", and the second function " Move" to move to the destination area 63, and place the object 61 on the destination area 63 by the second function "Grasp".

Similarly, the subtask sequence generator 36 causes the peripheral device 8 to execute the subtasks that can be executed by the peripheral device 8 indicated by the subtask information I4 and the time-series control inputs of the peripheral device 8 indicated by the control input information Ic. Generate a subtask sequence. In this case, the subtask of the peripheral device 8 has at least a function for moving the object on the transport device 82 (also called "movement function"). The movement function has at least one of movement speed and acceleration as a parameter, and the subtask sequence generation unit 36 performs movement specified by the above-described parameter based on the time-series control input of the peripheral device 8 indicated by the control input information Ic. Generate subtask sequences containing functions. As subtasks of the peripheral device 8, not only the movement function but also various functions such as a function for instructing a change in movement direction and a function for instructing a pause may be registered in the subtask information I4.

(6) Processing flow
FIG. 7 is an example of a flowchart showing control processing executed by the control device 1 in the first embodiment.

First, the state measurement unit 30 of the control device 1 generates state information Im indicating the measurement result of the object in the work space 6 based on the output signal S3 supplied from the measurement device 7 . The abstract state setting unit 31 sets the abstract state in the work space 6 based on the state information Im (step S11). Next, the target logical formula generator 32 determines a target logical formula Ltag from the target task specified by the input signal S1 (step S12). In this case, the target logical expression generation unit 32 references the constraint information I2 to add the constraint on the execution of the target task to the target logical expression Ltag. In addition, the process of step S12 may be performed before step S11.

Then, the time step logical expression generator 33 converts the target logical expression Ltag into a time step logical expression Lts representing the state at each time step (step S13). In this case, the time step logical expression generator 33 determines the target number of time steps, and calculates the logical sum of the candidates φ representing the state at each time step that satisfies the target logical expression Ltag with the target number of time steps. Generate as the expression Lts. In this case, preferably, the time step logical expression generation unit 33 refers to the operation limit information I3 to determine the feasibility of each candidate φ, and selects the candidate φ that is determined to be impracticable as the time step Exclude from logical expression Lts.

Next, the abstract model generator 34 determines an abstract model Σ suitable for the target task based on the state information Im generated in step S11 and the abstract model information I5 (step S14). This abstract model Σ is a hybrid system that expresses dynamics switching according to the success or failure of the dynamics switching conditions for the robot 5 or the peripheral device 8 using discrete values. Then, the control input generator 35 determines the control input of the robot 5 and the control input of the peripheral device 8 that satisfy the abstract model Σ and the time step logical expression Lts and optimize the evaluation function (step S15). Then, the subtask sequence generator 36 determines a subtask sequence to be executed by the robot 5 and the peripheral device 8 from the control input determined by the control input generator 35, and outputs the subtask sequence to the robot 5 and the peripheral device 8 (step S16). . In this case, the subtask sequence generator 36 transmits the control signal S4 indicating the subtask sequence of the robot 5 to the robot 5 via the interface 13, and transmits the control signal S5 indicating the subtask sequence of the peripheral device 8 via the interface 13. to the peripheral device 8.

(7) Modification
The configuration of the functional blocks of the processor 11 shown in FIG. 4 is an example, and various modifications may be made. For example, instead of the processor 11 having the state measuring unit 30, the measuring device 7 may have a function corresponding to the state measuring unit 30 to generate the state information Im and supply the state information Im to the control device 1. good. In another example, the information of the candidate φ of the motion sequence to be commanded to the robot 5 and the peripheral device 8 is stored in advance in the storage device 4, and the control device 1 optimizes the control input generator 35 based on the information. to run. Thereby, the control device 1 selects the optimum candidate φ and determines the control input of the robot 5 and the peripheral device 8 . In this case, the control device 1 does not have to have functions corresponding to the abstract state setting section 31 , the target logical expression generating section 32 and the time step logical expression generating section 33 . Further, information about execution results of some functional blocks of the processor 11 shown in FIG. 4 may be stored in the application information storage unit 41 in advance.

<Second embodiment>
FIG. 8 is a schematic configuration diagram of the control device 1A in the second embodiment. The control device 1A in the second embodiment monitors the states of the robot 5 and the peripheral device 8 during operation according to the subtask sequence, and makes adjustments necessary for the robot 5 and the peripheral device 8 to execute given subtask sequences. It differs from the first embodiment in that it is performed. Henceforth, the code|symbol same as 1st Embodiment is attached|subjected about the component same as 1st Embodiment, and the description is abbreviate|omitted.

The control device 1A has the hardware configuration shown in FIG. 2 described in the first embodiment. As shown in FIG. 8, the processor 11 of the control device 1A functionally includes a state measuring section 30, an operation sequence generating section 37, and an integrated control section .

Based on the output signal S3 and the object model information I6, the state measurement unit 30 generates state information Im by performing the same processing as the state measurement unit 30 of the first embodiment, and sends the state information Im to the operation sequence generation unit 37. and the integrated control unit 38, respectively. Note that the processing corresponding to the state measuring unit 30 may be executed by the measuring device 7 .

The motion sequence generation unit 37 generates a subtask sequence (also referred to as “robot motion sequence Sr”) to be executed by the robot 5 and the peripheral device 8 based on various information stored in the application information storage unit 41 and the state information Im. A subtask sequence (also referred to as a "peripheral device operation sequence Sp") to be executed by each is generated. Here, the operation sequence generation unit 37 includes the abstract state setting unit 31, the target logical expression generation unit 32, the time step logical expression generation unit 33, the abstract model generation unit 34, and the control input generation unit shown in FIG. 4 in the first embodiment. 35 and a subtask sequence generation unit 36 .

The integrated control unit 38 controls the robot operation sequence Sr, the peripheral device operation sequence Sp, the state information Im, the state signal "S6" transmitted from the robot 5, the state signal "S7" transmitted from the peripheral device 8, and the The robot 5 and the peripheral device 8 are controlled based on. Here, the state signal S6 is an output signal of a sensor that detects the state of the robot 5 or a signal indicating the state of the robot 5 generated by the robot control section 51 . The state signal S7 is an output signal of a sensor that detects the state of the peripheral device 8 or a signal indicating the state of the peripheral device 8 generated by the peripheral device control section 81 . These status signals are information that directly or indirectly indicate the degree of progress of the subtasks of the robot 5 and the peripheral device 8 .

The integrated control unit 38 controls the state of the robot 5 and the peripheral device 8 measured during execution of the action sequence by the robot 5 and the peripheral device 8 (also referred to as "measured state"), and The predicted states of the robot 5 and the peripheral device 8 (also referred to as "predicted states") are compared. Note that the integrated control unit 38 estimates the measurement state based on at least one of the state signal S6 and the state signal S7, or the state information Im. Then, when the above-described measured state does not match the predicted state, the integrated control unit 38 controls at least one of the robot 5 and the peripheral device 8 so that the above-described measured state and predicted state are brought closer to each other.

FIG. 9 is an example of a flowchart showing control processing of the robot 5 and the peripheral device 8 executed by the control device 1A in the second embodiment. In addition, in the flowchart of FIG. 9, the state measurement part 30 always generates the state information Im based on the output signal S3 supplied from the measuring device 7 .

First, the motion sequence generator 37 generates the robot motion sequence Sr and the peripheral device motion sequence Sp based on the state information Im and the application information. The integrated control unit 38 transmits a control signal S4 indicating the robot operation sequence Sr to the robot 5, and transmits a control signal S5 indicating the peripheral device operation sequence Sp to the peripheral device 8 (step S21).

After that, the integrated control unit 38 measures each state of the robot 5 and the peripheral device 8 (step S22). In this case, the integrated control unit 38 recognizes the measurement state of the robot 5 from at least one of the state signal S6 and the state information Im, and recognizes the measurement state of the peripheral device 8 from at least one of the state signal S7 and the state information Im.

Next, the integrated control unit 38 determines whether or not the measured states of the robot 5 and the peripheral device 8 match the predicted states predicted based on the operation sequence (step S23). In this case, in the first example, the integrated control unit 38 measures and predicts the positions of the robot 5 and the peripheral device 8 as the aforementioned measurement state and prediction state. Then, the integrated control unit 38 determines that the measured state matches the predicted state when the difference between these positions (that is, the distance) is within a predetermined difference. In this case, in addition to the positions of the robot 5 and the peripheral device 8, it may be determined whether or not the measured state matches the predicted state in consideration of the posture. In a second example, the integrated control unit 38 recognizes the actual progress of the operation sequence (for example, the number or rate of completed subtasks) as the measurement state based on the state signals S6 and S7, and recognizes the operation sequence as the prediction state. Recognize the expected progress at the current timestep of the sequence. Then, the integrated control unit 38 determines that the measured state matches the predicted state when these progress degrees match.

If the measured states of the robot 5 and the peripheral device 8 do not match the predicted state based on the operation sequence (step S23; No), the integrated control unit 38 generates and outputs at least one of the control signal S4 and the control signal S5. (step S25). Thereby, the integrated control unit 38 controls at least one of the robot 5 and the peripheral device 8 so that the state of the robot 5 and the peripheral device 8 matches the predicted state based on the operation sequence. A specific aspect of step S25 will be described later.

On the other hand, if the measured states of the robot 5 and the peripheral device 8 match the predicted states based on the operation sequence (step S23; Yes), or after step S25 is executed, the control device 1A determines whether the target task has been completed. (step S24). The control device 1A determines whether or not the target task has been completed, for example, by recognizing the state of the object based on the output signal S3 supplied from the measuring device 7 . In another example, the control device 1A determines that the target task has been completed when the state signal S6 indicating the normal completion of the robot operation sequence Sr is received from the robot 5 . The control device 1A determines that the target task has been completed when it receives both the state signal S6 indicating the normal end of the robot operation sequence Sr and the state signal S7 indicating the normal end of the peripheral device operation sequence Sp. good too.

Here, the processing of step S25 will be supplementarily explained.

In the first example, when the state of at least one of the robot 5 and the peripheral device 8 is delayed from the state predicted based on the operation sequence, the integrated control unit 38 controls the entire operation sequence by the amount of the delay. At least one of the control signal S4 and the control signal S5 is generated for shifting the completion timing of . As a result, the integrated control unit 38 can prevent an operation that requires timing synchronization between the robot 5 and the peripheral device 8 from being unable to execute due to the operation delay of either the robot 5 or the peripheral device 8. Suppress preferably. In this case, for example, the integrated control unit 38 includes, in the control signal S4 or the control signal S5, information indicating the execution timing of each new subtask in consideration of the delay described above. In this case, the integrated control unit 38 instructs the one of the robot 5 and the peripheral device 8 whose motion is more advanced than the other to delay the motion timing by the motion delay time of the robot 5 and the peripheral device 8. may In still another example, the integrated control unit 38 may notify the robot 5 and the peripheral device 8 of the operation sequence reflecting the execution timing of each new subtask by using the control signal S4 or the control signal S5. When the integrated control unit 38 determines that the target task cannot be completed by adjusting the operation timings of the robot 5 and the peripheral device 8, it instructs the operation sequence generation unit 37 to regenerate the operation sequence. good too.

In the second example, when the integrated control unit 38 measures that either the robot 5 or the peripheral device 8 has stopped, the measured state of the robot 5 and the peripheral device 8 does not match the predicted state based on the operation sequence. Judge immediately. In this case, for example, the integrated control unit 38 transmits a control signal to the one that has not stopped, instructing it to stop the operation. In another example, the integrated control unit 38 may instruct the operation sequence generation unit 37 to regenerate the operation sequence, and output a warning to notify the administrator that an abnormality has occurred. You may

As described above, according to the second embodiment, the control device 1A monitors whether the robot 5 and the peripheral device 8 are operating according to the given subtask sequences after generating the subtask sequences, and operates according to the subtask sequences. The robot 5 and the peripheral device 8 can be controlled to do so.

<Third Embodiment>
FIG. 10 is a schematic configuration diagram of the control device 1B in the third embodiment. The control device 1B in the third embodiment generates a subtask sequence taking into account the fact that the object 61 and the obstacle 62 are moved by the transport device 82 even during the calculation of the subtask sequence, taking into account the calculation time of the subtask sequence. This differs from the first embodiment in that respect. Henceforth, the code|symbol same as 1st Embodiment is attached|subjected about the component same as 1st Embodiment, and the description is abbreviate|omitted.

The control device 1B has the hardware configuration shown in FIG. 2 described in the first embodiment. As shown in FIG. 10, the processor 11 of the control device 1B functionally includes a state measuring section 30, an operation sequence generating section 37, and a state predicting section 39. As shown in FIG.

Based on the output signal S3 and the object model information I6, the state measurement unit 30 generates state information Im by performing the same processing as the state measurement unit 30 of the first embodiment, and sends the state information Im to the state prediction unit 39. supply. Note that the processing corresponding to the state measuring unit 30 may be executed by the measuring device 7 .

Based on the state information Im, the state prediction unit 39 generates information (also called “predicted state information Imp”) indicating the state of the object in the work space 6 after a predetermined time (also called “calculation consideration time”) has elapsed. do. The calculation consideration time mentioned above is the calculation time expected to be required for calculation of the subtask sequence by the action sequence generator 37 and is stored in advance in the memory 12 or the storage device 4 .

In this case, the state prediction unit 39 predicts at least the objects placed on the conveying device 82 (the object 61 and the obstacle 62 in FIG. ) after a predetermined time has elapsed. Then, the state prediction unit 39 generates predicted state information Imp by updating the state information Im based on the predicted state of the placed object. For example, the state prediction unit 39 predicts the transport speed of the peripheral device 8 during calculation of the subtask sequence based on the state signal S7 related to the transport speed supplied from the peripheral device 8, and calculates the estimated transport speed and the calculation consideration time. Based on this, the state of the placed object after the calculation consideration time has elapsed is predicted.

The state prediction unit 39 determines the transport speed of the peripheral device 8 used for the above prediction based on the limit value or the past average value of the transport speed of the peripheral device 8 stored in the application information storage unit 41. good too. Further, the state prediction unit 39 may estimate the transport speed of the peripheral device 8 based on the state information Im when the peripheral device 8 is in operation. The state prediction unit 39 also recognizes the conveying direction based on at least one of the state signal S7, the state information Im, and the application information, like the conveying speed. Further, when an object other than the placed object (for example, the robot 5) is in motion, the state prediction unit 39 also predicts the state of the object after a predetermined period of time, and reflects the prediction result in the predicted state information Imp. You may In this case, the state prediction unit 39 uses time-series images acquired by the measuring device 7 within a predetermined period of time before generation of the predicted state information Imp to detect and move a moving object based on optical flow or the like based on image differences. A speed calculation may be performed.

The motion sequence generation unit 37 generates a robot motion sequence Sr to be executed by the robot 5 and a peripheral device motion sequence Sp to be executed by the peripheral device 8 based on the various information stored in the application information storage unit 41 and the predicted state information Imp. and respectively. The operation sequence generator 37 then transmits a control signal S4 indicating the robot operation sequence Sr to the robot 5, and transmits a control signal S5 indicating the peripheral device operation sequence Sp to the peripheral device 8. FIG. The operation sequence generation unit 37 includes the abstract state setting unit 31, the target logical expression generation unit 32, the time step logical expression generation unit 33, the abstract model generation unit 34, the control input generation unit 35, and the subtasks shown in FIG. 4 in the first embodiment. It has a function corresponding to the sequence generator 36 .

FIG. 11 is an example of a flowchart showing control processing of the robot 5 and the peripheral device 8 executed by the control device 1B in the third embodiment.

First, based on the output signal S3 supplied from the measuring device 7, the state measurement unit 30 generates state information Im indicating the measurement result of the object in the work space 6 (step S31). Based on the state information Im, the state prediction unit 39 generates predicted state information Imp indicating the state of each object in the work space 6 after the calculation consideration time has elapsed (step S32). Then, the motion sequence generator 37 generates the robot motion sequence Sr and the peripheral device motion sequence Sp based on the predicted state information Imp and the application information, and outputs these motion sequences to the robot 5 and the peripheral device 8 (step S33). ). In this case, the operation sequence generator 37 supplies the robot 5 with a control signal S4 indicating the robot operation sequence Sr in which the operation sequence calculation time is taken into account, and controls the control signal S4 to indicate the peripheral device operation sequence Sp in consideration of the operation sequence calculation time. A signal S5 is provided to the peripheral device 8;

Note that the third embodiment may be combined with the second embodiment. In this case, the processor 11 shown in FIG. 10 further functions as an integrated control unit 38 and controls the robot 5 and the peripheral device 8 so that the state of the robot 5 and the peripheral device 8 matches the predicted state based on the subtask sequence. .

<Fourth Embodiment>
FIG. 12 is a schematic configuration diagram of a control device 1C in the fourth embodiment. As shown in FIG. 12, the control device 1C has an operation sequence generator 37C.

The motion sequence generating means 37C is based on the robot motion information "Ir" indicating the motion characteristics of the robot that executes the task, and the peripheral device information "Ip" indicating the motion characteristics of the peripheral device that performs the task-related transfer with the robot. , motion sequences “Sra” and “Spa” indicating motions to be executed by the robot and the peripheral device, respectively.

In this context, a "robot" is any kind of robot that uses objects to perform tasks. In addition, the "peripheral device" is a device that transfers objects related to a task to a robot. It may be an industrial device that performs The "object" refers to an object to be transferred between the robot and the peripheral device, and corresponds to a tool, a part (member), or a case that accommodates these.

The robot motion information Ir is information on the motion characteristics of the robot 5 among the application information in the first to third embodiments (for example, constraint information I2, motion limit information I3, subtask information I4, and abstract model information I5). Further, the peripheral device information Ip is information about motion characteristics of the peripheral device 8 among the application information in the first to third embodiments (for example, constraint condition information I2, motion limit information I3, subtask information I4, abstract model information I5). is. The operation sequence generating means 37C includes, for example, the abstract state setting unit 31, the target logical expression generating unit 32, the time step logical expression generating unit 33, the abstract model generating unit 34, the control input generating unit 35, and the subtask sequence generating unit in the first embodiment. It is composed of a part 36 . Also, the operation sequence generating means 37C may be the operation sequence generating section 37 in the second embodiment or the third embodiment. The motion sequences Sra and Spa are the subtask sequences generated by the subtask sequence generator 36 in the first embodiment, or the robot motion sequences Sr generated by the motion sequence generator 37 in the second or third embodiment, and the peripheral devices. It can be an operation sequence Sp.

FIG. 13 is an example of a flowchart executed by the control device 1C in the fourth embodiment. Based on the robot motion information Ir and the peripheral device information Ip, the motion sequence generating means 37C generates motion sequences Sra and Spa indicating motions to be executed by the robot and the peripheral device, respectively (step S31).

According to the configuration of the fourth embodiment, the control device 1C can suitably generate the motion sequence of the robot and peripheral devices necessary for executing the task.

Note that in each of the above-described embodiments, the program can be stored using various types of non-transitory computer readable media and supplied to a processor or the like that is a computer. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (eg, flexible discs, magnetic tapes, hard disk drives), magneto-optical recording media (eg, magneto-optical discs), CD-ROMs (Read Only Memory), CD-Rs, CD-R/W, semiconductor memory (eg, mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (Random Access Memory)). The program may also be delivered to the computer on various types of transitory computer readable medium. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. Transitory computer-readable media can deliver the program to the computer via wired channels, such as wires and optical fibers, or wireless channels.

In addition, part or all of each of the above-described embodiments can be described as the following supplementary remarks, but is not limited to the following.

[Appendix 1]
Based on robot motion information indicating motion characteristics of a robot that executes a task and peripheral device motion information indicating motion characteristics of a peripheral device that transfers an object related to the task to the robot, the robot and the peripheral device are controlled. A control device having action sequence generating means for generating action sequences each indicating an action to be performed.

[Appendix 2]
1. The method according to appendix 1, further comprising integrated control means for controlling the motion of the robot and the peripheral device based on the state of the robot and the peripheral device measured during execution of the motion sequence and the motion sequence. controller.

[Appendix 3]
The integrated control means determines that the state of the robot and the peripheral devices measured during execution of the motion sequence does not match the predicted state of the robot and the peripheral devices when the motion sequence is followed. 3. The control device according to appendix 2, which adjusts the operation timing of at least one of the robot and the peripheral device, if the

[Appendix 4]
further comprising state prediction means for predicting the state of the robot in the work space after a predetermined time;
4. The action sequence generating means according to any one of Appendices 1 to 3, wherein the action sequence generating means generates the action sequence based on the state predicted by the state prediction means, the robot action information, and the peripheral device action information. Control device as described.

[Appendix 5]
5. The control device according to appendix 4, wherein the predetermined time is set to a calculation time expected to be required for calculation of the action sequence by the action sequence generating means.

[Appendix 6]
The robot motion information is at least one of motion limit information about motion limits of the robot, subtask information indicating subtasks executable by the robot, and abstract model information about an abstracted model of dynamics related to the robot. including
The peripheral device operation information includes operation limit information about the operation limit of the peripheral device, subtask information indicating subtasks that the peripheral device can execute, and abstract model information about a model that abstracts the dynamics related to the peripheral device. The control device according to any one of claims 1 to 5, comprising at least one of

[Appendix 7]
The abstract model information included in the robot motion information includes information indicating conditions for switching dynamics related to the robot,
7. The control device according to appendix 6, wherein the abstract model information included in the peripheral device operation information includes information indicating a dynamics switching condition related to the peripheral device.

[Appendix 8]
The peripheral device is a transport device that transports the object,
8. The control device according to any one of appendices 1 to 7, wherein the robot is a robot that performs at least an operation of picking up the object.

[Appendix 9]
The action sequence generating means receives an input designating the object or the type of the object by displaying an image of the work space of the robot on a display device, and generates the action sequence based on the input. 9. The control device according to any one of appendices 1 to 8.

[Appendix 10]
The operation sequence generating means is
a logical formula conversion means for converting the task into a logical formula based on temporal logic;
a time step logical expression generation means for generating a time step logical expression, which is a logical expression representing the state of each time step for executing the task, from the logical expression;
Subtask sequence generation means for generating, as the operation sequence, a sequence of subtasks to be executed by each of the robot and the peripheral device based on the time step logical expression. Control device as described.

[Appendix 11]
by computer
Based on robot motion information indicating motion characteristics of a robot that executes a task and peripheral device motion information indicating motion characteristics of a peripheral device that transfers an object related to the task to the robot, the robot and the peripheral device are controlled. A control method that generates a sequence of actions, each indicating an action to be performed.

[Appendix 12]
Based on robot motion information indicating motion characteristics of a robot that executes a task and peripheral device motion information indicating motion characteristics of a peripheral device that transfers an object related to the task to the robot, the robot and the peripheral device are controlled. A recording medium storing a program that causes a computer to function as an action sequence generating means for generating an action sequence indicating actions to be executed.

Although the present invention has been described with reference to the embodiments, the present invention is not limited to the above embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. That is, the present invention naturally includes various variations and modifications that a person skilled in the art can make according to the entire disclosure including the scope of claims and technical ideas. In addition, the disclosures of the cited patent documents and the like are incorporated herein by reference.

Reference Signs List 1, 1A to 1C control device 2 input device 3 display device 4 storage device 5 robot 6 work space 7 measurement device 8 peripheral device 41 application information storage unit 100 robot control system

Claims (9)

Based on robot motion information indicating motion characteristics of a robot that executes a task and peripheral device motion information indicating motion characteristics of a peripheral device that transfers an object related to the task to the robot, the robot and the peripheral device are controlled. an action sequence generating means for generating an action sequence indicating an action to be performed by each ;
When the states of the robot and the peripheral devices measured during execution of the motion sequence do not match the states expected of the robot and the peripheral devices when the robot and the peripheral devices operate according to the motion sequence, the robot and the peripheral devices Integrated control means for adjusting operation timing of at least one of the peripheral devices;
A control device having Based on robot motion information indicating motion characteristics of a robot that executes a task and peripheral device motion information indicating motion characteristics of a peripheral device that transfers an object related to the task to the robot, the robot and the peripheral device are controlled. an action sequence generating means for generating an action sequence indicating an action to be performed by each;
a state prediction means for predicting a state in the work space of the robot after a predetermined time;
The action sequence generating means generates the action sequence based on the state predicted by the state prediction means, the robot action information, and the peripheral device action information.
Control device. 3. The control device according to claim 2 , wherein said predetermined time is set to a calculation time expected to be required for said operation sequence generation means to calculate said operation sequence. Based on robot motion information indicating motion characteristics of a robot that executes a task and peripheral device motion information indicating motion characteristics of a peripheral device that transfers an object related to the task to the robot, the robot and the peripheral device are controlled. each having an operation sequence generating means for generating an operation sequence indicating an operation to be performed;
The robot motion information is at least one of motion limit information about motion limits of the robot, subtask information indicating subtasks executable by the robot, and abstract model information about an abstracted model of dynamics related to the robot. including
The peripheral device operation information includes operation limit information about the operation limit of the peripheral device, subtask information indicating subtasks that the peripheral device can execute, and abstract model information about a model that abstracts the dynamics related to the peripheral device. including at least one of
The abstract model information included in the robot motion information includes information indicating conditions for switching dynamics related to the robot,
The abstract model information included in the peripheral device operation information includes information indicating conditions for switching dynamics related to the peripheral device.
Control device. The peripheral device is a transport device that transports the object,
The control device according to any one of claims 1 to 4 , wherein the robot is a robot that performs at least an action of picking up the object. The action sequence generating means receives an input designating the object or the type of the object by displaying an image of the work space of the robot on a display device, and generates the action sequence based on the input. The control device according to any one of claims 1 to 5, wherein The operation sequence generating means is
a logical formula conversion means for converting the task into a logical formula based on temporal logic;
a time step logical expression generation means for generating a time step logical expression, which is a logical expression representing the state of each time step for executing the task, from the logical expression;
Subtask sequence generation means for generating a sequence of subtasks to be executed by each of said robot and said peripheral device as said operation sequence based on said time step logical expression. The control device according to . by computer
Based on robot motion information indicating motion characteristics of a robot that executes a task and peripheral device motion information indicating motion characteristics of a peripheral device that transfers an object related to the task to the robot, the robot and the peripheral device are controlled. generating a sequence of actions each indicating an action to be performed;
When the states of the robot and the peripheral devices measured during execution of the motion sequence do not match the states expected of the robot and the peripheral devices when the robot and the peripheral devices operate according to the motion sequence, the robot and the peripheral devices A control method for adjusting operation timing of at least one of peripheral devices . Based on robot motion information indicating motion characteristics of a robot that executes a task and peripheral device motion information indicating motion characteristics of a peripheral device that transfers an object related to the task to the robot, the robot and the peripheral device are controlled. an action sequence generating means for generating an action sequence indicating an action to be performed by each ;
When the states of the robot and the peripheral devices measured during execution of the motion sequence do not match the states expected of the robot and the peripheral devices when the robot and the peripheral devices operate according to the motion sequence, the robot and the peripheral devices Integrated control means for adjusting the operation timing of at least one of the peripheral devices
A program that makes a computer function as a

Images