JP2021030407A - Information processing device, information processing method and program - Google Patents

Abstract

To cause a robot to execute work even though an instruction from a user is insufficient.SOLUTION: According to the present invention, an information processing device causes a robot to execute a work group composed of a plurality of work pieces. The information processing device includes input means for inputting instruction information indicating at least one portion of the work group, and specification means for specifying a work group including work indicated by the indication information in the work group of a plurality of patterns as a candidate for a work group to be executed by the robot.SELECTED DRAWING: Figure 1

Description

Regarding work instructions to the robot.

Conventionally, a work instruction of a robot device or the like having a voice recognition function is programmed in advance by associating a specific work with a specific command word, so that the instructor utters a specific command word. By doing so, the robot was performing a specific task. On the other hand, in Patent Document 1, a combination of a plurality of operations is executed for one work instruction.

JP-A-2002-337075

However, in Patent Document 1, when the robot is made to perform a plurality of operations, the robot can execute only a combination of the operations already taught in response to one instruction from the user. For example, in a robot that assembles a product made by combining a plurality of parts, there is a case where an instruction or a work group that is a combination of information and work is not taught in advance as to which product is to be made. In that case, if the robot lacks instructions for the robot to identify a series of work groups, the robot may not be able to perform some of the necessary tasks in the series of work groups. Therefore, if the instructions from the user are insufficient for the robot, the robot cannot specify a series of work groups, and the robot cannot execute the series of work groups intended by the user's instructions.

Therefore, an object of the present invention is to make the robot execute a work group even if there is a lack of instructions from the user.

The information processing device according to the present invention that solves the above problems is an information processing device that causes a robot to execute a work group composed of a plurality of tasks, and is an input for inputting instruction information indicating at least a part of the tasks in the work group. It is characterized by having means and specific means for specifying a work group including the work indicated by the instruction information as a candidate of the work group to be executed by the robot among the work groups of a plurality of patterns.

According to the present invention, the robot can execute the work group even if the instruction from the user is insufficient.

Diagram showing a configuration example of an information processing system Block diagram showing a functional configuration example of an information processing device Flow chart explaining the processing executed by the information processing system Diagram explaining the state transition of an object Diagram showing an example of the state of an object Diagram showing an example of image information Diagram showing an example of the target state Diagram showing the coordinate system of an information processing system Block diagram showing a functional configuration example of an information processing device Flow chart explaining the processing executed by the information processing system Diagram showing an example of GUI Block diagram showing a functional configuration example of an information processing device Flow chart explaining the processing executed by the information processing system Block diagram showing a functional configuration example of an information processing device The figure which shows the hardware configuration example of an information processing apparatus Flow chart explaining the processing executed by the information processing device Flowchart explaining the process executed by the work group identification unit Flowchart explaining the process executed by the work group identification unit

Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. The configuration shown in the following embodiments is only an example, and the present invention is not limited to the illustrated configuration.

<First Embodiment>
In the first embodiment, in the process of packing food for a lunch box by a robot, when the instruction given by the instructor is insufficient, the robot specifies a work group to be executed by the robot by the information processing apparatus, so that the robot can use the user. An example of performing the work instructed in is described. For example, some factories are veteran workers and others are temporarily hired for part-time jobs. In the latter case, the robot is not always well-versed in how to give instructions. For example, if you do not remember the name or model number of the product to be made, you may not be able to tell the robot using the name or model number of the product to be made next. Moreover, in a factory that produces a large number of items in a small number, the production items are changed many times in a day. In such a field, it takes more time and effort to instruct the robot to change the work one by one, such as associating the model number of the product with the work content of the robot in advance, as the number of robots and the number of production items increases. In addition, since factory workers spend a lot of time not having their hands free during work, it takes less time and effort to give instructions by voice than to give instructions to a robot using a communication terminal such as a tablet. However, noise in the factory may prevent you from entering complete voice instructions into the device of interest. That is, when a factory worker gives a voice instruction to a collaborating robot, the user does not always input an appropriate instruction. An embodiment effective under such a situation will be described below.

For example, the instructor gives a third object to a robot that is performing the work of arranging a second object on the first object and arranging the third object on the second object. Instead of, let's do the work of arranging a fourth object. At this time, the command word spoken by the instructor is not only an instruction to change to the fourth object, but also a second object is placed on the first object and the fourth object is placed on the second object. You need instructions to place the object in. Therefore, if the instructor omits the contents or gives an inaccurate order of instructions, the robot cannot be made to execute the work group intended by the instructor. In other words, since only the information on the changes is given to the robot, the missing information must be supplemented to the robot itself. Therefore, in the present embodiment, a device for causing the robot to execute a desired work group even with only a partial instruction by the user will be described.

FIG. 1 is a configuration example of an information processing system 100 including the information processing device 10 according to the present embodiment. The voice input device 1 is a device that converts sound into an electric signal, and specifically, a microphone or the like. The voice input device 1 acquires the voice emitted by the instructor as voice information. The voice information is an operation instruction from the instructor. Specifically, if the production target changes from an arbitrary bento to a pickled plum lunch, it is an instruction such as "Because the next bento is pickled plum, set it." The voice information output from the voice input device 1 is taken in as an input of the information processing device 10. The voice input device 1 may be arranged in a fixed manner with respect to the environment, or may be mounted on a moving mechanism portion such as a robot. Further, a plurality of voice input devices 1 may be arranged. When a plurality of voice input devices 1 are arranged, only the voice in a specific direction may be acquired by using the phase difference. Alternatively, by using a directional microphone, only the sound in a specific direction may be acquired.

The image pickup device 2 is a visual sensor such as a digital camera using an electronic image pickup device having a lens, and acquires the state of the target object 5 as image information. Specifically, the image information is image information obtained by capturing an image of the initial state in which the target object 5 is supplied. Two-dimensional color image information is acquired as image information, and distance image information is acquired in the case of a stereo camera. The image information output from the image pickup apparatus 1 is taken in as an input of the information processing apparatus 10. The light source 6 irradiates an illumination that reveals the pattern or edge of the target object. Further, when the light source 6 is a projector, a distance image can be acquired by irradiating the pattern light. The image pickup device 2 and the light source 6 may be mounted on a moving mechanism portion such as a robot, in addition to fixing the arrangement with respect to the image pickup target. Further, a plurality of image pickup devices 2 may be arranged.

The target object 5 mainly refers to a material existing in the work space of the robot. For example, those discharged from the manufacturing apparatus one after another by a belt conveyor or the like, or those arranged near the belt conveyor and simultaneously within the imaging range of the imaging apparatus 2 are targeted. Further, the target object 5 may be a part of an area of the object, not the entire object. In the present embodiment, the case where the target object moves within the range of the image pickup apparatus 2 is shown, but when the work is performed on a plurality of arranged target objects without using the belt conveyor, the system can be moved. It can be dealt with by doing.

The robot 20 is, for example, an articulated robot, and includes a manipulator 202 such as a robot arm, a gripping device 203 such as a robot hand, and a controller 201 that controls the manipulator 202 and the gripping device 203. Further, the robot 10 is provided with a position / posture changing mechanism capable of changing the position / posture of the gripping device 203 by changing the angle of each joint of the manipulator 202. The position / orientation changing mechanism may be driven by an electric motor or an actuator that operates by a fluid pressure such as hydraulic pressure or pneumatic pressure. This position / orientation changing mechanism is driven according to the control information output from the information processing device 10. Further, the robot 20 is not limited to the articulated robot, and may be a movable machine capable of numerical control (NC). The controller 201 controls the manipulator 202 and the gripping device 203 based on the position / attitude control information sent from the generation unit 107. The controller 201 may be a robot controller attached to the manipulator 202, a programmable logic controller (PLC), or any other device that can control the manipulator 202 and the gripping device 203.

The robot 20 executes a work group that is continuously executed in a plurality of patterns. Specifically, the work of moving to the position / posture for capturing an image in order to recognize the target object 5 which is a material, and the work of operating the target object 5 based on the position / posture control information presented by the information processing apparatus 10. Execute.

The gripping device 203 is a tool for realizing an operation according to the type of the target object by the robot 20, and is a hand that has a chuck mechanism that can be driven by a motor and can grip the target object, or sucks the target object by air pressure. A hand using a suction pad can be used. The gripping device 203 is detachably attached to the manipulator 202 and can be replaced according to the type of the target object. Further, the gripping device 203 is not always necessary, and may not be necessary as long as the target object can be operated by the manipulator 202. For example, in the case of the work of pushing out the target object, the robot 20 is composed of a uniaxial cylinder and does not have to include the gripping device 203.

FIG. 2 is a functional block diagram of the information processing system 100 and the information processing device 10. The information processing device 10 includes an instruction input unit 101, a visual information acquisition unit 102, a work storage unit 103, an estimation unit 104, a state storage unit 105, a work group identification unit 106, and a generation unit 107.

The instruction input unit 101 accepts the input of an instruction from the user as instruction information.

The visual information acquisition unit 102 acquires the visual information output from the image pickup apparatus 2. The visual information acquisition unit 102 converts the acquired visual information into image information and outputs it to the estimation unit 104. The method of converting visual information into image information will be described later. The image information includes the captured image and the recognition result of the object included in the captured image. The visual information acquisition unit 102 is composed of, for example, a capture board or a memory (RAM).

The work storage unit 103 stores a plurality of patterns of work groups to be continuously executed by the robot. For example, it is a series of operations of the information processing system 100 for grasping, transporting, and arranging the target object 5. The work for the robot to operate the target object 5 and the work group indicating the order thereof are also described as a work pattern.

The estimation unit 104 estimates at least a part of the work of the work group from the work group stored in the work storage unit 103 and the instruction information input. The estimation unit 104 sends work information about the estimated work to the work group identification unit 106. Further, the estimation unit 104 estimates the state of the target object from the input instruction information or image information based on the state transition information stored in the state storage unit 105. A detailed description will be described later.

The state storage unit 105 stores the text information obtained by converting the voice information acquired by the voice input device 1 and the information regarding the state and operation of the target object 5, and inputs the information to the work group identification unit 106. Here, the information regarding the state of the target object 5 includes the current state (initial state) in which the target object 5 is supplied, the target state (target state) after the work on the target object 5 is completed, and the target state (target state). It is a state in the middle of the process of transitioning from the current state to the target state (intermediate state). Further, the information regarding the operation of the target object 5 is an operation performed by the information processing system 100 on the target object 5 in the process of transitioning the target object 5 from the initial state to the target state. The operation includes actions on the target object 5 that change the position and orientation of the target object 5, such as gripping, transporting, and arranging by the robot 20, and the position and orientation of the imaging device 2 and the gripping device 203 for inspecting the target object 5. It is an action that changes. Further, the state storage unit 105 stores the state transition information of the target object 5. The state transition information is information on the initial state of the target object which is a material, information on an intermediate state or a target state constructed by using the material, and information on whether or not each state can be transitioned to the next state. Information on the initial state, intermediate state, and target state is stored as a combination of image information and text information. In the case where a plurality of target objects 5 are included, the relative positional relationship between the target objects 5 can be known based on one or more of the image information and the text information. Further, the working storage unit 103 may store information related to the operation of the object. For example, when the specific target object 5 is operated by the information processing system 100, information on the gripping position with respect to the target object 5 and information on the gripping force at the time of gripping may be stored. Further, the working storage unit 103 may store information regarding the transition method between states. For example, when the target object 5 is transported from one state to another when the information processing system 100 transports the target object 5, the information on the transport speed may be limited or the locus of the transport path may be stored. Further, the working storage unit 103 may store the target object 5 as information for each specific object, or may store it as information for each class of objects. However, in the former case, it is assumed that only the objects stored in advance are targeted when the processing of the information processing system 100 is executed. Whereas objects that are not memorized are not targeted, in the latter case, objects that are not memorized in advance can also be classified into either class. For example, when there are "apple", "mandarin orange", and "peach" as the target object 5, state transition information may be stored for each of "apple", "mandarin orange", and "peach", and the three are collectively called "fruit". You may remember it in class.

The work group identification unit 106 identifies, as the work indicated by the instruction information, the work estimated by the estimation unit 104 from the plurality of patterns of work groups as a candidate for the work group to be executed by the robot. Among the work groups stored in the work storage unit 103, the work group including the work estimated by the estimation unit 104 is specified. If the estimated work is included in multiple patterns of work groups, identify work group candidates. By acquiring additional information, the work group to be further executed is specified from the work group candidates. A detailed description will be described later. The work group specifying unit 106 sends the specified work group to the generation unit 107.

The generation unit 107 generates control information of the robot 20 for operating the target object 5 based on the work group specified by the work group identification unit 106 and the image information sent from the visual information acquisition unit 102. Here, the direction and amount of movement of the robot are generated as control information from the position / posture to be taken by the robot and the current position / posture. The generation unit 107 sends the generated control information to the robot 20.

The outline of the process executed by the information processing apparatus 10 will be described with reference to FIG. In the following description, the description of the process (step) is omitted by adding S at the beginning of each process (step). In S1601, the CPU 11 initializes. In S1602, the instruction input unit 101 receives the input of the instruction. In S1603, the estimation unit 104 estimates the work content from the input instruction. In S1604, the work group specifying unit 106 identifies a work group including the work indicated by the instruction information as a candidate of the work group to be executed by the robot from the work groups of a plurality of patterns. In S1605, the generation unit 107 generates control information of the robot 20 for operating the target object 5 based on the specified work group and the image information sent from the visual information acquisition unit 102. In S1606, the instruction input unit 101 determines whether or not there is an additional instruction. If there is an end instruction, the process ends. If there is no end instruction, the process returns to S1602. If the instruction input is not newly performed, the same work group may be continuously executed.

FIG. 3 is a flowchart showing a process executed by the information processing system 100 and the information processing device 10. The process shown in FIG. 3 is realized by the CPU 11 of the information processing apparatus 10 shown in FIG. 15 reading and executing a program stored in the ROM 12 or the external memory 14. However, a part or all of the processing of FIG. 3 may be realized by dedicated hardware. The process of FIG. 3 is started, for example, when the operator starts the information processing system 100. However, the start timing is not limited to when the information processing system 100 is started.

The outline of the processing executed by the information processing system 100 will be described. In S1, the CPU 11 initializes the system. This corresponds to the process of S1601 in FIG. In S2, the voice input device 1 inputs the voice information instructed by the user to the robot to the information processing device. This corresponds to the process of S1602 in FIG. In S3, the estimation unit 104 estimates the work content from the input instruction. This corresponds to the process of S1603 in FIG. In S4, the controller 201 determines the position and orientation of the image pickup device 2. In S5, the controller 201 moves and controls the manipulator 202 to the imaging position. In S6, the imaging device 2 images the target object 5 in the supply region. In S7, the visual information acquisition unit 102 acquires image information obtained by capturing an image of the target object in the initial state. In S8, the work group identification unit 106 identifies a work group including the work indicated by the instruction information as a candidate for the work group from the work groups of a plurality of patterns. This corresponds to the process of S1604 in FIG. Further, the work group to be executed by the robot is specified based on the auxiliary information and the candidate work group. In S9, the work group specifying unit 106 determines the work order based on the specified work group. In S10, the controller 201 determines the shooting position and orientation of the target object 5. In S11, the controller 201 controls the manipulator 202 for imaging. In S12, the image pickup apparatus 2 takes an image of the target object 5 from the position and orientation determined in S10. In S13, the visual information acquisition unit 102 acquires the image information of the target object 5 output by the image pickup apparatus 2 and sends it to the generation unit 107. In S14, the generation unit 107 generates the position / attitude control information of the robot 20. This corresponds to the process of S1605 in FIG. In S15, the controller 201 controls the manipulator 202 and the gripping device 203 for operating the target object 5. In S16, the manipulator 202 executes operations on the target object 5 such as the operation, gripping, transporting, and arranging of the robot 20 based on the control result of S15. In S17, the CPU 21 determines whether or not there is the next target object 5.

First, in S1, the CPU 11 performs a system initialization process. That is, the CPU 11 loads the program stored in the ROM 12 or the external memory 14 and expands it on the RAM 13 so that it can be executed. In addition, the parameters of each device connected to the information processing device 10 are read and returned to the initial position to make them usable.

In S2, the voice input device 1 inputs the voice information instructed by the user to the robot to the information processing device. This corresponds to the process of S1602 in FIG. That is, the instruction input unit 101 inputs an instruction from the user. For example, the voice input device 1 starts inputting voice information when the instructor issues a specific keyword, and ends inputting voice information when the voice is not input for a certain period of time. The voice input method is not limited to this, and for example, the voice input device may start recording when the voice of a specific person is detected. The input instruction information is not limited to voice information. For example, it may be video information obtained by capturing a gesture or operation information by a predetermined operation from a keyboard or an information terminal.

In S3, the estimation unit 104 estimates the work content from the input instruction. This corresponds to the process of S1603 in FIG. Here, the recognition process of the voice information input in S2 is performed. The instruction information that is the result of recognizing the voice information includes information regarding the state of the target object. The instruction information is input to the estimation unit 104. The voice information recognition process converts voice information into text information using, for example, a hidden Markov model. Then, information on the state and operation of the target object 5 is extracted from the text information by the Seq2Seq (sequence to sequence) model using LSTM (Long short-term memory). However, the method of obtaining information on the state and operation of the target object 5 may be obtained by classifying the text information into words by morphological analysis and then estimating the part of speech for each word, or by other methods. Good. Voice information may be converted into instruction information using other existing techniques.

In S4, the controller 201 determines the position / orientation of the manipulator 202 at the time of photographing based on the preset position / orientation for the imaging by the imaging device 2. Alternatively, an imaging position / orientation determination unit (not shown) determines based on the current position / orientation of the robot and sends it to the controller 201. The position and orientation to be determined may be determined by any method as long as the target object 5 in the supply region can be imaged. If the supply area is wide and it is difficult to acquire visual information of the supply area with one imaging, a plurality of positions and orientations are determined, and the position and orientation are determined using the results of acquiring a plurality of visual information. You may. Further, in the target work, when visual information of a region different from the supply region is required in order to transition the target object 5 to the target state, the imaging position / posture may be determined for the target other than the supply region. For example, in the case of the work of moving the target object 5 from the supply area to an arrangement area different from the supply area, the imaging position / orientation for acquiring the visual information of the arrangement area may be determined and the image may be taken. In the present embodiment, it is assumed that only the materials (objects) necessary for the target state are prepared in the supply area. When changing to another target state, the material (object) placed in the supply area is changed according to the change.

In S5, the controller 201 controls the manipulator 202 for imaging. Specifically, it is determined how to move the manipulator 202 to the position / posture determined in S4. For example, in order to move the manipulator 202 with respect to the position / posture generated in S4, the target position / posture information is converted into the joint angle information of the manipulator 202 by forward kinematics. The command value for operating the actuator of each joint of the manipulator 202 is calculated. The manipulator 202 operates based on the calculation result.

In S6, the imaging device 2 images the target object 5 in the supply region. The target object at this time is defined as the target object in the initial state.

In S7, the visual information acquisition unit 102 acquires image information obtained by capturing an image of the target object in the initial state. That is, the image information of the supply area output by the image pickup apparatus 2 is acquired and input to the estimation unit 104. From this image information, the initial state of the target object can be specified by the estimation unit 104.

In S8, the work group identification unit 106 identifies a work group including the work indicated by the instruction information as a candidate for the work group from the work groups of a plurality of patterns. This corresponds to the process of S1604 in FIG.

The outline of the process of S8 will be described with reference to FIG. In S181, the work group identification unit 106 first identifies the instructed work. That is, the information about the work estimated in S3 is acquired. In S182, the work group identification unit 106 acquires a plurality of patterns of work groups to be continuously executed by the robot. In S183, the work group identification unit 106 collates a plurality of patterns of work groups with the instructed work, and identifies candidates for the work group. In S184, the work group identification unit 106 determines whether or not there are a plurality of work groups including the instructed work. If there are a plurality, the process proceeds to S185. If it can be identified as one work group, the process proceeds to S186. In S185, the work group specifying unit 106 acquires auxiliary information for specifying the work group to be executed by the robot. Here, based on the target object estimated from the image information and the state transition information about the target object, the work that can be performed using the target object is estimated. In S186, the work group identification unit 106 identifies the work group to be executed by the robot from the work group candidates based on the auxiliary information. By performing such processing, it is possible to identify a work group to be executed by the robot even in a situation where the instruction is insufficient or the instruction cannot be recognized well.

Specifically, a case where this system is used in the process of manufacturing Hinomaru bento in a bento manufacturing line will be described. The instructor aims to make a Hinomaru bento (target state) by first putting white rice in an empty lunch box and then putting dried plums on the white rice. At this time, since it is obvious to the instructor to put the white rice before putting the pickled plums in the lunch box, as a work instruction to the information processing system 100, he said, "Put the pickled plums in the lunch box." Suppose you make a statement. The voice input device 1 inputs this voice information to the instruction input unit 101. From this instruction, the robot can recognize the work of "putting dried plums". However, when the robot 10 executes only the instructed work, only the pickled plums are placed in the lunch box and there is no white rice. That is, the robot cannot make the desired Hinomaru bento from the instructions input by the user. That is, it can be said that the instruction information to the robot recognized from the voice information is an incomplete instruction because it refers only to the work of "installing the pickled plums in (the position where the lunch box has been taught)". Therefore, the estimation unit 104 identifies the work group intended by the user from the input instructions, and causes the robot to execute continuous work.

In S181, the work group identification unit 106 accepts the work estimated by the estimation unit 104 in order to specify the work group. Here, specifically, the work of "putting dried plums" (work 3 of FIG. 4A) is specified. In S182, the work group identification unit 106 acquires a plurality of patterns of work groups continuously executed by the robot from the work storage unit 103. Specifically, for example, in the work group (called work group F) for making "Hinomaru bento (target state F)", "install a lunch box (work 1) → white rice is a lunch box" in FIG. 4 (A). Put it in a box (work 2) → put dried plums (work 3) ". In addition, the work group G and the work group E are stored in the work storage unit 103. A specific example of the work group is shown in FIG. 4 (C). In S183, the work group identification unit 106 identifies a candidate for the work group including the work instructed by the user by collating the work group of the plurality of patterns with the work received in S181. For example, from FIG. 4C, the work group including the work 3 is specified as the work group F.

In S184, the work group identification unit 106 determines whether or not there are a plurality of work groups including the instructed work. If there are a plurality, the process proceeds to S185. If it can be identified as one work group, the process proceeds to S186. Here, since the work 3 is not included in the other work groups other than the work group F specified as the candidate, the process proceeds to S186. Here, for example, as the work group H, consider a work group of "work 1-> work 2-> work 3-> work 4". At this time, in S183, the work group F and the work group H are specified as candidates for the work group. In that case, the process proceeds to S185.

In S185, the work group identification unit 106 acquires auxiliary information for specifying the work group to be executed by the robot. Multiple cases are assumed for auxiliary information. As a first case, the case where the supplementary information is a selection instruction input by the user will be described. For example, the candidate of the work group is output to a display device or the like, and the user is made to input an instruction. As the second case, the case where the material information is related to the material used in any of the work groups of a plurality of patterns will be described. Here, the material estimated from the image information and the state transition information indicating the target state using the material are acquired. Material information is information that identifies a material existing in the robot's work space. Specifically, the result of recognizing the material existing in the robot work space is used based on the image obtained by capturing the robot work space. Here, it is assumed that only the materials used for production are installed in the work space. As a third case, a case where the auxiliary information is production information indicating the sales status of the production item will be described. For example, the production information about the past sales quantity is stored in the state storage unit 105. Identify the work group corresponding to the production items whose past sales quantity is larger than the predetermined value, and prioritize the production of the best-selling production items. Information other than these may be used as the auxiliary information.

In S186, the work group identification unit 106 determines the work group to be executed by the robot from the work group candidates based on the auxiliary information. When the work group can be specified as one in S184, the work group is determined as the work group to be executed by the robot. When the auxiliary information is an input instruction, the work group instructed by the user is specified. Since the user can complement the instructions, the robot can accurately recognize the intention of the user, and human error can be reduced. If the auxiliary information is material information, identify a work group that includes work on the recognized material. For example, when the captured image includes a lunch box, white rice, and dried plums, steps 1 to 3 can be performed. On the other hand, work 4 is infeasible because it does not contain pickles. From these recognition results, the work group F is specified. When recognizing a material using image information, the robot itself can complement the instruction without taking the trouble of the user, so that the work can be executed efficiently. When the auxiliary information is production information, the work group corresponding to the production item with the highest sales quantity is specified from the work group including the instructed work. In this case as well, the robot itself can complement the instructions without taking the trouble of the user, so that the work can be executed efficiently.

In S9, the work group specifying unit 106 determines the execution order of the work based on the specified work group. The work storage unit 105 stores a plurality of patterns of work groups for the robot 20 to operate the target object 5. The work pattern is the operation of the information processing system 100, and may be gripping, transporting, arranging, or other work. Since the order in which the work is executed differs depending on the work group, the timing for executing the instructed work is determined according to the specified work group. Depending on the work group, there are tasks in no particular order. At that time, for example, the work may be performed in the order in which the distance between the material and the robot is short.

In S10, the controller 201 determines the shooting position and orientation of the target object 5. The controller 201 determines the position and orientation in which the manipulator 202 moves for shooting by the image pickup device 2. Alternatively, it is determined by an imaging position / orientation determining means (not shown) and sent to the controller 201. The imaging position / posture determined in S4 targets the supply region, but since the target object 5 is targeted in S10, the imaging position / posture suitable for determining the gripping position / posture of the target object 5 is determined. The position and orientation to be determined may be determined by any method as long as the target object 5 can be imaged.

In S11, the controller 201 controls the manipulator 202 for imaging. In S12, the image pickup apparatus 2 takes an image of the target object 5 from the position and orientation determined in S10. In S13, the visual information acquisition unit 102 acquires the image information of the target object 5 output by the image pickup apparatus 2 and sends it to the generation unit 107.

In S14, the generation unit 107 generates the position / attitude control information of the robot 20. When gripping the target object 5, the generation unit 107 acquires the image information of the target object 5 captured in S12 and determines the gripping position / posture of the target object 5. The determination method may be any method as long as the gripping position and posture of the target object 5 can be obtained. For example, if the template image of the target object 5 is stored, it is obtained based on the gripping position and posture set in advance with respect to the object coordinate system of the template. Specifically, first, the gripping position and posture are set with respect to the object coordinate system set in advance in the template of the target object 5. At this time, since the gripping position posture differs depending on the type of the gripping device 3, the setting is performed according to each gripping method such as gripping by a chuck or gripping by suction. Then, when the processing of the information processing system 100 is executed, if the position and orientation of the target object 5 are obtained by template matching, the object coordinate system of the target object 5 is obtained, so that the gripping position and orientation can be relatively obtained. Alternatively, the gripping position / posture may be directly obtained from the image information of the target object 5 by using the model learned in advance by deep learning. Alternatively, when the gripping device 203 grips with the chuck, the gripping position / posture may be determined by obtaining the facing surface of the target object 5 based on the gripping opening / closing width. Alternatively, when the gripping device 203 grips by suction, the gripping position / posture may be determined by obtaining a flat portion capable of sucking the target object 5 based on the dimensions of the suction pad, or may be determined by a method other than these. Good.

Further, in S14, the position / attitude control information of the robot when transporting or arranging the target object 5 is also determined. As for the method of determining the arrangement position / posture control information, for example, the image information of the target object 5 captured in S12 is acquired and determined based on the work group determined in S10, similarly to the determination of the gripping position / posture. When the target object 5 is placed on another target object 5 based on the work group determination result, since the relative positional relationship between the two objects is known, the target object 5 is gripped based on the position / orientation information of the placement destination target object 5. The placement position and orientation of the target object 5 are determined. Further, the generation unit 107 may determine the position / attitude control information not only as a certain point but also as a plurality of continuous points and loci. For example, the transport path for transporting the target object 5 from the determined grip position posture to the arrangement position posture may be determined. Further, the position / attitude control information for operations other than gripping, arranging, and transporting may be determined. For example, the position and orientation of the robot 20 for moving the target object 5 without grasping it (glass press manipulation) and the imaging position and orientation of the robot 20 for inspecting the target object 5 may be determined. The position and posture may be determined for the purpose of work other than these.

In S15, the controller 201 controls the manipulator 202 and the gripping device 203 for operating the target object 5. Specifically, it is determined how to control the manipulator 202 and the gripping device 203 based on the position and orientation determined in S14. Here, the position / posture determined in S15 is the position / posture in the image coordinate system, but since it is necessary to convert to the robot coordinate system in order to control the robot 20, coordinate conversion is performed by taking as an example the case of obtaining the gripping position / posture. Will be described.

FIG. 8 is a diagram showing the relationship of the coordinate system. The coordinate system Σc of the image pickup device 2, the base coordinate system Σb of the manipulator 202, the mechanical interface coordinate system Σm at the tip of the manipulator 202, the tool coordinate system Σt at the tip of the gripping device 203, and the target object coordinate system when gripping the target object 5. Treat Σb in a unified manner. Therefore, as shown in FIG. 8, the world coordinate system Σw is set as the reference coordinate system in the work space. First, let the displacement from the world coordinate system Σw to the manipulator-based coordinate system Σb be (BX, BY, BZ). Further, a 3 × 3 rotation matrix representing the posture of the manipulator 202 is defined as BM. The displacement from the manipulator base coordinate system Σb to the manipulator tip coordinate system Σm is (MX, MY, MZ). Further, the 3 × 3 rotation matrix representing the posture of the tip of the manipulator is MM. Further, the displacement from the manipulator tip coordinate system Σm to the coordinate system Σt of the gripping device 203 is (TX, TY, TZ). Further, the rotation matrix of 3 × 3 representing the posture of the tip of the gripping device 203 is TM.

The tip of the gripping device 203 is a portion where the gripping device 3 comes into contact with the target object 5, and when the gripping device 203 is a chuck as shown in FIG. 8, it is the center of the chuck tip. The displacement from the manipulator tip coordinate system Σm to the image pickup device coordinate system Σc is (CX, CY, CZ). Further, the 3 × 3 rotation matrix representing the posture of the image pickup apparatus 2 is CM.

Further, the displacement from the image pickup device coordinate system Σc to the target object coordinate system Σb of the target object 5 is (BX, BY, BZ). Further, the 3 × 3 rotation matrix representing the posture of the target object 5 is BM. Here, the displacement of the target object 5 as seen from the world coordinate system Σw is (WX, WY, WZ), and the 3 × 3 rotation matrix representing the posture is WM.

Here, consider the case where the gripping device 203 attached to the tip of the manipulator 202 is in contact with the target object 5. Assuming that the displacement from the manipulator base coordinate system Σb to the manipulator tip coordinate system Σm is (MX1, MY1, MZ1) and the 3 × 3 rotation matrix representing the posture of the manipulator tip is MM1, the following mathematical formula (1) is established.

Further, when the target object 5 is imaged by the image pickup apparatus 2, the displacement from the manipulator base coordinate system Σb to the manipulator tip coordinate system Σm is defined as (MX2, MY2, MZ2). Assuming that the 3 × 3 rotation matrix representing the posture of the tip of the manipulator is MM2, the following mathematical formula (2) holds.

Since the above formulas (1) and (2) represent the position and orientation of the target object 5 in the world coordinate system Σw, the following formula (3) holds.

From the mathematical formula (3), when the following formula is known, the position and orientation of the manipulator 203 when the target object 5 is gripped can be obtained. (The position and orientation of the manipulator 202 when the target object 5 is imaged, the relationship between the position and orientation of the manipulator tip coordinate system Σm and the image pickup device coordinate system Σc, the relationship between the position and orientation of the image pickup device coordinate system Σc and the target object 5, the manipulator tip coordinates. (Relationship between the system Σm and the position / orientation of the gripping device 203.) Therefore, the position / orientation of the manipulator 202 for gripping the target object 5 can be obtained from the image obtained by the imaging device 2 capturing the target object 5.

Here, an example of how to obtain each displacement and rotation matrix will be described. (BX, BY, BZ) and BM are obtained from the positional relationship with the world coordinate system Σw set at the time of installing the manipulator 202. (MX, MY, MZ) and MM are obtained by forward kinematics from the joint angle information of the manipulator 202. (TX, TY, TZ) and TM are obtained from the dimensions of the gripping device 203. (CX, CY, CZ) and CM are obtained from the dimensions of the image pickup apparatus 2. Alternatively, it may be obtained by calibration from the relative position-posture relationship between the image pickup apparatus 2 and the manipulator 202. For example, in each state in which the manipulator 202 takes a plurality of different positions and postures, the relative positional relationship with the manipulator 202 obtained by imaging a known two-dimensional marker with the image pickup device 2 may be used. (BX, BY, BZ) and BM can be obtained by imaging the target object 5 with the imaging device 2. Although the world coordinate system Σw and the base coordinate system Σb of the manipulator 202 are considered separately here, they may be considered as being matched.

Further, although the relationship of the coordinate system when grasping the target object 5 has been described here, the same consideration will be given to the arrangement. When the placement position / orientation is determined in S14, the coordinate system when the target object 5 is placed can be derived in the same manner by replacing the target object coordinate system Σo of the target object 5 with the placement destination. Therefore, the manipulator 202 and the gripping device 203 Can be controlled.

In S16, the manipulator 202 executes operations on the target object 5 such as the operation, gripping, transporting, and arranging of the robot 20 based on the control result of S15. When performing pick-and-place, the manipulator 202 first moves to the gripping position posture, and the gripping device 203 grips the target object 5. Next, the manipulator 202 for the operation of transporting and arranging the target object 5 is controlled. The waypoint of transportation may be determined in S14, may be set in advance, or may be determined by performing route planning based on the image information obtained by the image pickup apparatus 2. Then, based on the arrangement position / orientation determined in S14, the manipulator 202 moves to the arrangement position / orientation of the target object 5 and arranges the target object 5.

In S17, the CPU 21 determines whether or not there is the next target object 5. The CPU 21 determines that the process is terminated when the target object 5 does not exist, and terminates the process shown in FIG. If the target object 5 exists, it is determined that the processing will be continued, and the process returns to S10. The determination method is performed based on, for example, the presence / absence of the target object 5 to be operated based on the work group determination result and the presence / absence of the target object 5 in the supply area. The presence or absence of the target object 5 in the supply region is determined based on the image information obtained by photographing the supply location of the target object 5 with the imaging device 2. Alternatively, a sensor may be arranged in the supply area and the determination may be made from the sensor information. As for the sensor information, for example, a weight sensor may be arranged in the supply area, and the weight of the supply area may be measured to measure the remaining number of the target objects 5 by weighing. Alternatively, the determination may be made by a method other than these.

In the present embodiment, the information processing system 100 operates the target object 5 according to the instruction of the instructor. Therefore, an example has been described in which both the setting mode in which the information processing system 100 is on standby and the operation mode in which the information processing system 100 is operating exist in one flowchart. This assumes the use of a collaborative robot that does not require a safety fence, which has become widespread in recent years, as the robot 20, and intervenes in order for a person to enter the work range and instruct the operation while the robot 20 is in operation. This is because it is assumed. However, it is not always necessary to realize it with one flowchart, and the setting mode and the operation mode may be clearly separated and executed with different flowcharts.

<Modification example 1>
As a method of specifying the work group, there is also a method of specifying the work group from the estimated state of the target object. FIG. 4A is a diagram showing an example of image information of the target object and the transition of the object state in each state. Further, the state storage unit 105 stores image information and text information of each state of the target object 5 as shown in FIG. 4 (B). For example, as shown in FIG. 4A, text information corresponding to image information indicating a specific target object for each state is stored as state transition information. Specifically, the state A stores the image information of the lunch box and the text information of "lunch box". The state B stores the image information of white rice and the text information of "white rice". The state C stores the image information of pickled plums and the text information of "pickled plums". The state D stores the image information of the pickles and the text information of "pickles". The state E stores the image information of the white rice lunch box and the text information of "white rice lunch box". The state F stores the image information of Hinomaru bento and the text information of "Hinomaru bento". The state G stores the image information of the pickled lunch and the text information of "pickled lunch". However, a plurality of image information and text information may be stored in one state. In addition, the unit of the object that stores the state information may be a unique object unit such as "white rice" or "pickled plum", or a class such as "rice" that collectively includes "white rice", "red rice", and "vinegared rice". It may be a unit. Further, in FIG. 4A, as the initial states of the target object 5, states A to D are initial states, states E are intermediate states in the process of transitioning from the initial state to the target state, and states F and G are after the work is completed. It is a target state. Further, the state storage unit 105 stores information on whether or not each state transition is possible, and as shown in FIG. 4A, states A, B to state E, states C, E to state F, and states D, Each transition is possible from E to state G. FIG. 4C is a table conceptually explaining a work group in which the work required to achieve the intermediate state or the target state is associated with the target object in the initial state or the intermediate state.

The process of estimating the state of the target object by the estimation unit 104 will be described in detail with reference to FIG. The estimation unit 104 acquires state transition information from the state storage unit 105 (S191). Consider the case where the instructor gives an incomplete instruction such as "The next lunch box is pickled plums, so set it." At this time, the estimation unit 104 acquires the instruction recognized from the instruction input unit 101 (S192). The estimation unit 104 estimates the target object estimated from the voice related to the operation of the robot based on the state transition information regarding the target state using the target object (S193). In this example, the text information "Umeboshi" included in the instruction information is extracted, and it is estimated that "Umeboshi in the initial state" is supplied by comparing with the state transition information. Therefore, the estimation unit 104 identifies that the target object is "umeboshi" and that state is the "initial state" from the object state transitions based on the voice information.

The estimation unit 104 determines whether or not there is an achievable target state using the predetermined target object specified from the instruction information, based on the information (state transition information) regarding the candidate of the target state using the target object. Judgment (S194). FIG. 5 is a diagram showing candidates for a target state with a solid line and a target object in each unspecified state with a dotted line from a state in which the target object and its state are specified based on voice information (instruction information). .. As shown in FIG. 5, it can be estimated that a part of the target object (specifically, pickled plum) is in the initial state only by the voice information. However, it may not be possible to determine the initial state of another target object (presence or absence of the target object) or which intermediate state or target state the user intends. At this stage, umeboshi is estimated as the initial state from the voice information, but since there is no intermediate state or target state composed only of umeboshi, the intermediate state or target state cannot be determined. That is, in order to achieve a certain target state, it is recognized that there is a shortage of white rice lunch in the middle state, or a lunch box in the initial state and white rice in the initial state, and it is determined that the target state cannot be achieved. If there is a target state or an intermediate state that can be achieved only by pickling plums, the process proceeds to S197. Detailed processing will be described later.

When the estimation unit 104 determines that the target object of motion and the state of the object cannot be estimated only by the instruction information (No in S194), the estimation unit 104 estimates a candidate for an achievable target state using the first target object (No). S195). The method of object recognition is performed by semantic segmentation (semantic segmentation) using, for example, deep learning. The class classified by the semantic domain division corresponds to the object (or / or class) unit stored in the state storage unit 105. In the classification in the semantic domain division, the probability of being classified into each class for each domain is obtained, and the class having the highest probability is classified from among them. When calculating the probability, for example, the Softmax function is used to perform processing in which the probability of being classified into each class for each region is a positive value and the sum is 1. Here, the result that it cannot be classified into any class may be output by setting the threshold value without necessarily classifying into the class with the highest probability. For example, when the Softmax function is used, the probability of finding it falls within the range of 0 to 1, so the maximum value is 1. At this time, when the threshold value is set to 0.8, if there is a class obtained with the highest probability of 0.8 or more, it is classified into that class, but if it is less than 0.8, it is the maximum value. However, it is unlikely that it is in that class, and it cannot be classified into any class. If it can be classified into a class, the unit of the object (or / or class) stored in the state storage unit 105 corresponding to the class is selected, but if it cannot be classified into the class, the object (or / or class) Assuming that the unit cannot be selected, processing continues. FIG. 6 is an example of the acquired image information. For example, when FIG. 6 is an input image, it is determined that "lunch box", "pickled plum", and "white rice" are included in the image among the information stored in the state transition diagram. Based on the result of object recognition, it can be determined that the initial state includes "lunch box" and "polished rice" in the initial state as shown in FIG. 7 in addition to the "pickled plum" shown in FIG. , B, and C are determined as the target state. Further, the state E is estimated as a candidate for the intermediate state in order to reach the state F. In this way, the target state is estimated and sent to the work identification means 106. In FIG. 7, the states E and F shown by the broken lines are the target states of the candidates estimated based on the obtained audio information and image information.

As an example of the estimation method of the estimation unit 104, the case where the voice information alone is insufficient and the image information is used for complementation has been described. However, when the voice information is not insufficient, the target state is selected only by the voice information. You may. That is, when the intermediate state or the target state using the target object in the initial state can be specified as one (Yes in S194), the process proceeds to S195 and the target state is estimated. In S196, the work group identification unit 106 identifies a work group including the instructed work according to the situation in which the robot 20 is placed, based on the estimated target state and the instructed work. It is assumed that the target state and the work group are linked in advance in the state transition information. For example, in S195, when the intermediate state E "white rice lunch" is estimated, the transition from the state A "lunch box" and the state B "white rice" occurs. At this time, the relative positional relationship between the target objects 5, that is, the positional relationship of the state B "white rice" with respect to the state A "lunch box" is known, so that the "white rice" can be put into the "lunch box". Determine the behavior. Specifically, pick and place is performed from the work pattern stored in the state storage unit 105 by using the work pattern of gripping, transporting, and arranging. Here, with respect to the memory of the work pattern, a portion that changes as a parameter and a portion that does not change are determined in advance, and when the work is executed by the information processing system 100, the portion that changes as a parameter is determined based on the recognition result. For example, when gripping the target object 5, the gripping position posture changes as a parameter each time based on the recognition result, but the speed when approaching the gripping position is the work pattern, and the position passed before moving the gripping position is relative to the gripping position. The positional relationship is a fixed value. However, when the state storage unit 105 stores the transition information, the information at the time of such a transition may be changed based on the information. For example, when the state storage unit 105 also stores information on the transport speed of the state B "white rice" for the state transition when the state B "white rice" is put into the state A "lunch box". is there. In this case, the work group identification unit 106 may change the speed of the work pattern stored in the work storage unit 103 based on the information in the state storage unit 105. The work group identification unit 106 determines a work pattern for transitioning the target object 5 from one state to another in this way for each state transition. At that time, the order of the state transitions is also determined so that the state transitions are established. For example, as a target state, since the state F is transitioned from the state E, the work group is determined so that the operation for transitioning to the state F is performed after the operation for transitioning to the state E.

Here, when there is no shortage of voice information, the content of the work instruction by the instructor is "Lunch box, white rice, and pickled plums are supplied. First, put the white rice on the lunch box. Next, the lunch box. This is a case where all the states are accurately indicated, such as "Put the pickled plums on the white rice in the box. This completes the work." In this case, the initial states A, B, C, the intermediate state E, and the target state F as shown in FIG. 7 can be selected as the target state using only the voice information. However, it may be confirmed that there is no error in the selection by considering the image information with respect to the result selected by the audio information. Specifically, when the image information as shown in FIG. 6 is acquired, it can be seen that there are "white rice", "lunch box", and "pickled plum" by performing object recognition on the image. Therefore, it may be confirmed that there is no contradiction with the selection result based on the voice information, and the accuracy of the work executed by the information processing system 100 may be increased.

In addition, when changing only a part of the target object 5 due to a change of destination or a model change, the target state is determined by replacing the candidate of the target state estimated before the change with the changed object. May be good. For example, in the bento box to be manufactured, the pickled plums placed on the white rice are changed to pickles (the Hinomaru bento as shown in the state F in FIG. 4 is changed to the pickled lunch box shown in the state G). Because of this change, it is assumed that the instructor gave an instruction such as "change from pickled plums to pickles." In this case, in FIG. 7, the "pickled plum" state C, which was selected as the target state candidate, is replaced with the "pickled" state D, and the target state candidate is reselected. Therefore, referring to the state storage unit 105, the target state is selected as the "pickled lunch" in the state G as the target state obtained from the initial states A, B, and D, and the "white rice lunch" in the state E without changing the intermediate state. To do.

Further, even if the target object 5 itself is not stored in the state storage unit 105, the target state may be selected based on the class information indicating that the target object 5 is similar to the target object 5. For example, suppose you want to change the white rice contained in a lunch box to red rice. At this time, it is assumed that "white rice" is registered as an object in the state storage unit 105, but "red rice" is not registered. However, in the state storage unit, by storing not only the object unit of "white rice" but also the class unit of "rice", it corresponds to the unregistered "red rice". When memorizing in class units, for example, in the "rice" class, rice (food using rice) such as "white rice", "vinegared rice", and "wheat rice" is memorized. Therefore, when the information processing system 100 executes the work, even if "red rice" is not stored in the state storage unit 105, if it is classified into the "rice" class by the object recognition in the estimation unit 104, it is "red rice". May be treated in the same way as "white rice". At this time, in the object recognition, the recognition is performed not in the object unit (white rice, vinegared rice, barley rice, red rice, etc.) but in the class unit (rice).

Further, regarding the case where the voice information is insufficient, the case where the content of the statement by the instructor is insufficient has been described, but the case where the information processing system 100 cannot hear the content even if the instructor is speaking without lack. It may be judged that there is a lack of voice information. For example, when the voice of the instructor is low or affected by noise, it may be determined that the voice information is insufficient even if the instructor gives instructions without any shortage.

Further, when the instruction content from the instructor does not include the information in the initial state but only the information in the target state, the target state in the initial state may be selected by calculating back from the target state. For example, when the instructor gives an instruction such as "make a Hinomaru bento", the Hinomaru bento in the state F of FIG. 4 is determined as the target state, so that the states A and B are the states up to the state F. , C is the initial state, and the state E is the intermediate state, and is estimated as a candidate for the target state.

Further, in the estimation unit 104, the intermediate state from the initial state to the target state is not necessarily one, and there may be a plurality of states. When there are a plurality of work groups, the work group may be further specified by the instruction input by the user.

If there is no work group that can be achieved even if the work group identification unit 106 sufficiently collects auxiliary information about the target object, an error notification may be output. For example, even if it is known from the voice information that the pickled plums are to be operated, if the pickled plums are not in the operable area of the robot, an error notification is given. Further, the missing object may be estimated so that the robot recognizes whether the missing object is replenished or exists in the operable area. For example, the estimation unit 104 estimates the missing target object (second target object) from the difference between the target state candidate and the first target object. For example, when Hinomaru bento is a candidate for the target state, the lunch box A and the white rice B are specified as the second target objects. Next, based on the premise that the target state intended by the user is within the range that can be operated by the robot, it is determined whether or not the robot can obtain a second object that is missing from the target state. That is, the estimation unit 104 determines whether or not the second target object is available based on the predetermined information. If the second target object is available, it can be determined that the target state using the first target object and the available second target object is the target state intended by the user. If the second target object identified so far is not available, the robot cannot specify what to make or what to do, so it outputs an error notification and further informs the user. Request information, etc. Here, as auxiliary information, the image information of the target object 5 is further acquired from the visual information acquisition unit 102. Specifically, the recognition result is acquired by inputting the acquired image information into a trained model that outputs the state of the target object included in the image. If the recognition result includes the specified second target object, it is possible to create a target state using the target object included in the recognition result.

As described above, in the present embodiment, first, based on the voice information obtained by the instruction input unit 101 and the image information obtained by the visual information acquisition unit 102, the initial state and the target state from the working storage unit 103 and them are stored. Select a candidate in the middle of connecting. Next, an operation capable of transitioning between the selected target states is selected from the state storage unit 105, and a work group is determined. Then, based on the work group determination result and the image information, the position / attitude control information of the robot 20 for executing the work is determined, and the work is executed by the robot 20.

By doing so, when the user instructs the information processing system 100 to perform the work, the work intended by the instructor can be executed even if the information processing system 100 lacks information for executing the work. Therefore, even a user who is not accustomed to operating the robot can easily execute the work using the robot.

<Second embodiment>
Next, a second embodiment of the present invention will be described. In the second embodiment, if the instruction content is insufficient with respect to the first embodiment described above, is the information complemented when the information processing system 100 determines the work group appropriate by the output device 3? It is configured to ask the instructor for confirmation. Specifically, the work group is specified as in the first embodiment described above. Then, the output device 3 is configured so that the instructor can confirm the work group specified by the information processing system 100 by image or voice information. When the instructor confirms, it is possible to distinguish whether the decision is made based on the voice information, the image information, or the information processing system 100 by supplementing the information. By doing so, the instructor can confirm what kind of work group is specified based on what kind of information before the information processing system 100 starts the work. Therefore, when a work different from the intention of the instructor is determined, the work content can be corrected before the information processing system 100 starts the work, so that the work can be redone and the work efficiency can be improved.

FIG. 9 is a diagram showing a configuration example of an information processing system 100'provided with the information processing device 10'in the present embodiment. FIG. 9 is an example of a device configuration, and does not limit the scope of application of the present invention.

The output device 3 is composed of, for example, a display when the information processing device 10'visualizes the work group determination result and displays it as an image. The output device 3 may be a device other than the display as long as the visualized work group can be displayed as an image. For example, the output device 3 may be configured by a projector and displayed by projection. Further, it may be composed of a device such as an HMD (Head Mounted Display) or an AR (Augmented Reality) glass, and may be displayed by the instructor wearing the device. Further, the output device 3 is composed of, for example, a speaker when the information processing device 10'makes the specified work group audible and outputs it as voice. By using a directional speaker as the speaker, sound may be output only in a specific direction. Further, display / / and audio output may be performed by combining a plurality of these devices.

The work group identification unit 106'identifies from a plurality of patterns of work groups, those including the work indicated by the instruction information as candidates for the work group to be executed by the robot. This identifies the work group of the information processing system 100'. The work group identification unit 106'sends the specified work group to the generation unit 107 and the output unit 108.

The output unit 108 displays the work group specified by the work group identification unit 106'on the output device 3 or outputs the work group by voice so that the instructor can confirm it. When displaying, the work group is visualized by converting it into image information (this is called work group information). The visualized work group information is displayed on the output device 3. In addition, when outputting as voice, the work group is made audible by the work group information converted into natural language as voice. The audible work group information is output as voice by the output device 3.

FIG. 10 is a flowchart showing a work procedure executed by the information processing system 100'and the information processing device 10'in the present embodiment. The parts whose processing contents are different from those shown in FIG. 3 shown in the first embodiment will be described.

In S8', the work group identification unit 106'identifies a plurality of patterns of work groups including the work indicated by the instruction information as candidates for the work group to be executed by the robot. The work group identification unit 106'sends the specified work group to the generation unit 107 and the output unit 108.

In S18, the output unit 108 outputs the work group information for visualizing and / and audible the work group received from the work group identification unit 106'. FIG. 11 is an example of a work group visualized by the output unit 108. The output unit 108 displays the initial state, the intermediate state, and the target state of the work group determined by the work group identification unit 106'so that the instructor can confirm each. At the time of display, it is possible to distinguish whether each state is determined by the voice information instructed by the instructor, the image information, or the information processing system 100'by supplementing the information.

For example, in FIG. 11, the initial state of "umeboshi" was determined based on the instruction from the instructor. On the other hand, the "lunch box" and "white rice" are determined by the image information, and the information information system 100'is based on the initial state of the "white rice lunch" which is in the middle state and the "Hinomaru bento" which is the target state. It is a state determined by complementing. At this time, "Umeboshi" is displayed as "sound" so that it can be understood that it is determined by voice information, and "lunch box" and "white rice" are displayed as "picture" so that it can be understood that it is determined by image information. .. Moreover, since these initial states are determined based on the actual information of the audio information or the image information, they are displayed in a solid line frame. On the other hand, "white rice bento" and "Hinomaru bento" are determined based on the actual information, such as displaying in the frame of the broken line because the information processing system 100'determined by complementing the information, and complementing the information. The states determined in are displayed separately. When the information processing system 100'displays the complemented information, only the state determined by the system to have the highest probability may be displayed with respect to the instruction content of the instructor, or a plurality of states may be displayed together with the probability determined by the system. The status may be displayed. For the probability, for example, when determining using image information by deep learning, the value of the likelihood of the result of object recognition is used. However, the probability may be obtained by a method other than this, or may be displayed by a different method.

Further, the state displayed in FIG. 11 is any of the initial state, the intermediate state, and the target state associated with the work group specified by the work group identification unit 106', but other states are determined. You may display the status that is not displayed. For example, the work group identification unit 106'may display the work group specified as a candidate for the work group, or any work group stored in the work storage unit 103 may be displayed. Whether or not to display may be changed based on the content of the voice instruction from the instructor, the display range may be set in advance, or may be set by another method.

Further, when the work group cannot be specified in the work group specifying unit 106', it may be displayed that the work group cannot be specified. In that case, by displaying up to the specified part, it is visualized to what extent the information processing system 100'can determine the work group intended by the instructor.

Moreover, you may display the transition method of each state. For example, a transition method such as grasping the "umeboshi", transporting it to the top of the "polished rice lunch box", and arranging it, or the operation of the robot 20 at that time may be displayed.

Further, when the output unit 108 makes the output unit 108 audible so that it can be confirmed by voice, the determination result of the work group identification unit 106'is converted into natural language. When converting to natural language, for example, based on the text information stored in the state storage unit 105, information about the state is obtained, visualized and displayed, and the state is based on what kind of information. Is output as a voice.

In S19, the output unit 108 displays the visualized / and audible work group on the output device 3 and outputs the voice so that the instructor can confirm the work group. In the case of visualization, the display method may be a moving image or text information, and if the display device is a device capable of AR display such as an HMD or AR glass, the display may be superimposed on the actual object. You may. The transition method may be displayed for each state transition, a series of work group flows may be displayed, or only a part of the transitions may be displayed. When making it audible, the work group converted into natural language is output as voice by the speaker.

Further, the working state may be confirmed only before the work, or may be confirmed during the work. If it is possible to check during work, the contents may be updated based on the change in the state transition. For example, even if the information processing system 100'can be checked in which state in the state transition diagram. Good. Further, the setting for confirming the working state may be changed by instructing by voice, may be set in advance, or may be set by another method.

In S20, the information processing system 100'performs voice input from the instructor. Here, the instructor gives an instruction to continue the work if there is no problem based on the confirmation result. If there is a problem, give instructions to correct the work group. When the correction instruction is given, the information processing system 100'updates the determined work group, so that the target state is selected again and the work group is determined again. At that time, new visual information is acquired.

As described above, in the present embodiment, the work group to be executed by the robot is specified by the voice information and the image information. Then, the specified work group can be confirmed by the instructor by one or more methods of visualization and audibility. By doing so, the instructor can confirm what kind of work is to be executed before the information processing system 100'starts the work, and if a work different from the instructor's intention is decided, before the work starts. Since the work contents can be corrected, the work redoing can be reduced and the work efficiency can be improved.

<Third embodiment>
Next, the third embodiment will be described. In the third embodiment, in contrast to the first and second embodiments described above, when handling an object that is not stored in the working storage unit 103, image information of the intermediate state and the target state after the state transition is provided. It is configured to continue the work by generating it and confirming the status with the instructor. Specifically, the work group to be executed by the robot is specified as in the first and second embodiments described above. Then, as in the second embodiment described above, the identified work group is confirmed with the instructor. At this time, if it is determined that the target object 5 is an object that is not stored in the state storage unit 105, confirmation is performed by generating images of the intermediate state and the target state. Then, as a result of confirmation by the instructor, if it is determined that the generated image is correct, the image is newly registered in the state storage unit 105. In this way, even when a new object is targeted, the work can be continued while confirming the intention of the instructor, so that the system downtime can be shortened and the productivity can be improved.

FIG. 12 is a diagram showing a configuration example of an information processing system 100 ″ including the information processing device 10 ″ in the present embodiment. FIG. 12 is an example of a device configuration, and does not limit the scope of application of the present invention.

The instruction input unit 101 ″ converts the voice information acquired by the voice input device 1 into text information, and sends information regarding the state and operation of the target object 5 to the estimation unit 104. When a state is newly registered in the state storage unit 105 ″, information regarding the state and operation of the target object 5 is also sent to the state storage unit 105 ″.

The visual information acquisition unit 102 ″ acquires the visual information output from the image pickup apparatus 1. The visual information acquisition unit 102 ″ converts the acquired visual information into image information and outputs it to the estimation unit 104. When a state is newly registered in the state storage unit 105 ″, image information is also sent to the state storage unit 105 ″.

The state storage unit 105 ″ stores the state transition information of the target object 5. When registering new state transition information, text information regarding the state and operation of the target object 5 is received from the instruction input unit 101 ″ and image information of the target object 5 is received from the visual information acquisition unit 102 ″ and registered.

The work group identification unit 106 ″ identifies, from a plurality of patterns of work groups, those including the work indicated by the instruction information as candidates for the work group to be executed by the robot. The work group specifying unit 106 ″ sends the specified work group to the generation unit 107, the output unit 108 ″, and the image generation means 109.

When the work group specified by the work group identification unit 106 ″ is displayed on the output device 3, the output unit 108 ″ is visualized by converting the work group into an image. When an image is generated by the image generating means 109, the image information is received from the image generating means 109 and displayed.

When the image generation means 109 determines that the intermediate state and the target state are new states based on the work group specified by the work group specifying unit 106 ″, the image generating means 109 generates an image of the intermediate state and the target state. The generated image information is sent to the output unit 108 ″.

FIG. 13 is a flowchart showing a work procedure executed by the information processing system 100 ″ and the information processing apparatus 10 ″ in the present embodiment. The parts whose processing contents are different from those of FIGS. 3 and 10 shown in the first and second embodiments will be described respectively.

In S8 ″, the work group identification unit 106 ″ specifies from a plurality of patterns of work groups, those including the work indicated by the instruction information as candidates for the work group to be executed by the robot. The work group specifying unit 106 ″ sends the specified work group to the generation unit 107, the output unit 108 ″, and the image generation means 109.

In S21, it is determined whether or not there is an object that is not stored in the state storage unit 105 ″. The determination method is, for example, based on the content of the voice instruction from the instructor, when an object not registered in the state storage unit 105 ″ is instructed, it is determined that the object is a new object. Alternatively, if there is a contradiction between the voice information of the instruction content and the acquired image information when determining the work group, the output device 3 confirms with the instructor to determine whether the object is a new object. Good. When it is determined that the object is a new object, an image of the intermediate state and the target state is generated in S22. If it is determined that it is not a new object, the work group is visualized in S18 ″.

In S22, images of the intermediate state and the target state are generated. The generation method is performed, for example, by synthesizing an image of a new object with the object stored in the state storage unit 105 ″. The region of the new object may be obtained by, for example, background subtraction, instance segmentation using deep learning, or other methods. By superimposing the obtained object region on the image of a known object, an image of an intermediate state and a target state including a new object is generated. Alternatively, deep learning may be used to directly generate images of the intermediate state and the target state. For example, using deep learning, multiple images in the initial state are input, images in the intermediate state and target state corresponding to the input are output, and learning is performed by GAN (Generative Advanced Network) so that images can be generated. May be good.

In S18 ″, the output unit 108 ″ visualizes the work group received from the work group identification unit 106 ″.

In S20 ″, the information processing system 100 ″ performs voice input from the instructor. Here, the instructor determines whether or not the generated image is appropriate as an intermediate state or a target state, and inputs the determination result to the system by voice.

In S23, in response to the input result of S20 ″, the subsequent processing is changed depending on whether or not the generated image is appropriate as an intermediate state or a target state. If it is determined to be appropriate, the process is performed in S25, and if it is determined to be inappropriate, the process is performed in S24.

In S24, the system receives an instruction for new registration of the intermediate state and the target state from the instructor. Since the image generation means 109 could not generate an appropriate image, the instructor is asked to actually create an intermediate state and a target state, and each state is registered in the state storage unit 105 ″.

In S25, the image information generated in S22 or the actual object state created by the instructor in S24 is registered in the state storage unit 105 ″. The information to be registered is a combination of image information and text information. Here, as the image information, if it is determined in S23 that the image information generated in S22 is appropriate, the generated image information is registered. On the contrary, when it is determined that it is not appropriate, the image information is obtained and registered by imaging the state created by using the real object requested by the instructor in S24. The text information is obtained by converting the voice information instructed by the instructor into text information together with the registration of the image, and is registered. At the time of registration, it may be selected whether to register in units of objects or in units of classes. When registering in class units, the target class may be selected from the memorized classes, or a new class may be created. The selection method may be determined by displaying the registration destination on the output device 3 by classifying the class based on the image information to be registered and confirming with the instructor. It may be decided by displaying the candidate options on the output device 3 and letting the instructor select an appropriate class from them.

In addition, information on whether or not transition is possible is also registered. At that time, the transition method may also be registered. For example, when registering an object state, the registration method is determined by displaying the same transition method as other objects in the class on the output device 3 and confirming with the instructor when the object state is registered in class units. May be good. The output device 3 may display a candidate transition method, and the instructor may select an appropriate method from the transition methods.

In S26, the state after the transition is completed is newly registered in the state storage unit 105 ″. In S25, since the generated image or the image created by the instructor is registered, the result of the actual work of the information processing system 100 ″ is not registered. In S26, the data is increased by registering the image information of the result of the actual work of the information processing system 100 ″. Since the process of increasing the image information is performed here, the information registered in S25 is applied to the text information, the transition possibility information, the transition method, and the like.

As described above, in the present embodiment, the target state is selected by the audio information and the image information, and the work group is determined. When determining the work group, if it is determined that the target object 5 is an object that is not stored in the state storage unit 105'', the work group is confirmed to the instructor by generating images of the intermediate state and the target state. I do. Then, as a result of confirmation by the instructor, if it is determined that the generated image is correct, the image is newly registered in the state storage unit 105 ″. By doing so, even when a new object is targeted, the work can be continued while confirming the intention of the instructor, so that the system downtime can be shortened and the productivity can be improved.

[Hardware configuration]
The information processing device 10 is composed of, for example, a personal computer (PC). FIG. 15 is an example of the hardware configuration of the information processing device 10. The information processing device 10 includes a CPU 11, a ROM 12, a RAM 13, an external memory 14, an input unit 15, a display unit 16, a communication I / F 17, and a system bus 18. The CPU 11 comprehensively controls the operation of the information processing apparatus 10, and controls each component (11 to 17) via the system bus 18. The ROM 12 is a non-volatile memory for storing a program required for the CPU 11 to execute a process. The program may be stored in an external memory 14 or a removable storage medium (not shown). The RAM 13 functions as a main memory and a work area of the CPU 11. That is, the CPU 11 loads a program required from the ROM 12 into the RAM 13 when executing the process, and executes the program to realize various functional operations.

The external memory 14 stores, for example, various data and various information necessary for the CPU 11 to perform processing using a program. Further, in the external memory 14, for example, various data and various information obtained by the CPU 11 performing processing using a program are stored. The input unit 15 is composed of, for example, a keyboard or mouse pointing device, and an operator can give an instruction to the information processing device 10 via the input unit 15. The display unit 16 is composed of a monitor such as a liquid crystal display (LCD). The communication I / F 17 is an interface for communicating with an external device. The system bus 18 communicatively connects the CPU 11, ROM 12, RAM 13, external memory 14, input unit 15, display unit 16, and communication I / F17. In this way, the information processing device 10 can communicate with the external devices such as the voice input device 1, the image pickup device 2, the output device 3, the voice output device 4, the light source 6, and the robot 20 via the communication I / F17. It is connected and controls the operation of these external devices.

<Effect of Examples>
According to the first embodiment, when the user instructs the information processing system 100 to perform the work, the work intended by the instructor can be executed even if the information processing system 100 lacks information for executing the work. .. Therefore, even a user who is not accustomed to operating the robot can easily execute the work using the robot.

According to the second embodiment, when the instructor can confirm what kind of work is to be executed before the information processing system 100'starts the work, and the work different from the instructor's intention is decided. Can modify the work contents before the work starts. Therefore, the reworking of the work is reduced, and the work efficiency can be improved.

According to the third embodiment, even when a new object is targeted, the work can be continued while confirming the intention of the instructor, so that the system downtime can be shortened and the productivity can be improved.

<Other Embodiments>
In the second and third embodiments, the voice input device 1, the image pickup device 2, and the output device 3 are used not only for confirming the determination result of the work group but also as an interface with the system used by another instructor. You may. For example, when the instructor A calls to the instructor B who uses another system in another line through the voice input device 1, the instructor B in another line is instructed through the output device 3 of the system. You can confirm A's call. Alternatively, even for the user C who is using the personal computer, the information input by the instructor A to the information processing system 100'can be confirmed by the user C using the display or the speaker connected to the personal computer. Good. On the contrary, information may be sent from the instructor B and the user C to the instructor A.

In the second embodiment, the information that can be confirmed by the output unit 108 may be the target state selection result instead of the work group confirmation result. However, when the target state selection result can be confirmed by the output device 3, the intermediate state may not be determined or the transition information including the operation may not be known, which is indicated.

In the third embodiment, when an appropriate work pattern is not stored in the state storage unit 105 for the newly registered target object 5, a new work is performed by the position / attitude control information acquisition means 110 and the motion generation means 111. Register the pattern. FIG. 14 is a configuration example in which the position / attitude control information acquisition means 110 and the motion generation means 111 are added to the device configuration according to the third embodiment. The position / attitude control information acquisition unit 110 receives the position / attitude control information from the controller 201 ″. The motion generating means 111 generates motions based on the control information acquired by the position / attitude control information acquisition unit. The information acquired by the position / attitude control information acquisition means 110 is the position / attitude control information, but the motion generated by the motion generation means 111 is a certain work pattern in which these control informations are continuously connected. As for the method of generating the motion, the instructor may teach the robot 20'''the motion by direct teaching, the motion may be generated by using the teaching pendant, or the motion may be generated by writing a program. May be generated, or any other method may be used.

Of the above-mentioned processing units, the estimation unit 104 and the like may be processed by using a machine-learned trained model instead. In that case, for example, a plurality of combinations of input data and output data to the processing unit are prepared as training data, knowledge is acquired from them by machine learning, and output data for the input data is obtained based on the acquired knowledge. Generates a trained model that outputs as a result. The trained model can be constructed by, for example, a neural network model. Then, the trained model performs the processing of the processing unit by operating in collaboration with the CPU, GPU, or the like as a program for performing the same processing as the processing unit. The trained model may be updated after a certain process if necessary.

The present invention is also realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiment is supplied to the system or device via a network for data communication or various storage media. Then, the computer (or CPU, MPU, etc.) of the system or device reads and executes the program. Further, the program may be recorded and provided on a computer-readable recording medium.

1 Voice input device 2 Imaging device 3 Output device 5 Target object 6 Light source 10 Information processing device 20 Robot 100 Information processing system 201 Controller 202 Manipulator 203 Gripping device

Claims (15)

An information processing device that causes a robot to execute a work group consisting of multiple tasks.
An input means for inputting instruction information indicating at least a part of the work in the work group, and
An information processing apparatus comprising: a specific means for specifying a work group including a work indicated by the instruction information as a candidate for a work group to be executed by the robot among the work groups having a plurality of patterns. The information processing apparatus according to claim 1, wherein the specific means specifies a work group to be executed by the robot based on supplementary information when there are a plurality of patterns of the candidate work groups. .. The information processing apparatus according to claim 2, wherein the supplementary information is a selection instruction input by the user. The information processing apparatus according to claim 2, wherein the supplementary information is material information related to a material used in any of the plurality of patterns of work groups. The information processing device according to claim 4, wherein the material information is information for identifying a material existing in the work space of the robot. The information processing apparatus according to claim 4 or 5, wherein the material information is image information obtained by photographing a material existing in the work space of the robot. The information processing device according to any one of claims 1 to 6, wherein the input means inputs the instruction information which is voice information emitted by a user. The information processing apparatus according to any one of claims 1 to 7, further comprising a storage means for storing the plurality of patterns of work groups. The information processing apparatus according to any one of claims 1 to 8, wherein the specific means determines the order of work to be performed based on the specified work group. The information processing apparatus according to any one of claims 1 to 9, further comprising a generation means for generating control information of the robot based on the work group specified by the specific means. The information processing apparatus according to any one of claims 1 to 10, further comprising a presentation means for presenting the work determined by the determination means and the order of the work to the user. The information processing apparatus according to claim 9, wherein the presentation means presents the work by generating an image showing the work and the order of the work. The information processing apparatus according to any one of claims 1 to 12, further comprising a learning means for learning control information of the work when a work not included in the work group is instructed. A program for causing a computer to function as each means included in the information processing apparatus according to any one of claims 1 to 13. An information processing method that causes a robot to execute a work group consisting of multiple tasks.
An input process for inputting instruction information indicating at least a part of the work in the work group, and
An information processing method comprising:, among the plurality of patterns of the work group, a specific step of specifying a work group including the work indicated by the instruction information as a candidate of the work group to be executed by the robot.

Images