JP7471999B2 - Information processing device - Google Patents

Description

The present invention relates to an information processing device.

Patent document 1 discloses an information processing device that recognizes a shape drawn by a user and executes a command corresponding to the recognized shape.

JP 2007-42050 A

By the way, a user may wish to apply a physical action, such as irradiating light or projecting an image, to a desired position in a space, such as a room, to make the space more comfortable or to decorate the space. However, with conventional technology, it was not possible to apply a physical action to a user's desired position.

In order to solve the above problems, an information processing device according to a preferred embodiment of the present disclosure is an information processing device capable of controlling an action device that applies a physical action to an action surface, and includes: a storage device that stores identification information indicating a type of processing of the action device and identification graphic information indicating a graphic corresponding to the type of processing in association with each other; a detection unit that detects position information indicating a position of an indicator with respect to the action surface; a determination unit that determines, based on the position information, whether a graphic included in a trajectory image indicating a trajectory of the position of the indicator is similar to a graphic indicated by the identification graphic information; a process determination unit that determines, when the determination result of the determination unit is positive, a process indicated by the identification information associated with the identification graphic information as a specific process to be executed by the action device; a position determination unit that determines, based on the position information, an action position on the action surface that receives the action generated by the specific process; and an execution unit that causes the action device to execute the specific process that applies the action to the action position.

The present invention allows a physical effect to be exerted on a user's desired location.

1 is a schematic diagram illustrating an action providing system 1 according to a first embodiment. FIG. 1 is an explanatory diagram showing the configuration of an action providing system 1. FIG. 13 is a diagram showing an example of the contents of first graphic information Z1I. FIG. 13 is a diagram showing an example of a irradiation processing figure SPL. 13 is a diagram showing an example of a trajectory image TP projected onto an action surface SC. FIG. 2 is a flowchart showing the operation of the action providing system 1. FIG. 11 is a schematic diagram illustrating an action providing system 1a according to a second embodiment. FIG. 1 is an explanatory diagram showing the configuration of an action providing system 1a. FIG. 2 is a diagram showing an example of a landscape image PPLA. FIG. 13 is a diagram showing an example of an application generated image PPAL. FIG. 13 is a diagram showing an example of the contents of first graphic information Z1Ia. FIG. 2 is a diagram showing an example of a landscape display figure SPLA. FIG. 13 is a diagram showing an example of an application display graphic SPAL. 13 is a diagram showing a projection example of a projection image PRJ including a landscape image PPLA. 13 is a diagram showing a projection example of a projection image PRJ including an application-generated image PPAL. FIG. 2 is a flowchart showing the operation of the action providing system 1a. FIG. 2 is a flowchart showing the operation of the action providing system 1a. FIG. 13 is an explanatory diagram showing the configuration of an action providing system 1b in a first modified example. FIG. 13 is a diagram showing an example of the contents of first graphic information Z1Ib. FIG. 13 is a diagram showing an example of a blowing processing figure SPW. FIG. 13 is an explanatory diagram showing the configuration of an action providing system 1c in a second modified example. FIG. 13 is a diagram showing an example of the contents of first graphic information Z1Ic. FIG. 13 is a diagram showing an example of a localization processing figure SPS. FIG. 13 is an explanatory diagram showing the configuration of an action providing system 1d in a third modified example. FIG. 13 is an explanatory diagram showing the configuration of an action providing system 1e in a fourth modified example. A figure showing an example of the contents of second graphic information Z2I. FIG. 4 is a diagram showing an example of a first size figure SP1. FIG. 13 is a diagram showing an example of a second size figure SP2. FIG. 4 is a diagram showing an example of a trajectory image TP. FIG. 13 is an explanatory diagram showing the configuration of an action providing system 1f in a fifth modified example. FIG. 4 is a diagram showing an example of a list image DP indicated by list image information DPI. FIG. 13 is a diagram showing an example of a trajectory image TP when the determination result of the determination unit 24 is negative. 13 is a diagram showing an example of the contents of first graphic information Z1I after addition by an adding unit 31. FIG. FIG. 13 is an explanatory diagram showing the configuration of an action providing system 1g in a sixth modified example. FIG. 13 is a diagram showing an example of a memo pad display processing figure SPMM. FIG. 4 is a diagram showing an example of a trajectory image TP. 13 is a diagram showing an example of a memo pad image PPMM projected onto the action surface SC.

1. First embodiment FIG. 1 is a schematic diagram illustrating an action providing system 1 according to a first embodiment. The action providing system 1 provides a physical action to a position desired by a user. The action providing system 1 includes an information processing device 10, a pointer 15, a projection device 40, and an action device. The action providing system 1 is provided, for example, in a guest room RM in a hotel. The action device provides a physical action directly or indirectly to an inner wall surface in the guest room RM. Hereinafter, the inner wall surface in the guest room RM is referred to as an action surface SC. The physical action provided to the action surface SC means that at least one of visible light, sound, heat, vibration, and wind is provided to the action surface SC. In addition, directly providing a physical action means that the action device itself acts on the action surface SC. Examples of action devices that directly provide a physical action include a lighting device, a projection device, an audio device, a heating and cooling device, a vibration device, and a blower device. The term "indirectly exerting a physical action" means that the action device itself does not act on the action surface SC, but exerts a physical action on the action surface SC via some device. Examples of action devices that indirectly exert a physical action include a device that specifies the brightness of the light emitted by the lighting device according to the brightness of the environment, a generating device that generates image information showing an image projected by the projection device, a device that generates sound emitted by the acoustic device, a device that specifies the target temperature of the air conditioner according to the temperature of the environment, a device that specifies the vibration magnitude of the vibration device, and a device that specifies the air volume of the air blower. In the first embodiment, an example will be described in which the action device is a lighting device 50 that emits illumination light onto the action surface SC.

The information processing device 10 controls the action providing system 1. The information processing device 10 is any information processing device (for example, a stationary information device such as a server or a personal computer, or a portable information terminal such as a smartphone, a notebook computer, a wearable terminal, or a tablet terminal). The projection device 40 projects an image onto the action surface SC under the control of the information processing device 10. The lighting device 50 irradiates illumination light onto the action surface SC under the control of the information processing device 10. The indicator 15 is a pen- or rod-shaped operating device. The indicator 15 is operated by a user and is used to point to any position on the action surface SC.

1.1. Configuration of the action provision system 1 FIG. 2 is an explanatory diagram showing the configuration of the action provision system 1. The action provision system 1 has an infrared light imaging device 45 in addition to the information processing device 10, the projection device 40, and the lighting device 50 shown in FIG. 1. The infrared light imaging device 45 images the action surface SC in a state where infrared light is transmitted and visible light is blocked. The infrared light imaging device 45 transmits the imaging information obtained by imaging to the information processing device 10. However, the action provision system 1 detects the position of the indicator 15 using the infrared light imaging device 45, but is not limited to this. The action provision system 1 may not have anything other than the infrared light imaging device 45 and may detect the position of the indicator 15 using other hardware. For example, the action provision system 1 may have an electrically conductive transparent film stretched on the action surface SC. The action provision system 1 detects the position of the indicator 15 based on an electrical change that occurs when the transparent film is pressed by the indicator 15.

The information processing device 10 includes a processing device 20, a communication device 32, and a storage device 35. The elements of the information processing device 10 are connected to each other by one or more buses 38 for communicating information. Note that the term "apparatus" in this specification may be interpreted as other terms such as circuit, device, or unit. Furthermore, each element of the information processing device 10 is composed of one or more devices, and some elements of the information processing device 10 may be omitted.

The lighting device 50 has a light source 52, a driving mechanism 54, and a communication device 56. The elements of the lighting device 50 are connected to each other by a single or multiple buses 58 for communicating information. The light source 52 emits illumination light under the control of the information processing device 10. The light source 52 is formed of, for example, a xenon lamp, an ultra-high pressure mercury lamp, an LED (Light Emitting Diode), or a laser light source. The driving mechanism 54 controls the direction of the illumination light emitted from the light source 52 under the control of the information processing device 10. The communication device 56 is hardware (transmitting/receiving device) for communicating with other devices. The communication device 56 is also called, for example, a network device, a network controller, a network card, a communication module, etc.

The processing device 20 is a processor that controls the entire information processing device 10, and is composed of, for example, a single chip or multiple chips. The processing device 20 is composed of, for example, a central processing unit (CPU) that includes an interface with peripheral devices, an arithmetic unit, and a register. Some or all of the functions of the processing device 20 may be realized by hardware such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The processing device 20 executes various processes in parallel or sequentially.

The communication device 32 is hardware (transmitting/receiving device) for communicating with other devices. The communication device 32 is also called, for example, a network device, a network controller, a network card, a communication module, etc.

The storage device 35 is a recording medium readable by the processing device 20, and stores a plurality of programs including the control program PR executed by the processing device 20, the first graphic information Z1I, and various information used by the processing device 20. The storage device 35 may be composed of at least one of, for example, a ROM, an EPROM (Erasable Programmable ROM), an EEPROM (Electrically Erasable Programmable ROM), a RAM (Random Access Memory), etc. The storage device 35 may also be called a register, a cache, a main memory (primary storage device), etc.

Figure 3 is a diagram showing an example of the contents of the first graphic information Z1I. The first graphic information Z1I is information that associates the identification information ZAI with the identification graphic information ZPI. The identification information ZAI indicates the type of processing of the lighting device 50. The identification information ZAI is, for example, an identifier or a character string that indicates the type of processing of the lighting device 50. The type of processing of the lighting device 50 is, for example, an irradiation processing that irradiates illumination light and a turn-off processing that stops irradiating illumination light. In the first embodiment, the identification information ZAI registered in the first graphic information Z1I is the character string "irradiation processing" that indicates the irradiation processing.

The identification graphic information ZPI indicates a graphic corresponding to the processing of the action device. In the first embodiment, the identification graphic information ZPI is irradiation processing graphic information SPLI corresponding to the irradiation processing. Hereinafter, the graphic indicated by the irradiation processing graphic information SPLI is referred to as the "irradiation processing graphic SPL."

Figure 4 is a diagram showing an example of an irradiation processing figure SPL. In order for a user to memorize the irradiation processing figure SPL, it is preferable that the irradiation processing figure SPL is a figure that can be associated with an irradiation process. The irradiation processing figure SPL is, for example, a figure that simplifies the shape of a general lighting fixture. Alternatively, the irradiation processing figure SPL may be a figure that is a schematic representation of the shape of an object that emits light, such as the sun or a light bulb. In the example of Figure 4, the irradiation processing figure SPL shows a figure that simplifies the shape of a stand lighting fixture.

Returning to FIG. 2 for explanation, the processing device 20 reads the control program PR and executes the control program PR to function as a detection unit 21, a generation unit 22, a first projection control unit 23, a determination unit 24, a processing determination unit 25, a position determination unit 26, and an execution unit 27.

The detection unit 21 detects position information PI indicating the position of the indicator 15 relative to the action surface SC. Specifically, the indicator 15 has a light source that emits light in the infrared light region. The detection unit 21 detects position information PI within the action surface SC pointed to by the indicator 15 based on the imaging information output from the infrared light imaging device 45. The position information PI includes, for example, a series of coordinate information of the indicator 15 during the period when the indicator 15 is located near the action surface SC. This coordinate information indicates the coordinates of the indicator 15 when the action surface SC is a two-dimensional coordinate system.

The generation unit 22 generates trajectory information TI indicating a trajectory image TP based on the position information PI. The trajectory image TP indicates the trajectory of the position of the indicator 15.

The first projection control unit 23 causes the projection device 40 to project the trajectory image TP based on the trajectory information TI. FIG. 5 shows an example of the trajectory image TP projected onto the action surface SC.

Based on the trajectory information TI, the determination unit 24 determines whether the figure included in the trajectory image TP is similar to the irradiation processing figure SPL indicated by the identification figure information ZPI. For example, the determination unit 24 divides the trajectory image TP into one or more figures with continuous trajectories. Hereinafter, the divided figures are referred to as "divided figures". The determination unit 24 calculates the similarity between one divided figure and the irradiation processing figure SPL for each of the one or more divided figures. The similarity is a degree indicating the degree of similarity. The similarity between two figures can be calculated by applying various known techniques. For example, the determination unit 24 calculates the sum of the differences between the pixel values of multiple pixels included in the peripheral image of the feature point of one divided figure and the pixel values of multiple pixels included in the peripheral image of the feature point of the irradiation processing figure SPL as the similarity. The feature points are, for example, corner points and intersections. It can be said that the smaller the similarity, the more similar one divided figure is to the irradiation processing figure SPL. The method for calculating the similarity between the two figures is not limited to the above-mentioned method. For example, the determination unit 24 may calculate the similarity using machine learning such as deep learning. If the similarity is less than a predetermined threshold, the determination unit 24 outputs a positive determination result. If the similarity is equal to or greater than the predetermined threshold, the determination unit 24 calculates the similarity between another divided figure and the irradiation processing figure SPL and determines whether this similarity is less than a predetermined threshold. If the similarity for each of one or more divided figures is equal to or greater than a predetermined threshold, the determination unit 24 outputs a negative determination result. For the sake of simplicity, the following description will be given on the assumption that the trajectory image TP contains only one figure unless otherwise specified.

When the determination result of the determination unit 24 is positive, the process determination unit 25 determines the irradiation process as the specific process to be executed by the lighting device 50. In the first embodiment, the irradiation process is the process indicated by the identification information ZAI associated with the irradiation process figure SPL.

The position determination unit 26 determines an action position on the action surface SC that receives the physical action caused by the specific process based on the trajectory information TI. For example, the position determination unit 26 determines the center of gravity of the trajectory information TI as the action position. The center of gravity is the point at which the sum of the first moments of area in the target shape becomes zero.

The execution unit 27 causes the lighting device 50 to execute a specific process that applies an action to the action position. Specifically, first, the execution unit 27 issues an instruction to the drive mechanism 54 to set the direction of the illumination light emitted from the light source 52 to a direction that faces the action position. Second, the execution unit 27 issues an instruction to the light source 52 to emit illumination light. Based on these two instructions, the lighting device 50 executes a process to emit illumination light to the action position.

1.2 Operation of the action providing system 1 The operation of the action providing system 1 will be described with reference to FIG.

Figure 6 is a flowchart showing the operation of the action providing system 1. The processing device 20 of the information processing device 10 detects the position information PI based on the imaging information received from the indicator 15 (step S2). The process of step S2 corresponds to the detection unit 21. After the process of step S2 is completed, the processing device 20 generates trajectory information TI indicating the trajectory image TP based on the position information PI (step S4). The process of step S4 corresponds to the generation unit 22. After the process of step S4 is completed, the processing device 20 transmits a projection instruction to the projection device 40 to cause the projection device 40 to project the trajectory image TP (step S6). The process of step S6 corresponds to the first projection control unit 23. The projection instruction includes the trajectory information TI. When the projection instruction is received, the projection device 40 projects the trajectory image TP onto the action surface SC (step S8).

After the process of step S6 is completed, the processing device 20 judges whether the trajectory image TP is similar to the irradiation process figure SPL indicating the irradiation process (step S12). The process of step S12 corresponds to the judgment unit 24. If the judgment result of step S12 is positive, the processing device 20 determines the irradiation process as the specific process (step S14). The process of step S14 corresponds to the process decision unit 25. After the process of step S14 is completed, the processing device 20 determines the action position based on the trajectory information TI (step S16). The process of step S16 corresponds to the position decision unit 26. After the process of step S16 is completed, the processing device 20 transmits an irradiation instruction to the lighting device 50 to perform the irradiation process at the action position (step S18). The process of step S18 corresponds to the execution unit 27. When the irradiation instruction is received, the lighting device 50 irradiates the action position with illumination light (step S20). After the process of step S20 is completed, the action providing system 1 ends the series of processes shown in FIG. 6. If the determination result of step S12 is negative, the action providing system 1 also ends the series of processes shown in FIG. 6. If the processing of step S18 is completed or if the determination result of step S12 is negative, the action providing system 1 continues to project the trajectory image TP onto the action surface SC. Alternatively, if the processing of step S18 is completed or if the determination result of step S12 is negative, the processing device 20 may send an instruction to the projection device 40 to stop projecting the trajectory image TP onto the action surface SC.

1.3. Summary of the First Embodiment As described above, the information processing device 10 in the first embodiment is an information processing device 10 capable of controlling the illumination device 50 (an example of an "action device that exerts a physical action on an action surface") that irradiates illumination light onto the action surface SC. The information processing device 10 has a processing device 20 and a storage device 35. The storage device 35 stores identification information ZAI indicating the processing of the illumination device 50 and identification figure information ZPI indicating a figure corresponding to the processing in association with each other. The processing device 20 functions as a detection unit 21, a determination unit 24, a processing decision unit 25, a position decision unit 26, and an execution unit 27. The detection unit 21 detects position information PI indicating the position of the indicator 15 with respect to the action surface SC. The determination unit 24 determines whether or not a figure included in a trajectory image TP indicating the trajectory of the position of the indicator 15 is similar to a figure indicated by the identification figure information ZPI, based on the trajectory information TI (an example of "position information"). When the determination result of the determination unit 24 is positive, the process determination unit 25 determines the process indicated by the identification information ZAI associated with the identification graphic information ZPI as the specific process to be executed by the lighting device 50. The position determination unit 26 determines an action position on the action surface SC that receives the action caused by the specific process based on the trajectory information TI (an example of "position information"). The execution unit 27 causes the lighting device 50 to execute the specific process that applies an action to the action position.
According to the first embodiment, the action providing system 1 can irradiate the illumination light to a position desired by the user within the action surface SC. The user can feel as if the illumination device 50 is placed at the position desired by the user.

In the first embodiment, the action device is a lighting device 50 that irradiates the action surface SC with illumination light. The identification information ZAI indicates an irradiation process of irradiating illumination light, and the identification figure information ZPI indicates a figure corresponding to the irradiation process. If the figure included in the trajectory image TP is similar to the irradiation process figure SPL indicating the irradiation process, the process determination unit 25 determines the irradiation process as the specific process, and the execution unit 27 causes the lighting device 50 to execute the irradiation process of irradiating the action position with illumination light.
According to the first embodiment, the action providing system 1 can irradiate illumination light to a position on the action surface SC desired by the user. For example, the user draws an irradiation processing figure SPL at a position on the action surface SC near where the user is likely to be. The user can work, study, or read using the irradiation light irradiated to the position near where the user is likely to be.

2. Second embodiment In the first embodiment, the action device is the illumination device 50, but in the second embodiment, the action device is a generation device 60 that generates projection information PRJI indicating the projection image PRJ, which is different from the first embodiment. The second embodiment will be described below.

Figure 7 is a schematic diagram illustrating an action provision system 1a according to the second embodiment. The action provision system 1a differs from the action provision system 1 in that it has an information processing device 10a instead of the information processing device 10, and has a generation device 60 instead of the lighting device 50. The generation device 60 generates projection information PRJI that indicates a projection image PRJ projected by the projection device 40 onto the action surface SC.

2.1. Configuration of the action providing system 1a Fig. 8 is an explanatory diagram showing the configuration of the action providing system 1a. The generating device 60 is any information processing device (for example, a stationary information device such as a server or a personal computer, or a portable information terminal such as a smartphone, a notebook computer, a wearable terminal, or a tablet terminal). The generating device 60 has a processing device 62, a storage device 64, and a communication device 66. Each element of the generating device 60 is connected to each other by a single or multiple buses 68 for communicating information.

The processing device 62 is a processor that controls the entire generating device 60, and is composed of, for example, one or more chips. The processing device 62 is composed of, for example, a central processing unit including an interface with peripheral devices, an arithmetic unit, and registers. Note that some or all of the functions of the processing device 62 may be realized by hardware such as a DSP, ASIC, PLD, or FPGA. The processing device 62 executes various processes in parallel or sequentially.

The storage device 64 is a recording medium readable by the processing device 62, and stores a plurality of application programs executed by the processing device 62, scenery image information PPLAI, application-generated image information PPALI, and various information used by the processing device 62. The storage device 64 may be composed of at least one of, for example, a ROM, an EPROM, an EEPROM, a RAM, etc. The storage device 64 may be called a register, a cache, a main memory (main storage device), etc. The processing device 62 can execute a plurality of application programs. Hereinafter, the "application program" is referred to as an "app." The plurality of apps are, for example, an email app, a web app, and an SNS (Social Networking Service) app. The scenery image information PPLAI indicates a scenery image PPLA. The application-generated image information PPALI indicates an application-generated image PPAL. The application-generated image information PPALI is generated by a specific app among the plurality of apps. The specific app is an app that the user causes the generating device 60 to execute. In the second embodiment, an example will be described in which the specific application is an email application. The processing device 62 may obtain multiple applications and scenery image information PPLAI, etc. from other devices communicating via the Internet, and store them in the storage device 64.

Figure 9 is a diagram showing an example of a landscape image PPLA. The landscape image PPLA is, for example, an image showing the scenery that would be seen from a window if there was a window in the guest room RM, or an image captured in a scenic location.

Figure 10 is a diagram showing an example of an application-generated image PPAL. The application-generated image PPAL shown in Figure 10 shows an email list screen.

Referring back to FIG. 8 for the explanation, the communication device 66 is hardware (transmission/reception device) for communicating with other devices. The communication device 66 is also called, for example, a network device, a network controller, a network card, a communication module, etc.

The information processing device 10a includes a processing device 20a, a communication device 32, and a storage device 35a.

The memory device 35a stores a number of programs including the control program PRa executed by the processing device 20a, the first graphic information Z1Ia, and various information used by the processing device 20a.

11 is a diagram showing an example of the contents of the first graphic information Z1Ia. The first graphic information Z1Ia is information in which the identification information ZAIa and the identification graphic information ZPIa correspond to each other. The identification information ZAIa is, for example, an identifier or a character string indicating the processing of the generating device 60. The identification information ZAIa indicates the type of display processing for displaying a predetermined image included in the projection image PRJ. In the second embodiment, there are two types of predetermined images, and the display processing also corresponds one-to-one to each of the two predetermined images. An example of the predetermined image is the landscape image PPLA illustrated in FIG. 9 and the application-generated image PPAL illustrated in FIG. 10. The identification information ZAIa is a "landscape display processing" character string indicating the landscape display processing for displaying the landscape image PPLA, and an "app image display processing" character string indicating the application image display processing for displaying the application-generated image PPAL. In the following description, the display processing is a general term for the landscape display processing and the application image display processing.

The identification graphic information ZPIa indicates a graphic corresponding to the processing of the action device. In the second embodiment, the identification graphic information ZPIa is information indicating a graphic corresponding to the display processing. Hereinafter, the graphic indicating the display processing is referred to as a "display processing graphic." In the second embodiment, the display processing graphic includes a landscape display graphic SPLA indicating a landscape display processing, and an application display graphic SPAL indicating an application image display processing. Therefore, in the second embodiment, the identification graphic information ZPIa is landscape display graphic information SPLAI indicating the landscape display graphic SPLA, and application display graphic information SPALI indicating the application display graphic SPAL.

Figure 12 is a diagram showing an example of a landscape display figure SPLA. In order for the user to memorize the landscape display figure SPLA, it is preferable that the landscape display figure SPLA is a figure that can be associated with a landscape display process, similar to the irradiation process figure SPL. In the example of Figure 12, the landscape display figure SPLA is an image showing a window frame.

Figure 13 is a diagram showing an example of an application display figure SPAL. In order for the user to memorize the application display figure SPAL, similar to the irradiation processing figure SPL, it is preferable that the application display figure SPAL is a figure that can be associated with the application image display process. In the example of Figure 13, the application display figure SPAL is an image showing the icon of an email application.

Returning to FIG. 8 for explanation, the processing device 20a reads the control program PRa and executes the control program PRa, thereby functioning as a detection unit 21, a generation unit 22, a first projection control unit 23, a judgment unit 24a, a processing determination unit 25a, a position determination unit 26a, and an execution unit 27a.

The determination unit 24a determines whether the trajectory image TP is similar to a figure indicating a display process. For example, the determination unit 24a determines whether the trajectory image TP is similar to the landscape display figure SPLA, and if the trajectory image TP is similar to the landscape display figure SPLA, outputs the determination result as positive. If the trajectory image TP is not similar to the landscape display figure SPLA, the determination unit 24a determines whether the trajectory image TP is similar to the application display figure SPAL. If the trajectory image TP is similar to the application display figure SPAL, outputs the determination result as positive. If the trajectory image TP is not similar to the application display figure SPAL, the determination unit 24a outputs the determination result as negative.

When the judgment result of the judgment unit 24a is positive, the processing decision unit 25a decides that the specific processing is a processing for generating the projection information PRJI. More specifically, when the trajectory image TP resembles the landscape display figure SPLA, the processing decision unit 25a decides that the specific processing is a processing for generating the projection information PRJI showing the projection image PRJ including the landscape image PPLA. Also, when the trajectory image TP resembles the application display figure SPAL, the processing decision unit 25a decides that the specific processing is a processing for generating the projection information PRJI showing the projection image PRJ including the application generated image PPAL.

If the trajectory image TP is similar to the landscape display figure SPLA, the position determination unit 26a determines the area inside the outer edge of the trajectory image TP as the action position.

The execution unit 27a displays the specified image on the action surface SC by notifying the generation device 60 of a generation instruction to generate projection information PRJI indicating a projection image PRJ whose position is the action position, and an output instruction to output the projection information PRJI to the projection device 40. Specifically, when the trajectory image TP resembles a landscape display figure SPLA, the projection image PRJ indicated by the projection information PRJI generated by the generation instruction is an image in which the landscape image PPLA is located in an area inside the outer edge of the figure included in the trajectory image TP.

Figure 14 is a diagram showing an example of projection of a projection image PRJ including a landscape image PPLA. In the example of Figure 14, it is assumed that the user has drawn a trajectory image TP similar to the landscape display figure SPLA on the right side of the center of the action surface SC. The generation device 60 generates projection information PRJI indicating a projection image PRJ in which the landscape image PPLA is superimposed on the inner area relative to the outer edge of the trajectory image TP. Note that the projection image PRJ does not have to include the trajectory image TP, and may include the landscape display figure SPLA instead of the trajectory image TP.

Returning to FIG. 8 for explanation, if the trajectory image TP resembles the application display figure SPAL, the execution unit 27a notifies the generation device 60 of an execution instruction to execute a specific application, a generation instruction to generate projection information PRJI indicating a projection image PRJ including an application generated image PPAL generated by the specific application, and an output instruction to output the projection information PRJI to the projection device 40.

Figure 15 is a diagram showing an example of a projection image PRJ that includes an application-generated image PPAL. In the example of Figure 15, it is assumed that the user has drawn a trajectory image TP similar to the application display figure SPAL on the right-hand side of the center of the working surface SC. The generating device 60 projects the projection image PRJ, in which the application-generated image PPAL is located on the right-hand side of the center, onto the working surface SC.

2.2 Operation of the action providing system 1a The operation of the action providing system 1a will be described with reference to Figs.

Figures 16 and 17 are flowcharts showing the operation of the action providing system 1a. The processing of steps S2, S4, S6, and S8 is the same as the steps shown in Figure 6, so the explanation is omitted.

The processing device 20a judges whether the trajectory image TP is similar to the landscape display figure SPLA indicating the landscape display process (step S32). If the judgment result of step S32 is positive, the processing device 20a determines the process of generating projection information PRJI indicating the projection image PRJ including the landscape image PPLA as the specific process (step S34). Next, the processing device 20a determines the inner area relative to the outer edge of the trajectory image TP as the action position (step S36). The processing device 20a notifies the generation device 60 of a generation instruction to generate projection information PRJI indicating the projection image PRJ in which the position of the landscape image PPLA is the action position, and an output instruction to output the projection information PRJI to the projection device 40 (step S38). After the process of step S38 is completed, the processing device 20a ends the series of processes shown in FIG. 16 and FIG. 17.

The processing device 62 of the generating device 60 determines whether it has received an instruction to generate the projection information PRJI and an instruction to output the projection information PRJI (step S40). If the determination result of step S40 is positive, the processing device 62 generates projection information PRJI indicating the projection image PRJ in which the position of the landscape image PPLA is the action position, in accordance with the instruction to generate the projection information PRJI (step S42). Then, the processing device 62 outputs the projection information PRJI to the projection device 40 in accordance with the instruction to output the projection information PRJI (step S44).

If the judgment result of step S32 is negative, the processing device 20a judges whether the trajectory image TP is similar to the application display figure SPAL indicating the application image display process (step S52). If the judgment result of step S52 is positive, the processing device 20a determines the process of generating projection information PRJI indicating the projection image PRJ including the application generated image PPAL as the specific process (step S54). Next, the processing device 20a determines the action position based on the trajectory information TI (step S56). Then, the processing device 20a notifies the generation device 60 of an execution instruction to execute the specific application, a generation instruction to generate projection information PRJI indicating the projection image PRJ including the application generated image PPAL generated by the specific application, and an output instruction to output the projection information PRJI to the projection device 40 (step S58). After the process of step S58 is completed, the processing device 20a ends the series of processes shown in FIG. 16 and FIG. 17. Furthermore, if the determination result in step S52 is negative, the processing device 20a ends the series of processes shown in Figures 16 and 17.

After the process of step S44 is completed, or if the judgment result of step S40 is negative, the processing device 62 judges whether it has received an instruction to execute a specific application, an instruction to generate projection information PRJI, and an instruction to output projection information PRJI (step S60). If the judgment result of step S60 is positive, the processing device 62 executes the specific application according to the instruction to execute the specific application (step S62). Next, the processing device 62 generates projection information PRJI indicating a projection image PRJ including an application generated image PPAL generated by the specific application according to the instruction to generate the projection information PRJI (step S64). Then, the processing device 62 outputs the projection information PRJI to the projection device 40 according to the instruction to output the projection information PRJI (step S66). After the process of step S66 is completed, the processing device 62 ends the series of processes shown in FIG. 16 and FIG. 17. Also, if the judgment result of step S60 is negative, the processing device 62 ends the series of processes shown in FIG. 16 and FIG. 17.

The projection device 40 judges whether or not the projection information PRJI has been received (step S70). If the judgment result of step S70 is positive, the projection device 40 projects the projection image PRJ indicated by the projection information PRJI onto the working surface SC (step S72). For example, if the projection device 40 receives the projection information PRJI in the processing of step S44, the projection image PRJ projected onto the working surface SC includes a landscape image PPLA. On the other hand, if the projection device 40 receives the projection information PRJI in the processing of step S66, the projection image PRJ projected onto the working surface SC includes an application-generated image PPAL. After the processing of step S72 is completed, the projection device 40 ends the series of processes shown in FIG. 16 and FIG. 17. Also, if the judgment result of step S70 is negative, the projection device 40 ends the series of processes shown in FIG. 16 and FIG. 17.

Although omitted in Figures 16 and 17, when the generation device 60 is executing a specific app, the processing device 20a notifies the generation device 60 of the position information PI. The processing device 62 updates the app-generated image PPAL based on the received position information PI, and outputs projection information PRJI indicating the projection image PRJ including the updated app-generated image PPAL to the projection device 40. The projection device 40 projects the projection image PRJ including the updated app-generated image PPAL onto the working surface SC.

2.3. Summary of the Second Embodiment In the second embodiment, the action device is a generating device 60 that generates projection information PRJI indicating a projection image PRJ projected on the action surface SC by a projection device 40 that projects an image on the action surface SC. The identification information ZAIa indicates a display process that displays a predetermined image included in the projection image PRJ, and the identification figure information ZPIa indicates a figure corresponding to the display process. If the figure included in the trajectory image TP is similar to the figure indicating the display process, the processing determination unit 25a determines the processing of generating the projection information PRJI as the specific processing. The execution unit 27 notifies the generating device 60 of an instruction to generate projection information PRJI indicating a projection image PRJ in which the position of the predetermined image is the action position, and an instruction to output the projection information PRJI to the projection device 40, thereby displaying the predetermined image on the action surface SC.
According to the second embodiment, the action providing system 1a can display a predetermined image at a position desired by the user on the action surface SC.

Also, in the second embodiment, when the image included in the trajectory image TP is similar to a landscape display figure SPLA indicating a landscape display process (an example of a “display process”), the position determination unit 26a determines the inner area relative to the outer edge of the figure included in the trajectory image TP as the action position, and the execution unit 27a notifies the generation device 60 of an instruction to generate projection information PRJI indicating a projection image PRJ in which a landscape image PPLA (an example of a “specified image”) is located in the aforementioned area, and an instruction to output the projection information PRJI to the projection device 40.
According to the second embodiment, the action providing system 1a can display the scenery image PPLA at a position desired by the user. For example, the user can easily view the scenery image PPLA by drawing the scenery display figure SPLA at a position that is easy for the user to see. The user can, for example, relax by viewing the scenery image PPLA. Furthermore, by positioning the scenery image PPLA in an area inside the outer edge of the figure included in the trajectory image TP, the scenery image PPLA is displayed inside the window frame drawn by the user. The user can get the feeling that a window is provided at the position desired by the user.

In the second embodiment, the generating device 60 is capable of executing a plurality of programs. The application-generated image information PPALI indicating the application-generated image PPAL (an example of a "predetermined image") is generated by a specific application (an example of a "specific program") among a plurality of applications (an example of a "plurality of programs"). When the image included in the trajectory image TP resembles an application display figure SPAL indicating an application image display process (an example of a "display process"), the executing unit 27a notifies the generating device 60 of an instruction to execute the specific application, an instruction to generate projection information PRJI indicating a projection image PRJ including the application-generated image PPAL generated by the specific application, and an instruction to output the projection information PRJI to the projection device 40.
According to the second embodiment, the action providing system 1a can display the application generated image PPAL at a position desired by the user. For example, it is assumed that an object that the user does not want to be superimposed on the application generated image PPAL, such as a clock or a paper note, is attached to the action surface SC. When the user draws the application display figure SPAL at a position on the action surface SC where the above-mentioned object is not attached, the action providing system 1a can display the application generated image PPAL without superimposing the above-mentioned object on the application generated image PPAL. The user can operate a specific application while looking at, for example, the clock or the paper note.

3. Modifications The present disclosure is not limited to the above-described embodiments. Specific modified aspects are exemplified below. Two or more aspects selected from the following examples may be combined.

3.1. First Modification In the first embodiment, the function providing system 1 includes the lighting device 50. However, instead of the lighting device 50, the function providing system 1 may include a blower device 70.

3.1.1 Functions of the action providing system 1b in the first modified example Fig. 18 is an explanatory diagram showing the configuration of the action providing system 1b in the first modified example. The action providing system 1b differs from the action providing system 1 in that it has an information processing device 10b instead of the information processing device 10, and has a blower device 70 instead of the lighting device 50. The blower device 70 blows air to the action surface SC. The blower device 70 is, for example, an air conditioner, an electric fan, or a circulator.

The blower device 70 has a blower fan 72, a drive mechanism 74, and a communication device 76. The elements of the blower device 70 are connected to each other by a single or multiple buses 78 for communicating information. The blower fan 72 imparts kinetic energy to the air by rotating under the control of the information processing device 10b, and sends out the air that has received the kinetic energy as wind. The drive mechanism 74 controls the direction of the wind sent out from the blower fan 72 under the control of the information processing device 10b. The communication device 76 is hardware (transmitting/receiving device) for communicating with other devices. The communication device 76 is also called, for example, a network device, a network controller, a network card, a communication module, etc.

The information processing device 10b has a processing device 20b, a communication device 32, and a storage device 35b.

The memory device 35b stores a number of programs including the control program PRb executed by the processing device 20b, the first graphic information Z1Ib, and various information used by the processing device 20b.

Figure 19 is a diagram showing an example of the contents of the first graphic information Z1Ib. The first graphic information Z1Ib is information that associates the identification information ZAIb with the identification graphic information ZPIb. The identification information ZAIb indicates the type of processing of the air blowing device 70. The identification information ZAIb is, for example, an identifier or a character string indicating the type of processing of the air blowing device 70. In the first modified example, the identification information ZAIb is the "air blowing processing" character string indicating the air blowing processing that blows air.

The identification graphic information ZPIb indicates a graphic corresponding to the processing of the action device. In the first modified example, the identification graphic information ZPIb is air blowing processing graphic information SPWI corresponding to air blowing processing. Hereinafter, the graphic indicated by the air blowing processing graphic information SPWI will be referred to as the "air blowing processing graphic SPW."

Figure 20 is a diagram showing an example of an air blowing process figure SPW. In order for a user to memorize the air blowing process figure SPW, it is preferable that the air blowing process figure SPW is a figure that can be associated with air blowing. The air blowing process figure SPW is, for example, a figure that simplifies the shape of an air conditioner, an electric fan, or a circulator, which are examples of a blowing device 70. Alternatively, the air blowing process figure SPW may be a figure that simplifies wind. In the example of Figure 20, the air blowing process figure SPW shows a figure that simplifies wind.

Returning to FIG. 18 for explanation, the processing device 20b reads the control program PRb and executes the control program PRb to function as a detection unit 21, a generation unit 22, a first projection control unit 23, a determination unit 24b, a processing determination unit 25b, a position determination unit 26, and an execution unit 27b.

The determination unit 24b determines whether the trajectory image TP is similar to the air blowing process figure SPW. If the determination result of the determination unit 24b is positive, the process determination unit 25b determines the air blowing process as the specific process.

The execution unit 27b causes the blower device 70 to execute the blowing process to blow air to the operating position. Specifically, first, the execution unit 27b notifies the drive mechanism 74 of an instruction to set the direction of the air blown from the blower fan 72 to the operating position. Second, the execution unit 27b notifies an instruction to rotate the blower fan 72. Based on these two instructions, the blower device 70 executes the blowing process to blow air to the operating position.

3.1.2 Summary of the First Modification As described above, in the first modification, the action device is the air blowing device 70 that blows air to the action surface SC, the identification information ZAIb indicates an air blowing process to blow air, and the identification figure information ZPIb indicates an air blowing process figure SPW corresponding to the air blowing process. If the image included in the trajectory image TP resembles a figure indicating the air blowing process, the process determination unit 25b identifies the air blowing process as the specific process, and the execution unit 27 causes the air blowing device 70 to execute the air blowing process to blow air to the action position.
According to the first modified example, the action providing system 1b can send air to a position desired by the user. For example, the user can draw an air-blowing processing figure SPW near an object to be dried, so that the action providing system 1b can efficiently send air to the object to be dried.

3.2. Second Modification In the first embodiment, the action providing system 1 includes the lighting device 50. However, instead of the lighting device 50, the action providing system 1 may include an audio device 80.

3.2.2. Functions of the action provision system 1c in the second modified example FIG. 21 is an explanatory diagram showing the configuration of the action provision system 1c in the second modified example. The action provision system 1c differs from the action provision system 1 in that it has an information processing device 10c instead of the information processing device 10, and has an audio device 80 instead of the lighting device 50. The audio device 80 can adjust the position for localizing a sound image. The localization of a sound image is the position of a sound perceived by a human being. The adjustment of the localization of a sound image can be realized by using multiple speakers.

The acoustic device 80 has a speaker 82, a speaker 84, a control device 86, and a communication device 87. The elements of the lighting device 50 are connected to each other by a single or multiple buses 88 for communicating information. The speakers 82 and 84 emit sound. The control device 86 receives a stereo signal and generates a first acoustic signal indicating the sound emitted by the speaker 82 and a second acoustic signal indicating the sound emitted by the speaker 84. In order to change the position of the sound image, the control device 86 executes, for example, a process of changing the volume of one or both of the first acoustic signal and the second acoustic signal, and a process of adding a phase difference. The speaker 82 emits sound by an analog signal obtained by converting the first acoustic signal by a DA converter (not shown). Similarly, the speaker 84 emits sound by an analog signal obtained by converting the second acoustic signal by a DA converter (not shown). The communication device 87 is hardware (transmitting/receiving device) for communicating with other devices. The communication device 87 may also be called, for example, a network device, a network controller, a network card, a communication module, etc. In the second variant, the audio device 80 has two speakers, speaker 82 and speaker 84, but may have three or more speakers.

The information processing device 10c has a processing device 20c, a communication device 32, and a storage device 35c. The storage device 35c stores a plurality of programs including a control program PRc executed by the processing device 20c, audio information SI, first graphic information Z1Ic, and various information used by the processing device 20b. The audio information SI has a two-channel stereo signal (left and right) representing the audio of some kind of music. For example, the audio information SI represents the audio of BGM (Background Music) played in the guest room RM.

Figure 22 is a diagram showing an example of the contents of the first graphic information Z1Ic. The first graphic information Z1Ic is information that associates the identification information ZAIc with the identification graphic information ZPIc. The identification information ZAIc indicates a process for localizing a sound image. Hereinafter, the process for localizing a sound image is referred to as "localization processing." The identification information ZAIb is, for example, an identifier or a character string indicating the localization processing. In the second modified example, the identification information ZAIc is a "localization processing" character string indicating the localization processing.

The identification graphic information ZPIc indicates a graphic corresponding to the processing of the action device. In the second modified example, the identification graphic information ZPIc is localization processing graphic information SPSI corresponding to localization processing. Hereinafter, the graphic indicated by the localization processing graphic information SPSI is referred to as the "localization processing graphic SPS."

Figure 23 is a diagram showing an example of a localization processing figure SPS. In order for a user to memorize the localization processing figure SPS, it is preferable that the localization processing figure SPS is a figure that can be associated with localization processing. The localization processing figure SPS is, for example, a figure that simplifies the shape of an audio device 80 or the shape of a speaker 82 included in the audio device 80. In the example of Figure 23, the localization processing figure SPS is a figure that simplifies the shape of a speaker 82.

Returning to FIG. 21 for explanation, the processing device 20c reads the control program PRc and executes the control program PRc, thereby functioning as a detection unit 21, a generation unit 22, a first projection control unit 23, a determination unit 24c, a processing determination unit 25c, a position determination unit 26, and an execution unit 27c.

The determination unit 24c determines whether the trajectory image TP is similar to the localization processing figure SPS. If the determination result of the determination unit 24c is positive, the processing decision unit 25c decides that the localization processing is the specific processing.

The execution unit 27c causes the acoustic device 80 to execute localization processing. Specifically, the execution unit 27c notifies the control device 86 of an instruction to execute delay processing for the input stereo signal such that a sound image is localized at the action position. Secondly, the execution unit 27c transmits the audio information SI to the acoustic device 80. As a result, the acoustic device 80 executes localization processing to localize a sound image at the action position.

3.2.2 Summary of the second modified example As described above, in the second modified example, the action device is the acoustic device 80 capable of adjusting the position for localizing the sound image, the identification information ZAIc indicates the process for localizing the sound image, and the identification figure information ZPIc indicates the localization processing figure SPS corresponding to the process for localizing the sound image. If the image included in the trajectory image TP resembles the localization processing figure SPS indicating the process for localizing the sound image, the process determination unit 25c identifies the process for localizing the sound image as the specific process. The execution unit 27c causes the acoustic device 80 to execute the localization process for localizing the sound image at the action position.
According to the second modified example, the effect providing system 1c can localize a sound image at a position desired by the user.

3.3. Third Modification The action providing system 1 may allow the user to set the magnitude of the physical action to be applied to the action surface SC.

3.3.1 Function of the action provision system 1d in the third modified example Fig. 24 is an explanatory diagram showing the configuration of the action provision system 1d in the third modified example. In the diagram shown in Fig. 24, in order to prevent the drawing from becoming complicated, the projection device 40, the infrared light imaging device 45, the lighting device 50, the action surface SC, and the indicator 15 are omitted. The action provision system 1d differs from the action provision system 1 in that it has an information processing device 10d instead of the information processing device 10.

The information processing device 10d has a processing device 20d, a communication device 32, and a storage device 35d. The storage device 35d stores a plurality of programs including the control program PRd executed by the processing device 20d, the first graphic information Z1I, and various information used by the processing device 20d.

The processing device 20d reads the control program PRd and executes the control program PRd to function as a detection unit 21, a generation unit 22, a first projection control unit 23, a judgment unit 24, a processing determination unit 25, a position determination unit 26, a size determination unit 28, and an execution unit 27d.

The size determination unit 28 determines the magnitude of the physical effect that the lighting device 50 has on the action surface SC according to the size of the trajectory image TP. In the third modified example, the magnitude of the physical effect that the lighting device 50 has on the action surface SC is the amount of light emitted by the lighting device 50. The size of the trajectory image TP is, for example, the area of the trajectory image TP or the maximum width within the trajectory image TP. The size determination unit 28 monotonically increases the amount of light of the illumination light as the size of the trajectory image TP increases.

The execution unit 27d causes the lighting device 50 to execute a specific process that applies an effect to the action position according to the magnitude determined by the size determination unit 28.

As described above, in the third modified example, the processing device 20d also functions as a size determination unit 28 that determines the size of the physical action that the action device applies to the action surface SC according to the size of the trajectory image TP. The execution unit 27 causes the lighting device 50 to execute a specific process that applies an action according to the size determined by the size determination unit 28 to the action position.
According to the third modified example, the user can specify the magnitude of the physical action to be applied to the action surface SC without drawing a figure for specifying the magnitude of the physical action to be applied to the action surface SC. Also, the user can intuitively specify the magnitude of the physical action to be applied to the action surface SC.

In the third modified example, the lighting device 50 has been described as an example of an action device, but this is not limiting. The action device may be the generation device 60 or the air blowing device 70. When the action device is the generation device 60, the size determination unit 28 monotonically increases the size of the specified image in response to an increase in the size of the trajectory image TP. Also, when the action device is the air blowing device 70, the size determination unit 28 monotonically increases the air blowing speed in response to an increase in the size of the trajectory image TP.

3.4. Fourth Modification The action providing system 1 may be able to set the magnitude of the physical action to be applied to the action surface SC by a method different from that of the third modification. Note that, in the above description, it is assumed that the trajectory image TP contains one figure, but in the fourth modification, it is assumed that the trajectory image TP contains two or more figures.

Figure 25 is an explanatory diagram showing the configuration of action provision system 1e in the fourth modified example. In the diagram shown in Figure 25, in order to prevent the drawing from becoming too complicated, the projection device 40, the infrared light imaging device 45, the lighting device 50, the action surface SC, and the indicator 15 are omitted. Action provision system 1e differs from action provision system 1 in that it has an information processing device 10e instead of information processing device 10.

The information processing device 10e has a processing device 20e, a storage device 35e, and a communication device 32. The storage device 35e stores a plurality of programs including a control program PRe executed by the processing device 20e, the first graphic information Z1I, the second graphic information Z2I, and various information used by the processing device 20e.

Figure 26 is a diagram showing an example of the contents of the second graphic information Z2I. The second graphic information Z2I is information that associates size information ZBI with size graphic information ZQI. The size information ZBI indicates the magnitude of the effect that the lighting device 50 has on the action surface SC. In the fourth modified example, the size information ZBI is an "x candela" character string indicating that the amount of light irradiated by the lighting device 50 is x candelas, and a "y candela" character string indicating that the amount of light irradiated by the lighting device 50 is y candelas. Candela is a unit that expresses the amount of light. x and y are real numbers, and y is greater than x.

The magnitude graphic information ZQI shows a figure corresponding to the magnitude of the action. It is preferable that the magnitude graphic information ZQI is a figure that associates the magnitude of the action. In the fourth modified example, the first magnitude graphic information SP1I corresponds to the character string "x candela", and the second magnitude graphic information SP2I corresponds to the character string "y candela". The first magnitude graphic information SP1I shows the first magnitude graphic SP1. The second magnitude graphic information SP2I shows the second magnitude graphic SP2.

Fig. 27 is a diagram showing an example of the first size figure SP1. Fig. 28 is a diagram showing an example of the second size figure SP2. The first size figure SP1 and the second size figure SP2 are figures imitating a volume control knob attached to a general audio device or a scroll bar used in a general app. The knob SL in the first size figure SP1 is located lower than the knob SL in the second size figure SP2, which means that the amount of light emitted by the lighting device 50 is less.

Returning to FIG. 25 for explanation, the processing device 20e reads the control program PRe and executes the control program PRe to function as a detection unit 21, a generation unit 22, a first projection control unit 23, a judgment unit 24, a processing determination unit 25, a position determination unit 26, a size determination unit 28e, and an execution unit 27e.

When the image included in the trajectory image TP resembles the figure indicated by the size graphic information ZQI, the size determination unit 28e determines the magnitude of the effect indicated by the size information ZBI associated with the size graphic information ZQI as the magnitude of the physical effect that the lighting device 50 exerts on the action surface SC. The specific processing of the size determination unit 28e will be described with reference to FIG. 29.

Figure 29 is a diagram showing an example of a trajectory image TP. The trajectory image TP shown in Figure 29 includes a division figure TPS-1 and a division figure TPS-2. It is assumed that the determination unit 24 has determined that the division figure TPS-1 is similar to the irradiation processing figure SPL. The size determination unit 28e determines whether the remaining division figures of the trajectory image TP other than the division figure determined by the determination unit 24 to be similar to the irradiation processing figure SPL are similar to the figure indicated by the size figure information ZQI. Specifically, the size determination unit 28e determines whether the division figure TPS-2 is similar to the first size figure SP1. In the example of Figure 29, it is assumed that the size determination unit 28e has determined that the division figure TPS-2 is not similar to the first size figure SP1. Next, the size determination unit 28e determines whether the division figure TPS-2 is similar to the first size figure SP2. In the example of FIG. 29, it is assumed that the size determination unit 28e has determined that the division figure TPS-2 is similar to the second size figure SP2. The size determination unit 28e determines the y candela indicated by the "y candela" character string associated with the second size figure SP2 as the magnitude of the physical effect that the lighting device 50 exerts on the action surface SC.

Returning to FIG. 27 for explanation, the execution unit 27e causes the lighting device 50 to execute an irradiation process that imparts an effect at the action position according to the size determined by the size determination unit 28e. For example, if the image included in the trajectory image TP resembles the first size figure SP1, the execution unit 27e causes the lighting device 50 to execute an irradiation process that irradiates with a light amount of x candelas. Also, if the image included in the trajectory image TP resembles the second size figure SP2, the execution unit 27e causes the lighting device 50 to execute an irradiation process that irradiates with a light amount of y candelas.

3.4.2 Summary of the Fourth Modification As described above, in the fourth modification, the storage device 35e further stores the magnitude information ZBI indicating the magnitude of the action and the size graphic information ZQI indicating a graphic corresponding to the magnitude of the action. When the image included in the trajectory image TP resembles the graphic indicated by the size graphic information ZQI, the processing device 20e also functions as a size determination unit 28 that determines the magnitude of the action indicated by the size information ZBI associated with the size graphic information ZQI as the magnitude of the physical action that the lighting device 50 applies to the action surface SC. The execution unit 27e causes the lighting device 50 to execute a specific process that applies an action according to the magnitude determined by the size determination unit 28 to the action position.
In the third modified example, when the user attempts to specify a large magnitude of the physical effect on the action surface SC, the user is required to draw the trajectory image TP large, which places a heavy burden on the user. On the other hand, according to the fourth modified example, even when the user specifies a large magnitude of the physical effect on the action surface SC, the user is not required to draw the trajectory image TP large. Therefore, even when the user specifies a large magnitude of the physical effect on the action surface SC, the user can specify the large magnitude more easily than in the third modified example.

3.5. Fifth Modification The action providing system 1 may add processing of an action device corresponding to the trajectory image TP when the trajectory image TP is not similar to the figure indicated by the identification figure information ZPI.

3.5.1. Configuration of the action provision system 1f Fig. 30 is an explanatory diagram showing the configuration of the action provision system 1f in the fifth modified example. The action provision system 1f differs from the action provision system 1 in that it has an information processing device 10f instead of the information processing device 10. In the diagram shown in Fig. 30, in order to prevent the drawing from becoming complicated, the projection device 40, the infrared light imaging device 45, the lighting device 50, the action surface SC, and the indicator 15 are omitted.

The information processing device 10f includes a processing device 20f, a communication device 32, and a storage device 35f. The storage device 35f stores a plurality of programs including a control program PRf executed by the processing device 20f, the first graphic information Z1I, list image information DPI, and various information used by the processing device 20f.

Figure 31 is a diagram showing an example of a list image DP indicated by the list image information DPI. The list image DP includes character strings indicating the type of processing of the lighting device 50. As shown in the example of Figure 31, the list image DP includes a "lighting processing" character string and a "light-off processing" character string as processing of the lighting device 50.

Returning to FIG. 29 for explanation, the processing device 20f reads the control program PRf and executes the control program PRf, thereby functioning as a detection unit 21, a generation unit 22, a first projection control unit 23, a judgment unit 24, a processing determination unit 25, a position determination unit 26, an execution unit 27, a second projection control unit 29, and an addition unit 31.

If the judgment result of the judgment unit 24 is negative, the second projection control unit 29 causes the projection device 40 to project the list image DP.

The adding unit 31 adds identification information indicating the type of process selected by the user from the list image DP to the first graphic information Z1I in association with the information indicating the trajectory image TP. The process selected by the user is, for example, a process corresponding to the character string in the list image DP where the pointer 15 is located. The process of the adding unit 31 will be described with reference to Figures 31 and 32.

Figure 32 is a diagram showing an example of a trajectory image TP when the determination result of the determination unit 24 is negative. In the example of Figure 32, the user draws a figure similar to the irradiation processing figure SPL, and then draws a diagonal line segment, thereby drawing a figure that corresponds to the opposite process to the irradiation process, i.e., the turning off process.

Figure 33 is a diagram showing an example of the contents of the first graphic information Z1I after addition by the adding unit 31. In the example of Figure 32, it is assumed that the user has selected "lights-off process." The adding unit 31 adds trajectory information TI indicating the trajectory image TP to the storage device 35 as lights-off process graphic information. Furthermore, the adding unit 31 adds "lights-off process" to the first graphic information Z1I in association with the lights-off process graphic information.

3.5.2. Summary of Fifth Modification In the fifth modification, the processing device 20f functions as the second projection control unit 29 (an example of a "projection control unit") and also as the adding unit 31. When the determination result of the determining unit 24 is negative, the second projection control unit 29 causes the projection device 40 to project a list image DP including a character string indicating the type of processing of the action device. The adding unit 31 adds identification information indicating the type of processing selected by the user from the list image DP to the first graphic information Z1I (an example of a "memory device") in association with information indicating the graphic included in the trajectory image TP.
According to the fifth modification, the user can add a figure associated with the processing of the action device. For example, even if there is no identification graphic information ZPI associated with the type of processing of the action device, the user can add the identification graphic information ZPI associated with the type of processing of the action device. Furthermore, even if the user cannot remember the figure indicated by the identification graphic information ZPI included in advance in the first graphic information Z1I, the user can add a figure that is easy for the user to remember to the first graphic information Z1I.

3.6. Sixth Modification In the above-described aspects, the processing device 20 functions as the first projection control section 23 , but it does not have to function as the first projection control section 23 .

Figure 34 is an explanatory diagram showing the configuration of an action provision system 1g in the sixth modified example. The action provision system 1g differs from the action provision system 1 in that it has an information processing device 10g instead of the information processing device 10 and does not have a projection device 40.

The information processing device 10g has a processing device 20g, a communication device 32, and a storage device 35g. The storage device 35g stores a plurality of programs including the control program PRg executed by the processing device 20g, the first graphic information Z1I, and various information used by the processing device 20g.

The processing device 20g reads the control program PRg and executes the control program PRg, thereby functioning as a detection unit 21, a generation unit 22, a judgment unit 24, a processing decision unit 25, a position decision unit 26, and an execution unit 27. In other words, the processing device 20g differs from the processing device 20 in that it does not function as the first projection control unit 23.

In the sixth modified example, as in the first embodiment, a physical action is applied to a position on the action surface SC desired by the user. Even if the trajectory image TP is not projected onto the action surface SC, if the figure indicated by the identification figure information ZPI included in the first figure information Z1I is simple, the user can draw the figure indicated by the identification figure information ZPI.

3.7. Seventh Modification In the second embodiment, a landscape display process and an application image display process are described as examples of the display process, but the display process is not limited to the landscape display process and the application image display process. For example, the display process may be a memo pad display process that displays a memo pad. In the seventh modification, the identification figure information ZPI is information indicating a memo pad display process figure SPMM that indicates the memo pad display process. The specified image is a memo pad image PPMM that indicates a memo pad.

Figure 35 is a diagram showing an example of the memo pad display processing figure SPMM. In order for the user to memorize the memo pad display processing figure SPMM, similar to the irradiation processing figure SPL, it is preferable that the memo pad display processing figure SPMM is a figure that can be associated with a memo pad display process. In the example of Figure 35, the memo pad display processing figure SPMM is an image showing a memo pad.

Figure 36 is a diagram showing an example of a trajectory image TP. Figure 36 shows an example in which a user has drawn a memo pad display processing figure SPMM. Furthermore, in Figure 36, the memo pad display processing figure SPMM contains a character string ST1 handwritten by the user in Japanese. The character string ST1 is "Please clean the room."

In the seventh modified example, when a character string is included in the memo pad display processing figure SPMM, the execution unit 27 translates the character string into a second language different from the first language. The second language is, for example, the native language of the cleaner who cleans the guest room RM. In the seventh modified example, the second language is described as English. The memo pad image PPMM is displayed on the action surface SC by notifying the generation device 60 of a generation instruction to generate projection information PRJI indicating a projection image PRJ in which the position of the memo pad image PPMM is the action position, and an output instruction to output the projection information PRJI to the projection device 40. Furthermore, this memo pad image PPMM includes the character string translated into the second language.

Figure 37 shows an example of a memo pad image PPMM projected onto the working surface SC. The memo pad image PPMM includes a string ST2 that is an English translation of the string ST1. The string ST2 is "Please make up this room."

According to the seventh variant, for example, a user can convey a message to a cleaner whose native language is different from that of the user.

3.8. Eighth Modification In the second embodiment, the scenery image PPLA is described as an image showing a scenery that would be seen from a window if the guest room RM had a window, or an image captured in a scenic location, but is not limited to this. For example, the storage device 64 may store, as the scenery image information PPLAI, information showing an image captured in a certain location in the morning, information showing an image captured in the same location in the daytime, and information showing an image captured in the same location in the evening. The certain location may be any location, for example, outside the guest room RM or a scenic location. The execution unit 27a notifies the generation device 60 of an instruction to generate the projection information PRJI and an instruction to output the projection information PRJI. This generation instruction is an instruction to generate, as a predetermined image, the projection information PRJI showing the projection image PRJ in which the position of the image captured in the time period corresponding to the current time is the action position.
In addition, there may be a plurality of images showing the scenery that would be seen from the window if there were a window in the guest room RM. For example, when the action surface SC is the inner wall surface on the east side and the inner wall surface on the south side with the guest room RM as the center, the storage device 64 may store, as the scenery image information PPLAI, an image showing the scenery that would be seen from the window if there were a window on the east side of the guest room RM, and an image showing the scenery that would be seen from the window if there was a window on the south side of the guest room RM. The execution unit 27a notifies the generation device 60 of an instruction to generate the projection information PRJI and an instruction to output the projection information PRJI. This generation instruction is an instruction to generate, as a predetermined image, the projection information PRJI showing the projection image PRJ in which the position of the image corresponding to the action position is the action position. The image corresponding to the action position is, when the action position is within the east-side interior wall surface, an image showing the scenery that would be seen from a window if there were a window on the east side of the guest room RM, and, when the action position is within the south-side interior wall surface, an image showing the scenery that would be seen from a window if there were a window on the south side of the guest room RM.

3.9. Ninth Modification In each of the above-mentioned aspects, the determination unit 24 determines whether or not the figure indicated by the identification figure information ZPI is similar to the figure included in the trajectory image TP based on the trajectory information TI, but this is not limited thereto. The determination unit 24 may determine whether or not the figure indicated by the identification figure information ZPI is similar to the figure included in the trajectory image TP based on the position information PI, which is the base information of the trajectory information TI. For example, the determination unit 24 identifies two line segments connecting three positions in a continuous period in the position information PI. These line segments are a part of the trajectory indicated by the trajectory information TI. The determination unit 24 compares the intersections included in the part of this trajectory with the vertices of the figure indicated by the identification figure information ZPI as feature points to determine whether or not the figure indicated by the identification figure information ZPI is similar to the figure included in the trajectory image TP.

3.10. Tenth Modification In each of the above-mentioned aspects, the action providing system 1 has one action device, but may have multiple action devices. For example, the action providing system 1 may have a lighting device 50 and a blowing device 70. The determination unit 24 determines whether the trajectory image TP is similar to the irradiation processing figure SPL. If the trajectory image TP is similar to the irradiation processing figure SPL, the processing determination unit 25 determines the irradiation processing as the specific processing. If the trajectory image TP is not similar to the irradiation processing figure SPL, the determination unit 24 determines whether the trajectory image TP is similar to the air blowing processing figure SPW. If the trajectory image TP is similar to the air blowing processing figure SPW, the processing determination unit 25 determines the air blowing processing as the specific processing.

3.11. Eleventh Modification In each of the above-mentioned aspects, the position determination unit 26 determines the action position based on the trajectory information TI, but this is not limiting. For example, the position determination unit 26 may determine the action position based on the position information PI. Specifically, the position determination unit 26 determines, as the action position, a position indicated by an average value of each coordinate value indicated by a series of coordinate information included in the position information PI.

In the above-described aspects, the information processing device 10 is provided in the guest room RM, but it is not necessary to provide the information processing device 10 in the guest room RM as long as the information processing device 10 can access the projection device 40. The generation device 60 is similar to the projection device 40.

In each of the above-described aspects, the information processing device 10 and the projection device 40 may be integrated together. The information processing device 10 and the infrared light imaging device 45 may be integrated together. Furthermore, the information processing device 10 and the generation device 60 may be integrated together.

3.14. Fourteenth Modification The action providing system 1 is provided in the hotel guest room RM, but may be provided in a room in a home. The action providing system 1 may also be provided outdoors.

3.15. Fifteenth Modification The indicator 15 is not limited to an operation device, and may be a user's finger. When the indicator 15 is a user's finger, the action providing system 1 has an imaging device that images the action surface SC in a state where visible light is transmitted, instead of the infrared light imaging device 45. The detection unit 21 identifies the user's fingertip based on image information obtained from the imaging device. The detection unit 21 detects information indicating the identified position of the user's fingertip as position information PI.

3.16. Other Modifications (1) The block diagrams used to explain each aspect above show functional blocks. These functional blocks (components) are realized by any combination of hardware and/or software. The means for realizing each functional block is not particularly limited. That is, each functional block may be realized by one device that is physically and/or logically coupled, or may be realized by two or more devices that are physically and/or logically separated and directly and/or indirectly (e.g., wired and/or wirelessly) connected to each other and these multiple devices.

(2) The order of the processing steps, sequences, flow charts, etc. in each of the above-described aspects may be changed without causing any inconsistency. For example, the methods described herein present elements of various steps in an exemplary order and are not limited to the particular order presented.

(3) In each of the above aspects, the input and output information, etc. may be stored in a specific location (e.g., memory) or may be managed in a management table. The input and output information, etc. may be overwritten, updated, or added to. The output information, etc. may be deleted. The input information, etc. may be transmitted to another device.

(4) In each of the above aspects, the determination may be made based on a value represented by one bit (0 or 1), a Boolean value (true or false), or a comparison of numerical values (e.g., a comparison with a predetermined value).

(5) In each of the above-mentioned aspects, the storage device 35 is a recording medium readable by the processing device 20. Although ROM and RAM have been exemplified, the storage device 35 may be a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory device (e.g., a card, a stick, a key drive), a CD-ROM (Compact Disc-ROM), a register, a removable disk, a hard disk, a floppy (registered trademark) disk, a magnetic strip, a database, a server, or any other suitable storage medium. The program may also be transmitted from a network. The program may also be transmitted from a communication network via a telecommunications line.

(6) Each of the above-mentioned aspects may be applied to systems that utilize LTE (Long Term Evolution), LTE-A (LTE-Advanced), SUPER 3G, IMT-Advanced, 4G, 5G, FRA (Future Radio Access), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi), IEEE 802.16 (WiMAX), IEEE 802.20, UWB (Ultra-Wide Band), Bluetooth (registered trademark), or other suitable systems, and/or next-generation systems that are based on and extend these.

(7) In each of the above aspects, the described information, signals, etc. may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.
In addition, terms explained in this specification and/or terms necessary for understanding this specification may be replaced with terms having the same or similar meanings.

(8) Each function illustrated in Figures 2, 8, 18, 21, 24, 25, 30, and 34 may be realized by any combination of hardware and software. Furthermore, each function may be realized by a single device, or may be realized by two or more devices configured separately from each other.

(9) The programs exemplified in each of the above-described embodiments should be broadly interpreted to mean instructions, instruction sets, code, code segments, program code, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., regardless of whether they are called software, firmware, middleware, microcode, hardware description language, or by other names.
Software, instructions, etc. may also be transmitted or received over a transmission medium. For example, if the software is transmitted from a website, server, or other remote source using wired technologies, such as coaxial cable, fiber optic cable, twisted pair, and Digital Subscriber Line (DSL), and/or wireless technologies, such as infrared, radio, and microwave, these wired and/or wireless technologies are included within the definition of transmission media.

(10) In each of the above-described embodiments, information, parameters, etc. may be expressed as absolute values, as relative values from a predetermined value, or as corresponding other information.

(11) The names used for the above parameters are not intended to be limiting in any way. Furthermore, the formulas etc. using these parameters may differ from those explicitly disclosed herein.

(12) In each of the above-described embodiments, the information processing device 10 may be a mobile station. A mobile station may also be referred to by those skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or some other suitable term.

(13) In each of the above-described embodiments, the term "based on" does not mean "based only on," unless otherwise specified. In other words, the term "based on" means both "based only on" and "based at least on."

(14) Any reference to elements using designations such as "first," "second," etc., as used herein does not generally limit the quantity or order of those elements. These designations may be used herein as a convenient way to distinguish between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed therein or that the first element must precede the second element in some way.

(15) In each of the above embodiments, to the extent that the terms "including," "comprising," and variations thereof are used in this specification or the claims, these terms are intended to be inclusive, similar to the term "comprises." Furthermore, the term "or" as used in this specification or the claims is not intended to be an exclusive or.

(16) Throughout this application, where articles have been added by translation, such as a, an and the in English, these articles include the plural unless the context clearly indicates otherwise.

(17) It is clear to those skilled in the art that the present invention is not limited to the embodiments described in this specification. The present invention can be implemented in modified and altered forms without departing from the spirit and scope of the present invention as defined by the claims. Therefore, the description in this specification is for illustrative purposes only and does not have any limiting meaning on the present invention. In addition, multiple aspects selected from the aspects exemplified in this specification may be combined.

1, 1a, 1b, 1c, 1d, 1e, 1f, 1g...action providing system, 10, 10a, 10b, 10c, 10d, 10e, 10f, 10g...information processing device, 21...detection unit, 22...generation unit, 23...first projection control unit, 24, 24a, 24b, 24c...determination unit, 25, 25a, 25b, 25c...processing determination unit, 26, 26a...position determination unit, 27, 27a, 27b, 27c, 27d, 27 e...execution unit, 28, 28e...size determination unit, 29...second projection control unit, 31...addition unit, 35, 35a, 35b, 35c, 35d, 35e, 35f, 35g...storage device, 40...projection device, 50...lighting device, 60...generation device, 70...blower device, 80...acoustic device, DP...list of images, DPI...list of image information, PI...position information, PPAL...application-generated image, PPALI...application-generated image information , PPLA... scenery image, PPLAI... scenery image information, PRJ... projection image, PRJI... projection information, SC... action surface, SP1... first size figure, SP1I... first size figure information, SP2... second size figure, SP2I... second size figure information, SPAL... application display figure, SPALI... application display figure information, SPL... irradiation processing figure, SPLA... scenery display figure, SPLAI... scenery display figure information information, SPLI...irradiation processing figure information, SPS...positioning processing figure, SPSI...positioning processing figure information, SPW...airflow processing figure, SPWI...airflow processing figure information, TI...trajectory information, TP...trajectory image, Z1I, Z1Ia, Z1Ib, Z1Ic...first figure information, Z2I...second figure information, ZAI, ZAIa, ZAIb, ZAIc...identification information, ZPI, ZPIa, ZPIb, ZPIc...identification figure information.

Claims (10)

An information processing device capable of controlling an action device that applies a physical action to an action surface,
a storage device that stores identification information indicating a type of processing of the action device and identification graphic information indicating a graphic corresponding to the type of processing in association with each other;
A detection unit that detects position information indicating a position of a pointer with respect to the action surface;
a determination unit that determines whether or not a figure included in a trajectory image showing a trajectory of the position of the indicator is similar to a figure indicated by the identification figure information, based on the position information;
a process determining unit that, when a result of the determination by the determining unit is positive, determines a process indicated by identification information associated with the identification graphic information as a specific process to be executed by the action device;
a position determination unit that determines an action position on the action surface that receives the action generated by the specific process based on the position information;
an execution unit that causes the action device to execute the specific process of applying the action to the action position;
An information processing device comprising: the action device is an illumination device that irradiates illumination light onto the action surface,
the identification information indicates an irradiation process of irradiating the illumination light,
the identification graphic information indicates a graphic corresponding to the irradiation treatment,
If a figure included in the trajectory image is similar to a figure corresponding to the irradiation process,
The process determination unit determines the irradiation process as the specific process,
The execution unit causes the lighting device to execute the irradiation process at the action position.
The information processing device according to claim 1 . the action device is a generating device that generates projection information indicating a projection image to be projected onto the action surface by a projection device that projects an image onto the action surface;
the identification information indicates a display process for displaying a predetermined image included in the projection image,
the identification graphic information indicates a graphic corresponding to the display process,
If a figure included in the trajectory image is similar to a figure corresponding to the display process,
the process determination unit determines a process for generating the projection information as the specific process;
the execution unit notifies the generation device of an instruction to generate the projection information indicating the projection image in which the position of the predetermined image is the action position, and an instruction to output the projection information to the projection device, thereby displaying the predetermined image on the action surface.
The information processing device according to claim 1 . If an image included in the trajectory image is similar to a figure corresponding to the display process,
the position determination unit determines an inner area with respect to an outer edge of a figure included in the trajectory image as the action position;
the execution unit notifies the generation device of an instruction to generate the projection information indicating the projection image in which the predetermined image is located in the area, and an instruction to output the projection information to the projection device;
The information processing device according to claim 3 . The generating device is capable of executing a plurality of programs;
the information indicating the predetermined image is generated by a specific program among the plurality of programs,
If an image included in the trajectory image is similar to a figure corresponding to the display process,
the execution unit notifies the generation device of an instruction to execute the specific program, an instruction to generate the projection information indicating the projection image including the predetermined image generated by the specific program, and an instruction to output the projection information to the projection device.
The information processing device according to claim 3 . The action device is a blower device that blows air onto the action surface,
The identification information indicates an air blowing process to blow air,
the identification graphic information indicates a graphic corresponding to the air blowing process,
When an image included in the trajectory image resembles a figure corresponding to the air blowing process,
The process determination unit identifies the air blowing process as the specific process,
The execution unit causes the air blowing device to execute the air blowing process of blowing air to the operating position.
The information processing device according to claim 1 . The action device is an acoustic device capable of adjusting a position for localizing a sound image,
The identification information indicates a process for localizing a sound image,
the identification graphic information indicates a graphic corresponding to the processing for localizing the sound image,
If the image included in the trajectory image resembles a figure corresponding to a process for localizing a sound image,
the processing determination unit identifies a processing for localizing the sound image as the specific processing;
The execution unit causes the acoustic device to execute a process of localizing a sound image at the action position.
The information processing device according to claim 1 . a size determination unit that determines a size of the physical action that the action device applies to the action surface in accordance with a size of the trajectory image;
The execution unit causes the action device to execute the specific process of applying an action to the action position according to the magnitude determined by the magnitude determination unit.
The information processing device according to claim 1 . The storage device further stores magnitude information indicating the magnitude of the action and magnitude graphic information indicating a graphic corresponding to the magnitude of the action in association with each other;
a size determination unit that, when an image included in the trajectory image resembles a figure indicated by the size graphic information, determines a size of the action indicated by the size information associated with the size graphic information as a size of the physical action applied by the action device to the action surface;
The execution unit causes the action device to execute the specific process of applying an action to the action position according to the magnitude determined by the magnitude determination unit.
The information processing device according to claim 1 . a projection control unit that causes a projection device that projects an image onto the action surface to project a list image including a character string indicating a type of processing of the action device when the determination result of the determination unit is negative;
an adding unit that adds, to the storage device, identification information indicating a type of processing selected by a user from the list image in association with information indicating a figure included in the trajectory image,
The information processing device according to claim 1 .

Images