JP6369604B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

The present invention relates to an image processing apparatus , an image processing method , and a program.

  In recent years, augmented reality systems (AR (Augmented Reality) systems) using augmented reality technology that superimposes virtual objects as if they exist in real space have come to be used. For example, when an image including an AR marker is captured, an augmented reality system is known in which a virtual object image is synthesized and displayed in a captured image corresponding to the type and position of the marker.

  However, an augmented reality system using markers requires markers corresponding to the type of virtual object image to be displayed. Therefore, for example, in Patent Document 1, a marker posting mobile terminal stores a plurality of types of markers and virtual object images in association with each other, and the marker selected by the user is displayed on the marker posting mobile terminal. Thus, a technique is described in which a virtual object to be displayed can be selected from a plurality of types of virtual objects without carrying a printed matter of a plurality of markers.

JP 2005-250950 A

  However, also in Patent Document 1, there is one virtual object image associated with one AR marker, and in order to display another, the user performs a selection operation and displays it on the marker posting portable terminal. It was necessary to switch the type of marker to be used.

  An object of the present invention is to enable a plurality of different displays with one marker.

In order to solve the above problem, an image processing apparatus according to claim 1 is provided.
Image acquisition means ;
A frame-in direction recognizing unit for recognizing a direction in which the marker has been framed when a predetermined marker is framed in the first image acquired by the image acquiring unit ;
Generating means for generating a second image corresponding to the direction in which the marker recognized by the frame-in direction recognition means is framed;
It is characterized by providing .

  According to the present invention, it is possible to perform a plurality of different displays with one marker.

It is a block diagram which shows the functional structure of the image processing apparatus in this embodiment. It is a figure which shows the example of data storage of a frame in / frame out information storage part. It is a figure which shows the example of data storage of an operation | movement pattern database. It is a flowchart which shows the display control process performed by CPU of FIG. It is a figure for demonstrating the method of recognizing the direction in which the AR marker was framed in, and the direction which was out of frame. It is a figure which shows an example of the display operation by a display control process. It is a figure which shows the operation method of the frame in in the mobile terminal of the type held in a hand, and a frame out. It is a figure which shows the operation method of the frame in in a spectacles type HMD, and a frame out.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The present invention is not limited to the illustrated example.

<Configuration of Image Processing Device 1>
First, the configuration of the image processing apparatus 1 according to the present embodiment will be described.
As the image processing apparatus 1, for example, a mobile terminal such as a smartphone, a tablet terminal, a notebook PC (Personal Computer), or a handy terminal is applicable.

  FIG. 1 shows a functional configuration example of the image processing apparatus 1. As shown in FIG. 1, the image processing apparatus 1 includes a CPU (Central Processing Unit) 10, a RAM (Random Access Memory) 11, a storage unit 12, a communication unit 13, a display unit 14, an operation unit 15, a camera 16, and a current time. An acquisition unit 17 and the like are provided, and each unit is connected by a bus 18.

  The CPU 10 reads out the program stored in the storage unit 12 and develops it in the work area of the RAM 11, and executes various processes including a display control process described later according to the developed program. The CPU 10 functions as frame-in recognition means, frame-in direction recognition means, frame-out recognition means, frame-out direction recognition means, marker recognition means, and control means by executing display control processing.

  The RAM 11 is a volatile memory, and has a work area for storing various programs executed by the CPU 10, data related to these various programs, and the like.

The RAM 11 also includes a frame-in / frame-out information storage unit 111 for storing history information in the direction in which the AR (Augmented Reality) marker 5 (see FIG. 6) is framed in and in the direction in which the frame is out.
Here, the AR marker 5 is an image for defining information (for example, a virtual object image or the like) to be displayed on the screen of the display unit 14. In the frame-in, when the captured image acquired by the camera 16 is displayed on the screen of the display unit 14, the AR marker 5 enters the screen from a state where the AR marker 5 does not exist. That means. Frame out means that when the captured image acquired by the camera 16 is displayed on the screen of the display unit 14, the AR marker 5 displayed on the screen of the display unit 14 disappears from the screen. To go).

  FIG. 2 shows an example of data storage in the frame-in / frame-out information storage unit 111. As illustrated in FIG. 2, the frame-in / frame-out information storage unit 111 includes, for example, fields of “order”, “operation content”, and “direction”. The “order” stores the order in which the operations are performed. “Operation content” stores information indicating whether the operation is frame-in or frame-out. “Direction” stores information indicating the direction of frame-in or frame-out.

  The storage unit 12 includes an HDD (Hard Disk Drive), a semiconductor nonvolatile memory, and the like. For example, as illustrated in FIG. 1, the storage unit 12 includes a program storage unit 121 and an operation pattern database 122.

  The program storage unit 121 stores a system program executed by the CPU 10, various processing programs, data necessary for executing these programs, and the like. For example, the program storage unit 121 stores an AR marker application program. These programs are stored in the program storage unit 121 in the form of computer-readable program codes. The CPU 10 sequentially executes operations according to the program code.

As shown in FIG. 3, the operation pattern database 122 includes a series of frame in / frame out operation pattern information of the AR marker 5 and display information corresponding to the operation pattern (the display unit 14 according to the operation pattern). Are stored in association with each other). The information of the operation pattern includes the content (frame-in or frame-out) of each operation constituting the series of operations, the order, and the direction (for example, from the left, from the right, from the top, from the bottom). Since frame-out is an operation performed after a frame-in, an odd-numbered sequence operation is a frame-in and an even-numbered sequence operation is a frame-out. Further, the motion pattern database 122 stores a motion pattern consisting of only the first frame-in from each direction and display information corresponding to the motion in the direction of each frame-in.
The storage unit 12 stores a pattern file indicating the image pattern of the AR marker 5.

  The communication unit 13 includes a LAN (Local Area Network) adapter, a router, and the like, and performs communication connection with an external device via a communication network such as a LAN to perform data transmission / reception.

  The display unit 14 is configured by an LCD (Liquid Crystal Display) or the like, and is a display unit that performs various displays on the screen in accordance with a display control signal from the CPU 10.

  The operation unit 15 includes a cursor key, various function keys, and a shutter key. The operation unit 15 receives a pressing input of each key by the user and outputs operation information to the CPU 10. The operation unit 15 includes a touch panel or the like in which transparent electrodes are arranged in a grid so as to cover the surface of the display unit 14, detects a position pressed with a finger or a touch pen, and uses the position information as operation information. It outputs to CPU10.

The camera 16 includes an imaging element such as a lens, a diaphragm, a CCD (Charge Coupled Device), or a CMOS (Complementary Metal Oxide Semiconductor), and forms an optical image of a subject on the imaging element, and uses the image as an electrical signal. This is imaging means for outputting to the CPU 10.
The current time acquisition unit 17 is configured by an RTC (Real Time Clock) or the like, measures the current date and time, and outputs it to the CPU 10.

<Operation of Image Processing Apparatus 1>
Next, the operation of the image processing apparatus 1 in the present embodiment will be described.

  FIG. 4 shows a flowchart of the display control process executed by the image processing apparatus 1. The display control process is executed in cooperation with the CPU 10 and the AR marker application program stored in the program storage unit 121 when the operation unit 15 instructs the activation of the AR marker application.

  First, the CPU 10 activates the camera 16 (step S1). Note that after the camera 16 is activated, the camera 16 acquires a captured image every predetermined time during the execution of the display control process. The CPU 10 stores the captured image acquired by the camera 16 in the RAM 11 in association with the current time from the current time acquisition unit 17 and displays it on the screen of the display unit 14 in substantially real time.

  Next, the CPU 10 waits for recognition of the frame in of the AR marker 5 (step S2). Specifically, the CPU 10 performs recognition processing of the AR marker 5 by image processing on the captured image acquired by the camera 16 every predetermined time. The recognition process of the AR marker 5 can be performed by a known method. For example, a rectangular area surrounded by a black frame is recognized from the captured image, the image pattern in the area surrounded by the black frame is compared with the pattern file of the AR marker 5 stored in the storage unit 12, and the matching rate is The AR marker 5 is recognized when it is equal to or greater than a predetermined threshold. When the AR marker 5 is recognized from the state in which the AR marker 5 is not recognized, the CPU 10 recognizes that the AR marker 5 has entered the frame.

When the frame-in of the AR marker 5 is recognized (step S2; YES), the CPU 10 based on the plurality of captured images acquired from the camera 16 after the frame-in of the AR marker 5 is recognized. Is obtained, and the direction in which the AR marker 5 enters the frame is recognized based on the obtained locus (step S3).
Specifically, first, the CPU 10 sets the X axis, the Y axis, and the coordinate origin O (0, 0) on the captured image. Next, as shown in FIG. 5, the center coordinate P <b> 1 of the AR marker 5 is obtained from each of the captured image in which the AR marker 5 is recognized and the n captured images acquired by the camera 16 at predetermined time intervals thereafter. X, Y), P2 (X, Y)... Pn (X, Y) are obtained, and a regression curve L is drawn with respect to the obtained central coordinate group. Then, the side E (the side of the screen frame of the display unit 14) that intersects the regression curve at the closest position from the center coordinate P1 (X, Y) of the captured image in which the AR marker 5 is recognized is defined as the AR marker. 5 is recognized as the frame-in side, and the direction of the side where the AR marker 5 is framed in is recognized as the direction in which the AR marker 5 is framed in. For example, if the side where the AR marker 5 is framed in is left, the direction in which the AR marker 5 is framed is recognized as “from the left”.

  Next, the CPU 10 stores history information in the direction in which the AR marker 5 is framed in the frame-in / frame-out information storage unit 111 (step S4).

Next, the CPU 10 determines whether or not an operation pattern that matches the history stored in the frame-in / frame-out information storage unit 111 of the RAM 11 is stored in the operation pattern database 122 (step S5).
When it is determined that an operation pattern that matches the history stored in the frame-in / frame-out information storage unit 111 is stored in the operation pattern database 122 (step S5; YES), the CPU 10 Based on the display information stored in association with the matching operation pattern, display is performed on the display unit 14 (step S6). That is, the display unit 14 is caused to perform predetermined display according to the history of the direction in which the AR marker is framed in and the direction in which the AR marker is out. For example, a virtual object image corresponding to the operation pattern is synthesized and displayed at the position of the AR marker 5 in the captured image displayed on the display unit 14.
When it is determined that the operation pattern matching the history stored in the frame-in / frame-out information storage unit 111 is not stored in the operation pattern database 122 (step S5; NO), the CPU 10 proceeds to the process of step S7. To do. That is, the display of the currently displayed information is continued as it is.

  Note that, as described above, the motion pattern database 122 stores a motion pattern including only the first frame-in from each direction and display information corresponding to each frame-in direction in association with each other. . Therefore, when the frame-in is recognized for the first time from the initial state, a predetermined display according to the direction in which the AR marker 5 is framed is performed.

  Next, the CPU 10 waits for the recognition of the frame out of the AR marker 5 (step S7). During this time, the CPU 10 performs the above-described recognition process of the AR marker 5 on the captured image acquired by the camera 16 every predetermined time, and when the AR marker 5 is no longer recognized, the AR marker 5 is out of the frame. recognize.

  When the frame out of the AR marker 5 is recognized (step S7; YES), the CPU 10 starts measuring time with the internal clock (step S8). Further, the CPU 10 acquires the locus of the coordinates of the AR marker 5 based on the plurality of captured images acquired from the camera 16 immediately before the AR marker 5 is recognized as being out of the frame, and based on the acquired locus. The direction in which the AR marker 5 is out of the frame is recognized (step S9). Specifically, as shown in FIG. 5, the CPU 10 stores the center coordinates P11 (X, Y) and P12 of the AR marker 5 from the n captured images stored in the RAM 11 immediately before the AR marker 5 is not recognized. (X, Y)... P1n (X, Y) is obtained, and a regression curve L is drawn with respect to the obtained central coordinate group. Next, the AR marker 5 frames out the side E (side of the screen frame of the display unit 14) that intersects the regression curve L at the closest position from the most recent central coordinate P11 (X, Y) where the AR marker 5 is no longer recognized. The direction of the side where the AR marker 5 is out of frame is recognized as the direction where the AR marker 5 is out of frame. For example, when the AR marker 5 recognizes that the side out of the frame is left, the direction in which the AR marker 5 is out of frame is recognized as “from the left”.

  Next, the CPU 10 stores history information in the frame-out direction in the frame-in / frame-out information storage unit 111 of the RAM 11 (step S10).

  Next, the CPU 10 performs recognition processing of the AR marker 5 on the captured image after the frame-out, and determines whether or not the frame-in of the AR marker 5 has been recognized (step S11). When it is determined that the frame-in of the AR marker 5 has been recognized (step S11; YES), the CPU 10 returns to the process of step S3 and repeatedly executes the processes of step S3 to step S11.

When it is determined that the frame-in of the AR marker 5 has not been recognized (step S11; NO), the CPU 10 determines whether or not a predetermined time has elapsed from the start of timing (that is, after frame-out) (step S11). S12).
When it is determined that the predetermined time has not elapsed since the start of time measurement (that is, after frame out) (step S12; NO), the process returns to step S11.
When it is determined that a predetermined time has elapsed from the start of timing (that is, after frame out) (step S12; YES), the CPU 10 stores the captured image and frame-in / frame-out information storage unit 111 stored in the RAM 11. The stored history information is reset (erased) (step S13). Thereby, it will be in an initial state.

  Until the end instruction of the AR marker application is input from the operation unit 15, the processes in steps S2 to S13 are repeatedly executed. When an instruction to end the AR marker application is input from the operation unit 15 (step S14; YES), the CPU 10 ends the display control process.

Here, the display operation by the display control process will be described with reference to a specific example in FIG.
For example, when the image processing apparatus 1 is moved to the left in the initial state shown in FIG. 6A, the AR marker 5 is framed in from the left as shown in FIG. 6B. Here, since the operation history from the initial state until now (frame-in from the left) matches the operation pattern stored in the operation pattern database 122 as shown in FIG. 3, it corresponds to this operation pattern. Based on the attached display information (rabbit), a rabbit image is displayed at the position of the AR marker 5.

  Next, when the image processing apparatus 1 is moved to the right, as shown in FIG. 6C, the AR marker 5 is out of the frame from the left. When the image processing apparatus 1 is moved to the left from here, the AR marker 5 is framed in from the left as shown in FIG. Here, the operation history from the initial state until now (frame in from left → frame out from left → frame in from left) is not stored in the operation pattern database 122. Therefore, the rabbit is continuously displayed on the AR marker 5.

  Next, when the image processing apparatus 1 is moved upward, as shown in FIG. 6E, the AR marker 5 is framed out from below. When the image processing apparatus 1 is moved downward from here, the AR marker 5 is framed in from below as shown in FIG. Here, the operation history from the initial state until now (from left to frame in → from left to frame out → from left to frame in → from bottom to frame out → from bottom to frame in) is an operation pattern stored in the operation pattern database 122. Therefore, the cat image is displayed at the position of the AR marker 5 based on the display information (cat) associated with this operation pattern.

  In this way, as shown in FIG. 7, the image processing apparatus 1 is moved up, down, left, and right, and only one AR marker 5 is framed in / out with an operation pattern registered in the operation pattern database 122. It is possible to cause the unit 14 to perform a desired display according to the operation.

As described above, according to the image processing apparatus 1 in the present embodiment, the CPU 10 recognizes that the AR marker 5 has entered the frame in the screen of the display unit 14 based on the captured image from the camera 16. The direction in which the AR marker 5 enters the frame is recognized, and the display unit 14 performs a predetermined display corresponding to the direction in which the AR marker 5 enters the frame.
Therefore, even with the same single AR marker 5, it is possible to cause the display unit 14 to perform a plurality of different displays depending on the direction in which the AR marker 5 is framed. As a result, there is no need to perform a selection operation for selecting the type of AR marker, and convenience is improved.

Further, the CPU 10 stores the history information of the direction in which the AR marker 5 is framed in and the direction in which the AR marker 5 is out in the frame-in / frame-out information storage unit 111, and when the frame in of the AR marker 5 is recognized. The display unit 14 is caused to perform predetermined display according to the history of the direction in which the AR marker 5 stored in the frame-in / frame-out information storage unit 111 has been framed in and the direction in which the frame is out.
Therefore, even with the same one AR marker 5, it is possible to cause the display unit 14 to perform a plurality of different displays according to a series of operations in the direction in which the AR marker 5 is framed in and the direction in which the AR marker 5 is out. As a result, there is no need to perform a selection operation for selecting the type of AR marker, and convenience is improved.

  Further, since the CPU 10 recognizes the direction in which the AR marker 5 is framed in and the direction in which the AR marker 5 is out using the image processing technology, the CPU 10 can be realized with a simple device configuration without mounting hardware such as an acceleration sensor. Is possible.

  In addition, the description content in the said embodiment is a suitable example of an image processing apparatus, and is not limited to this.

  For example, in the above-described embodiment, the case where the image processing apparatus 1 is a mobile terminal of a type held in the hand of a smartphone or the like has been described as an example, but a glasses-type HMD (Head Mounted Display) or the like may be used. In this case, as shown in FIG. 8, since the marker can be framed in and out by the swing motion, the display on the display unit 14 can be switched hands-free.

  In the above embodiment, display information is associated in advance with an operation pattern that combines frame-in and frame-out from various directions. For example, an operation pattern that includes only frame-in (for example, Display information may be stored in association with the frame in from the right → the frame in from the left → the frame in from the top. When the frame-in of the AR marker 5 is recognized, the CPU 10 determines whether or not the history of the direction in which the AR marker 5 is framed matches the operation pattern stored in advance. It is good also as displaying on the display part 14 based on the display information according to the operation | movement pattern. Further, for example, display information may be stored in association with an operation pattern of only frame out (for example, frame out from right → frame out from left → frame out from top, etc.). When the frame-in of the AR marker 5 is recognized, the CPU 10 determines whether or not the history of the direction in which the AR marker 5 has been out of frame matches the pre-stored operation pattern. In some cases, display on the display unit 14 may be performed based on display information corresponding to the operation pattern.

In the above embodiment, when a series of operation history of frame-in and frame-out from the initial state matches the operation pattern stored in the operation pattern database 122, display according to the operation pattern is performed. However, it is not limited to this.
For example, the storage unit 12 stores in advance a frame-in direction and display information corresponding to the direction, and the CPU 10 recognizes each time the direction in which the AR marker 5 is framed is recognized. The display unit 14 may be caused to perform a predetermined display corresponding to the direction in which the AR marker 5 is framed based on the display information previously associated with the frame-in direction.
Further, for example, the storage unit 12 stores in advance the frame-out direction and the display information corresponding to the frame-out direction, and the CPU 10 recognizes the direction when the AR marker 5 recognizes the frame-out direction. The direction is stored in the RAM 11, and when the frame-in of the AR marker 5 is recognized, the AR marker 5 is immediately framed out based on the display information previously associated with the frame-out direction recognized immediately before. It is good also as making the display part 14 perform the predetermined | prescribed display according to a direction.
Further, for example, the storage unit 12 stores a frame-out direction and display information corresponding to the frame-out direction in advance, and the CPU 10 recognizes the frame-out direction of the AR marker 5 and recognizes the frame-out direction. Display information associated with the selected direction may be displayed. At this time, since the display information is displayed by using the frame out of the marker 5 as a trigger, the display information can be displayed even if the marker 5 does not enter the frame thereafter.
Even in this case, similarly to the above-described embodiment, even with the same AR marker 5, a plurality of different displays are performed on the display unit 14 depending on the direction in which the AR marker 5 is framed in or the direction in which the AR marker 5 is out. It becomes possible to make it.

  In the above embodiment, the direction in which the AR marker 5 is framed in and the direction in which the AR marker 5 is out of frame is recognized by the image processing technique. However, if the image processing apparatus 1 is equipped with an acceleration sensor or a gyro sensor. These directions may be used to recognize the frame-in direction and the frame-out direction.

  Moreover, in the said embodiment, although the display based on one AR marker 5 was demonstrated, there may be multiple types of AR marker. In this case, a plurality of operation pattern databases 122 are stored for each AR marker type, and display control processing is performed using an operation pattern database corresponding to the recognized AR pattern type.

  Further, not only the direction in which the AR marker 5 is framed in and the direction in which the AR marker 5 is out, but also the time from frame in to frame out may be a parameter for controlling the display contents.

  Also, as a computer-readable medium storing a program for executing each of the above processes, a non-volatile memory such as a flash memory, a portable recording medium such as a CD-ROM, in addition to a ROM, a hard disk, etc. Is also possible. A carrier wave is also used as a medium for providing program data via a predetermined communication line.

  In addition, the detailed configuration and detailed operation of each apparatus constituting the image processing apparatus can be changed as appropriate without departing from the spirit of the invention.

Although several embodiments of the present invention have been described, the scope of the present invention is not limited to the above-described embodiments, but includes the scope of the invention described in the claims and equivalents thereof.
The invention described in the scope of claims attached to the application of this application will be added below. The item numbers of the claims described in the appendix are as set forth in the claims attached to the application of this application.
[Appendix]
<Claim 1>
Imaging means;
Display means for displaying a captured image acquired by the imaging means;
Frame-in recognition means for recognizing that a predetermined marker has been framed in the screen of the display means;
Frame-in direction recognition means for recognizing the direction in which the marker is in frame;
Control means for causing the display means to perform a predetermined display according to the direction in which the marker is framed;
An image processing apparatus comprising:
<Claim 2>
Frame out recognition means for recognizing that the marker is out of frame from the screen of the display means;
Frame out direction recognition means for recognizing the direction in which the marker is out of frame;
Storage means for storing history information of the direction in which the marker is in and out of the frame;
Further comprising
When the marker frame-in is recognized after the marker frame-out is recognized, the control means stores the history of the marker-in direction and frame-out direction stored in the storage means. The image processing apparatus according to claim 1, wherein the display unit performs a predetermined display in response.
<Claim 3>
Imaging means;
Display means for displaying a captured image acquired by the imaging means;
Frame-in recognition means for recognizing that a predetermined marker has been framed in the screen of the display means;
Frame out recognition means for recognizing that the marker is out of frame from the screen of the display means;
Frame out direction recognition means for recognizing the direction in which the marker is out of frame;
Storage means for storing the direction in which the marker is out of frame;
When the marker frame-in is recognized after the marker frame-out is recognized, the display unit performs a predetermined display corresponding to the direction in which the marker stored in the storage unit is out-of-frame. Control means;
An image processing apparatus comprising:
<Claim 4>
The frame-in direction recognizing unit recognizes the coordinates of the marker after the frame-in of the marker is recognized based on a plurality of captured images acquired from the imaging unit after the frame-in of the marker is recognized. The image processing apparatus according to claim 1, wherein a trajectory is acquired, and a direction in which the marker is framed in is recognized based on the acquired trajectory.
<Claim 5>
The frame-out direction recognizing unit is configured to determine the coordinates of the marker immediately before the marker frame-out is recognized based on a plurality of captured images acquired from the imaging unit immediately before the marker frame-out is recognized. The image processing apparatus according to claim 2, wherein a trajectory is acquired and a direction in which the marker is out of frame is recognized based on the acquired trajectory.
<Claim 6>
Imaging means;
Display means for displaying a captured image acquired by the imaging means;
Marker recognition means for recognizing a predetermined marker in the screen of the display means;
Frame out direction recognition means for recognizing the direction in which the marker is out of frame;
Control means for causing the display means to perform a predetermined display according to the direction in which the marker is out of frame;
An image processing apparatus comprising:
<Claim 7>
A computer used in an image processing apparatus including an imaging unit,
Display means for displaying a captured image acquired by the imaging means;
Frame-in recognition means for recognizing that a predetermined marker has entered into the screen of the display means,
Frame-in direction recognition means for recognizing the direction in which the marker is in-frame,
Control means for causing the display means to perform a predetermined display according to the direction in which the marker is in-frame;
Program to function as.
<Claim 8>
A computer used in an image processing apparatus including an imaging unit,
Display means for displaying a captured image acquired by the imaging means;
Frame-in recognition means for recognizing that a predetermined marker has entered into the screen of the display means,
Frame-out recognition means for recognizing that the marker is out of frame from the display means screen;
Frame out direction recognition means for recognizing the direction in which the marker is out of frame,
Storage means for storing a direction in which the marker is out of frame;
When the marker frame-in is recognized after the marker frame-out is recognized, the display unit performs a predetermined display corresponding to the direction in which the marker stored in the storage unit is out-of-frame. Control means,
Program to function as.
<Claim 9> A computer used in an image processing apparatus including an imaging unit,
Display means for displaying a captured image acquired by the imaging means;
Marker recognition means for recognizing a predetermined marker in the screen of the display means;
Frame out direction recognition means for recognizing the direction in which the marker is out of frame,
Control means for causing the display means to perform a predetermined display according to the direction in which the marker is out of frame;
Program to function as.

1 Image processing apparatus 10 CPU
11 RAM
111 frame-in / frame-out information storage unit 12 storage unit 121 program storage unit 122 operation pattern database 13 communication unit 14 display unit 15 operation unit 16 camera 17 current time acquisition unit

Claims (12)

Image acquisition means ;
A frame-in direction recognizing unit for recognizing a direction in which the marker has been framed when a predetermined marker is framed in the first image acquired by the image acquiring unit ;
Generating means for generating a second image corresponding to the direction in which the marker recognized by the frame-in direction recognition means is framed;
An image processing apparatus comprising: Frame out direction recognition means for recognizing the direction in which the marker is out of frame when the marker is out of frame from within the first image acquired by the image acquisition means ;
Storage means for storing history information of the direction in which the marker is in and out of the frame;
Further comprising
When the marker is recognized after the marker has been framed out, the generating means stores the history of the direction in which the marker has been framed in and the direction in which the marker has been stored stored in the storage means. The image processing apparatus according to claim 1, wherein the second image is generated in response. A display control means for controlling the display means to display the second image;
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. The generating unit further includes a combining unit that combines the first image and the second image.
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. The generation unit generates an image representing an operation pattern of a predetermined object as the second image.
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. Said frame-in direction recognition means, on the basis of the first image that will be acquired from the image acquisition unit after the frame-in of the marker is recognized, the coordinates of the marker after frame-in of the marker is recognized And recognizing the direction in which the marker is framed in based on the acquired trajectory .
The image processing apparatus according to claim 1 , wherein the image processing apparatus is an image processing apparatus. The frame-out direction recognizing means is based on the first image acquired from the image acquisition means immediately before the marker frame-out is recognized, and the marker coordinates immediately before the marker frame-out is recognized. And recognizing the direction in which the marker is out of the frame based on the acquired locus.
The image processing apparatus according to claim 2 , wherein the image processing apparatus is an image processing apparatus. An image acquisition step;
A frame-in direction recognition step for recognizing a direction in which the marker has been framed when a predetermined marker has been framed in the first image acquired by the image acquisition step;
A generation step of generating a second image corresponding to the direction in which the marker recognized by the frame-in direction recognition step is framed;
An image processing method comprising: An image processing method executed by an image processing apparatus including a storage unit that stores history information of a direction in which the marker is framed in and a direction in which the marker is out of frame,
A frame-out direction recognition step for recognizing a direction in which the marker is out of frame when the marker is out of frame from within the first image acquired in the image acquisition step;
Further comprising
In the generation step, when the frame-in of the marker is recognized after the frame-out of the marker is recognized, the history of the direction in which the marker is framed in and the direction in which the marker is out is stored in the storage unit. Generating the corresponding second image,
The image processing method according to claim 8. Computer
Image acquisition means,
A frame-in direction recognizing unit for recognizing a direction in which the marker is framed in when a predetermined marker is framed in the first image acquired by the image acquiring unit;
Generating means for generating a second image corresponding to the direction in which the marker recognized by the frame-in direction recognizing means is framed;
A program characterized by functioning as Computer
Image acquisition means,
A frame-out direction recognition means for recognizing a direction in which the marker is out of frame when a predetermined marker is out of frame from within the first image acquired by the image acquisition means;
Generating means for generating a second image corresponding to the direction in which the marker recognized by the frame-out direction recognition means is out of frame;
A program characterized by functioning as A computer comprising storage means for storing history information of the direction in which the marker is framed in and the direction in which the marker is out;
Frame out direction recognition means for recognizing the direction in which the marker is out of frame when the marker is out of frame from within the first image acquired by the image acquisition means;
Further function as
When the marker is recognized after the marker has been framed out, the generating means stores the history of the direction in which the marker has been framed in and the direction in which the marker has been stored stored in the storage means. Generating the corresponding second image,
The program according to claim 10.

Images