WO2019111348A1 - Video processing device, video processing method, computer program, and video processing system - Google Patents

Abstract

The present invention achieves video processing suitable for a reciprocating movement of a specific part by a user. A prediction part 21e predicts an amount of time it takes for a wrist of a user to move between a close position B and a distant position T on the basis of a sensor output which is output from a movement information output device 3. An image selection part 21c selects first photographing frame data at a first timing at which the wrist of the user is at the close position B during the reciprocating movement and selects second photographing frame data at a second timing at which the predicted movement time has passed from the first timing. A time interval setting part 21d sets the time interval between the sets of photographing frame data, that is, the time interval from the first photographing frame data to the second photographing frame data, to a time interval based on the predicted movement time.

Description

Video processing apparatus, video processing method, computer program, and video processing system

The present invention relates to a video processing apparatus, a video processing method, a computer program, and a video processing system.

There is known a video processing system which reproduces video content prepared in advance in conjunction with the operation of a user. For example, Patent Document 1 proposes a training apparatus incorporating a video processing system. In this training apparatus, shooting frame data shot every time the vehicle travels a prescribed distance is displayed in conjunction with the traveling distance on the treadmill.

JP-A-9-220308

The training device of Patent Document 1 assumes exercise that continues to run in one direction, such as running, and does not assume reciprocating motion of a specific part by the user. The training device of Patent Document 1 has room for improvement in this respect.
The present invention has been made in view of such circumstances, and an object thereof is to realize video processing suitable for reciprocating motion of a specific part by a user.

In order to solve the above-mentioned problems, the present invention is an operation which periodically changes with reciprocating motion of a specific part of the user between the first position and the second position, which is output from the operation information output unit. An image processing apparatus that processes an image based on information, and an image storage unit that stores a plurality of captured image information captured in time series, and an image selection that selects the captured image information stored in the image storage unit. And a time interval setting unit for setting a time interval between the photographed image information, wherein the photographed image information corresponds to first photographed image information corresponding to the first position, and the second position. The image selection unit further includes a prediction unit that includes second captured image information, and predicts, based on the operation information, the movement time of the specific part between the first position and the second position. For the reciprocating motion The first specific image information is selected at the first timing at which the fixed part is located at the first position, and the specific part after the movement time predicted by the prediction unit has passed from the first timing is The second captured image information is selected at a second timing when the second position is reached, and the time interval setting unit determines the time between the captured image information from the first captured image information to the second captured image information. The interval is set to a time interval based on the movement time predicted by the prediction unit.

According to the present invention, it is possible to realize video processing suitable for reciprocating motion of a specific part by a user.

FIG. 1 is a diagram showing a schematic configuration of a video display system according to an embodiment of the present invention. It is a figure showing the appearance of an operation information output device. (A) is a figure which shows the whole structure of a rowing machine. (B) is a figure which shows a user's wrist of a rowing machine in a position away from a user's chest. (C) is a figure which shows a user's wrist of a rowing machine in a position close | similar to a user's chest. (A) is a figure explaining the hardware constitutions of a control device. (B) is a figure which shows the various data memorize | stored in the memory | storage part of a control apparatus. (C) is a figure which shows the functional block of a control part. It is a figure showing hardware constitutions of an operation information output device. (A) is a figure showing each axis of a gyro sensor. (B) is a figure showing each axis of an acceleration sensor. (C) is a figure explaining normalization of the sensor output outputted from a gyro sensor. It is a figure which shows the relationship between the sensor output from the gyro sensor after normalization, and the position of a wrist. It is a figure which shows switching of the introductory scene, the interlocking | linkage scene, and the finish scene contained in video content. (A) is a figure which shows the kind of imaging | photography frame data. (B) is a figure which shows the kind of audio | speech data. (C) is a figure which shows the metadata added with respect to imaging | photography flame | frame data and audio | voice data. (D) is a figure which shows the relationship between the interlocking scene of each period, and the reciprocation frequency of a wrist. It is a figure which shows the relationship between the marker data in each period of a interlocking scene, and imaging | photography frame data. (A) is a figure which shows the kind of audio | speech data. (B) is a figure which shows the production | generation procedure of the audio | voice data for reproduction | regeneration. (A) is a figure which shows reproduction | regeneration of the audio | voice data for reproduction | regeneration in a standard time interval. (B) is a figure which shows reproduction | regeneration of the audio | voice data for reproduction | regeneration in a time interval narrower than a standard time interval. (C) is a figure which shows reproduction | regeneration of the audio | voice data for reproduction | regeneration in a time interval wider than a standard time interval. (A) is a figure which shows a prediction time calculation procedure. (B) is a figure which shows the dispersion | variation in prediction time. It is a figure which shows the time interval of the imaging | photography flame | frame data in prediction time, and an interval adjustment curve. It is a figure which shows the time interval between the imaging | photography flame | frame data adjusted with the space | interval adjustment curve. It is a figure which shows the time interval between the imaging | photography flame | frame data adjusted by the other space | interval adjustment curve. It is a figure which shows production | generation of synthetic | combination frame data F (mix1)-F (mix 4). (A) is a figure which shows production | generation of the synthetic | combination frame data F (mix1). (B) is a figure which shows production | generation of synthetic | combination frame data F (mix2). (C) is a figure showing generation of synthetic frame data F (mix 3). (D) is a figure showing generation of synthetic frame data F (mix 4). (A) is a figure which shows a prediction time calculation procedure. (B) is a figure which shows the dispersion | variation in prediction time. It is a figure which shows the time interval of the imaging | photography flame | frame data in prediction time TS6 ', and an interval adjustment curve. It is a figure which shows production | generation of synthetic | combination frame data F (mix11)-F (mix14). (A) is a figure which shows production | generation of synthetic | combination frame data F (mix11). (B) is a figure which shows production | generation of synthetic | combination frame data F (mix12). (C) is a figure which shows production | generation of the synthetic | combination frame data F (mix13). (D) is a figure showing generation of synthetic frame data F (mix14). (A) is a flowchart which shows the signal monitoring process which CPU of a control apparatus performs. (B) is a flowchart which shows the main processing which CPU of a control device performs. It is a flowchart which shows the interlocking scene reproduction process which CPU of a control apparatus performs. It is a figure which shows the example which makes imaging frame data in which the time interval was adjusted into display frame data.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a diagram showing a schematic configuration of a video display system 1 according to an embodiment of the present invention.
<Schematic Configuration of Video Display System 1>
The video display system 1 shown in FIG. 1 stores video content (a plurality of shooting frame data shot in time series and audio data to be played back along with the plurality of shooting frame data), and controls playback of the video content. Control device 2, an operation information output device 3 (operation information output unit) which is attached to the user and is communicably connected to the control device 2 and transmits / receives various signals to / from the control device 2; And a sound output device 5 connected communicably to the control device 2 and capable of reproducing audio of the video content.

The control device 2 functions as a video processing device that processes video content to be displayed on the display device 4 and the sound output device 5 based on a sensor output (motion information, which will be described later) from the motion information output device 3. The control device 2 can use, for example, a general personal computer. The control device 2 will be described in detail later.

FIG. 2 is a view showing the appearance of the operation information output device 3. As shown in FIG. 2, the operation information output device 3 is attached to the housing 3 a housing the components such as the control unit 31 and the gyro sensor 32 (see FIG. 5) and the housing 3 a, and is used by the user's wrist. And a band 3b wound around the (specific part). The first mode switch 34 and the second mode switch 35, which are operated when switching the playback scene of the video content, and various operations related to the playback of the video content are operated on the surface of the housing 3a. A playback related switch 36 and a menu switch 37 operated when displaying a menu screen related to the video content are provided. As described later, the motion information output device 3 transmits, to the control device 2, a sensor output in which the signal level changes periodically as the user's wrist reciprocates.

As the display device 4, as shown in FIG. 1, a head mounted display is preferably used. Display frame data is wirelessly transmitted and received between the display device 4 and the control device 2. The display frame data is read from the frame buffer 23 (see FIG. 4) included in the control device 2 and transmitted. When the display frame data is received, the display device 4 displays an image according to the display frame data, specifically, a stereoscopic image which can be viewed stereoscopically.
The display device 4 is not limited to a head mounted display, and may be a liquid crystal display or a projector. Communication between the display device 4 and the control device 2 is not limited to wireless communication but may be wired communication.

As the sound output device 5, as shown in FIG. 1, headphones are preferably used. An acoustic signal based on audio data (described later) is wirelessly transmitted and received between the sound output device 5 and the control device 2. When the sound signal is input to the sound output device 5, the sound output device 5 outputs a sound according to the sound signal.
The sound output device 5 is not limited to headphones but may be a speaker. Communication between the sound output device 5 and the control device 2 is not limited to wireless communication but may be wired communication.

FIG. 3A is a diagram showing an entire configuration of the rowing machine 6. FIG. 3 (b) is a view showing that the user's wrist of the rowing machine 6 is located at a distance from the user's chest. FIG. 3C shows the wrist of the user of the rowing machine 6 in a position close to the chest of the user.
The video display system 1 according to the present embodiment reproduces the video content in conjunction with the operation of the user, for example, when using the rowing machine 6 shown in FIGS. 3 (b) and 3 (c). As video content reproduced when using the rowing machine 6, for example, a video of a boat competition according to the player's view is preferable.
The illustrated rowing machine 6 has a prismatic frame 61, a seat member 62 movable along the frame 61, on which the user's buttocks can be placed, and a frame that can rotate around the proximal end. And a damper having a substantially T-shaped handle 63 whose tip end is gripped by the user's hands, a damper whose one end is rotatably attached to the frame 61 and whose other end is rotatably attached to the handle 63 64 and a footrest member 65 on which the user's sole is placed.

As shown in FIGS. 3B and 3C, the user of the rowing machine 6 wears the motion information output device 3 on the wrist (specific part) at the time of training, and the display 4 on the head (head mounted display And the sound output device 5 (headphones).
The user uses the rowing machine 6 to perform bending and stretching exercises between the state in which the body is bent as shown in FIG. 3 (b) and the state in which the body is extended as shown in FIG. 3 (c). The user's wrist is positioned at a distant position away from the user's chest with the body bent and at a close position close to the user's chest with the body extended. That is, the user's wrist (specific part) reciprocates along the trajectory of the other end of the handle 63 between the close position and the separated position. Therefore, the motion information output device 3 mounted on the wrist of the user also reciprocates between the close position and the separated position.

<Control device 2>
The control device 2 will be described in detail. FIG. 4A is a diagram for explaining the hardware configuration of the control device 2. FIG. 4B is a view showing various data stored in the storage unit of the control device 2.

As shown in FIG. 4A, the control device 2 stores display frame data corresponding to an image to be displayed on the display device 4, and a control unit 21 serving as a main body of various controls, a storage unit 22 storing various data. A frame buffer 23 and a communication interface (I / F) 24 for transmitting and receiving information to and from other devices are provided.
The control unit 21 includes a central processing unit (CPU) 21 a that develops a program stored in the storage unit 22 in a random access memory (RAM) 21 b and executes the program.
The storage unit 22 is, for example, a hard disk drive (HDD) or a solid state drive (SSD), and as illustrated in FIG. 4B, a program executed by the CPU 21a and shooting frame data (shooting image data constituting video content) Information, voice data, and metadata are stored. Since the photographing frame data is stored, the storage unit functions as an image storage unit. Each data will be described in detail later.

Returning to FIG. 4A, the frame buffer 23 is a volatile memory that stores display frame data for one screen displayed by the display device 4. The display frame data is generated by the control unit 21 based on the shooting frame data stored in the storage unit 22. The control unit 21 writes the generated display frame data in the frame buffer 23. The generation of display frame data by the control unit 21 will be described later.
The communication interface 24 transmits and receives information to and from the operation information output device 3. The communication interface 24a transmits and receives information to and from the operation information output device 3. The communication interface 24b transmits and receives information to and from the display device 4. A communication interface 24c is provided. As the communication interface 24, for example, one conforming to a communication method or protocol such as Bluetooth (registered trademark) or WirelessUSB (Wireless Universal Serial Bus) can be used. In the case of wired communication, one conforming to a method such as RS-232C or USB (Universal Serial Bus) can be used.

FIG. 4C is a diagram showing functional blocks of the control unit 21. As shown in FIG. The functional block shown in FIG. 4C is realized by the CPU 21a developing a program stored in the storage unit 22 on the RAM 21b and executing the program.
The image selection unit 21 c selects shooting frame data stored in the storage unit 22 (image storage unit). The time interval setting unit 21d sets a time interval between the imaging frame data. The prediction unit 21e outputs the movement information output device 3 (movement information output time) of the movement time of the user's wrist (specific part) between one (first position) and the other (second position) of the proximity position and the separation position. Prediction based on the sensor output (operation information) output from
The display image information generation unit 21 f generates display frame data by performing alpha blending processing on a pair of shooting frame data which are adjacent to each other in accordance with the timing defined by the display device 4. The photographed image information selection unit 21g selects photographed frame data of any one cycle among photographed frame data of a plurality of cycles (a plurality of sets of photographed image information) based on a periodic change of sensor output (operation information). select. The communication control unit 21 h controls the communication interfaces 24 a to 24 c to communicate with the operation information output device 3, the display device 4, and the sound output device 5.

<Operation information output device 3>
The operation information output device 3 will be described. FIG. 5 is a diagram showing a hardware configuration of the operation information output device 3. As shown in FIG. 5, the operation information output device 3 is controlled mainly to perform various controls in addition to the first mode switch 34, the second mode switch 35, the reproduction related switch 36, and the menu switch 37 described above A communication interface (I / F) 38 for transmitting and receiving information to and from the control unit 2, a gyro sensor 32 for detecting an angular velocity with respect to a reference axis, an acceleration sensor 33 for detecting acceleration, There is.
The control unit 31 includes a CPU 31a (Central Processing Unit) that executes a program stored in the non-volatile memory 31b, and a volatile memory 31c used by the CPU 31a.

FIG. 6A shows the axes of the gyro sensor 32. FIG. FIG. 6 (b) is a view showing each axis of the acceleration sensor 33. FIG. 6C is a view for explaining normalization of the sensor output output from the gyro sensor 32.
As shown in FIG. 6A, the gyro sensor 32 detects angular velocities centered on the X axis, the Y axis, and the Z axis orthogonal to one another. As shown in FIG. 6B, the acceleration sensor 33 detects an acceleration for each of the X axis, the Y axis, and the Z axis orthogonal to each other.

The control unit 31 of the motion information output device 3 detects the angular velocity detected by the gyro sensor 32 and the acceleration sensor when the user is exercising on the rowing machine 6 (see FIGS. 3B and 3C). Based on the acceleration detected by 33, a sensor output that changes periodically as the user's wrist reciprocates is acquired.
For example, based on the sensor output of the acceleration sensor 33, the control unit 31 acquires the inclination of the housing included in the operation information output device 3 with respect to the gravity direction. Further, the control unit 31 obtains a sensor output by normalizing the angular velocity detected by the gyro sensor 32 with the reciprocating motion of the wrist according to the inclination of the housing 3a with respect to the gravity direction. For example, as shown in FIG. 6C, the control unit 31 selects the angular velocity of the two axes with a large amount of change from the angular velocity of the three axes output by the gyro sensor 32, and selects the angular velocity of the selected two axes. The sensor output is acquired by normalizing to one axis according to the inclination of the housing 3a. The control unit 31 transmits the acquired sensor output to the control device 2 via the communication interface 38.

FIG. 7 is a diagram showing the relationship between the sensor output NS from the gyro sensor 32 after normalization and the position of the wrist. The sensor output NS indicated by the alternate long and short dash line in FIG. 7 has a positive (+) value when the wrist moves in a direction away from the chest during exercise with the rowing machine 6, and the wrist approaches the chest It is negative (-) when moving. In addition, the sensor output NS indicates the value "0" when the movement of the wrist stops.
As can be seen from the position PW of the wrist shown by the solid line in FIG. 7, when the sensor output NS changes from a negative value to "0", the wrist is at the closest position B closest to the chest (see FIG. 3C). It can be said. Conversely, when the sensor output changes from a positive value to "0", it can be said that the wrist is the farthest separated position T (see FIG. 3B) from the chest. Thus, the sensor output NS changes its signal level periodically as the user's wrist reciprocates between the proximity position B and the separation position T.
In the example of FIG. 7, the position PW of the wrist leads the phase by about 1⁄4 cycle as compared with the sensor output NS. Therefore, based on the sensor output NS received from the motion information output device 3, the control unit 21 of the control device 2 obtains the position of the wrist (specific part) at the time of exercise on the rowing machine 6 (reciprocation of the wrist). can do.

<Playback Summary of Video Content>
FIG. 8 is a diagram showing the switching of the introductory scene, the interlocked scene, and the finish scene included in the video content.
In the video display system 1, the video content is played back by the user's instruction of playback. For example, in response to the user's operation of the reproduction related switch 36, reproduction of the video content is started. That is, the control unit 21 of the control device 2 starts the reproduction of the video content based on the operation information output by the reproduction related switch 36 when the operation is performed.
As shown in FIG. 8, the video content includes an introduction scene, an interlocking scene, and a finish scene. The introduction scene is a prologue scene. The interlocked scene is a scene to be reproduced interlockingly with the user's action (reciprocal movement of the wrist). The finish scene is an ending scene.
For example, when the video content is video content of a boat competition, the introduction scene is a scene from before the race to just before the start. Similarly, the interlocked scene is a scene from the start of the competition to the front of the goal, and the finish scene is a scene from the front of the goal to the rear of the goal.

The reproduction of the video content is started from the introduction scene. The reproduction of the introduction scene is performed regardless of the sensor output of the operation information output device 3. That is, the control unit 21 of the control device 2 reproduces the imaging frame data at a predetermined interval dt1 (see FIG. 10) and reproduces the audio data regardless of the sensor output of the operation information output device 3.
If the user gives an instruction to switch the scene during playback of the introduction scene, switching is made to playback of the interlocked scene. For example, in accordance with the operation of the first mode switching switch by the user, switching to playback of the interlocked scene is performed. In addition, even when the reproduction of the introduction scene is finished, it is possible to switch to the reproduction of the interlocked scene. In the interlocked scene, the control device 2 receives the sensor output from the operation information output device 3 and controls the reproduction of the interlocked scene based on the sensor output (operation information).
When the second mode switch 35 of the operation information output device 3 is operated by the user during the interlocked scene reproduction period, the control unit 21 of the control device 2 starts reproduction of the finish scene. The reproduction of the finish scene is performed regardless of the sensor output of the motion information output device 3, and the reproduction of the video content ends with the completion of the reproduction of the finish scene.

During exercise on the rowing machine 6, variations may occur in the time required for the user's wrist to reciprocate. By interlocking the reproduction of the video content in accordance with this variation, it is possible to give the user a feeling that oneself exists in the environment provided by the video content.
Therefore, the video display system 1 acquires a sensor output (a sensor output from the normalized gyro sensor 32) from the motion information output device 3 when reproducing the interlocked scene, and the interlocked scene is acquired according to the acquired sensor output. Control playback. In other words, the video display system 1 controls the reproduction of the video content in accordance with the cycle of the user's wrist reciprocation when using the rowing machine 6.
For example, five types of linked scenes are prepared: the first cycle to the fifth cycle, and immediately after switching to the linked scene, linked scenes having a specific cycle (for example, the first cycle) are reproduced. Thereafter, in accordance with the periodic change of the sensor output, the interlocked scene of the first period is sequentially switched to the interlocked scene of the appropriate period among the interlocked scenes of the fifth period. Further, even during the reproduction of the interlocked scene of one cycle, the reproduction of the interlocked scene is controlled in accordance with the reciprocating motion of the user's wrist. The details will be described below.

<Various data stored in storage unit 22>
First, various data stored in the storage unit 22 of the control device 2 will be described. FIG. 9A is a view showing the contents of shooting frame data.

<Photographed frame data>
As shown in FIG. 9A, the shooting frame data includes three types of shooting frame data of the introduction scene, shooting frame data of the interlocked scene, and shooting frame data of the finish scene.
As described above, in the introduction scene and the finish scene, the video content is reproduced regardless of the sensor output from the motion information output device 3. On the other hand, in the interlocked scene, the reproduction of the video content is controlled interlocked with the sensor output from the operation information output device 3.

The video display system 1 of the present embodiment includes five types from the interlocked scene of the first period to the interlocked scene of the fifth period, and switches the interlocked scene to be reproduced according to the cycle of the reciprocation of the wrist.
In the video content of the boat competition, the interlocking scene of the first cycle is selected when the cycle of the reciprocating movement of the wrist is the longest, and the interlocking scene of the fifth cycle is selected when the cycle of the reciprocating movement of the wrist is the shortest. Ru. For example, the linked scene of the first cycle is selected when the cycle of rowing all is the longest (the boat traveling speed is the slowest). The interlocking scene of the fifth period is selected when the period in which the row is over the all is the shortest (when the boat traveling speed is the fastest). Similarly, the interlocked scene in the second period to the interlocked scene in the fourth period are also selected according to the cycle length of the reciprocating movement of the wrist.

For example, the linked scene of the first cycle corresponds to the boat running scene in the case where the section from the start to the goal front is covered all at the longest repeat cycle, and the linked scene of the second cycle is from the start to the goal The run scene of the boat in the case where you go through the whole in the second longest repetition cycle corresponds to the section up to the section, the interlocking scene of the third cycle is the section from the start to the goal front in the third slow repetition cycle The traveling scene of the boat in the case of looking over the oar corresponds. Similarly, the linked scene of the fourth cycle corresponds to the traveling scene of the boat in the case where the section from the start to the front of the goal is covered in the second short repetition cycle, and the linked scene of the fifth cycle starts The run scene of the boat in the case where you go through all in the shortest repetition cycle corresponds to the section from the goal to the goal front.
In other words, it can be said that the interlocking scene of each cycle is a scene captured by changing the traveling speed of the boat for each cycle in the section from the start to the goal front. In other words, it can be said that the linked scenes in each cycle have the same content but different traveling speeds.
In the example shown in FIG. 9A, the code F (S) indicates a plurality of shooting frame data belonging to the introduced scene, and the code F (E) indicates a plurality of shooting frame data belonging to the finish scene. Similarly, reference signs F (F1) to F (F5) respectively indicate a plurality of shooting frame data belonging to the interlocked scenes of the first to fifth cycles.

<Voice data>
FIG. 9 (b) is a diagram showing the type of audio data. As shown in FIG. 9B, the audio data, like the shooting frame data, comprises three types of scenes: an introductory scene, an interlocking scene, and a finish scene. Each audio data is paired with corresponding shooting frame data. Therefore, the audio data of the interlocked scene includes five types of audio data from the interlocked scene of the first period to the interlocked period of the fifth period.
The audio data may be in any format as long as it can be handled by the control device 2. For example, it may be uncompressed PCM (Pulse Code Modulation) format or compressed MP3 (MPEG-1 Audio Layer-3) format.

<Metadata>
FIG. 9C is a diagram showing metadata added to shooting frame data and audio data. FIG. 9D is a diagram showing the relationship between the linked scene in each cycle and the round-trip frequency of the wrist.
In the present embodiment, as metadata, a range of the round-trip frequency of the wrist corresponding to each period (first period to fifth period) of the interlocked scene, and a plurality of shooting frame data F (F1 belonging to the interlocked scene of each period ) To F (F5), marker data to be added to specific imaging frame data, and start time data indicating start time for each of a plurality of reproduction audio data.

The reciprocating frequency of the wrist is the number of times the reciprocating motion of the wrist is repeated per unit time (for example, one minute). In the present embodiment, the control unit 21 of the control device 2 acquires the sensor output from the operation information output device 3.
As shown in FIG. 9C, the range of the round-trip frequency of the wrist is lower than the frequency H2 for the interlocked scene of the first period and higher than the frequency H1 for the interlocked scene of the second period, The frequency is H4 or less. Similarly, the frequency is higher than frequency H3 and lower than frequency H6 for the interlocked scene in the third period, and higher than frequency H5 and lower than frequency H8 for the interlocked scene in the fourth period, and the fifth period The frequency of H7 or higher is set for the interlocked scene of

As shown in FIG. 9D, the larger the ordinal number added after the code H in the frequency, the higher the frequency. Accordingly, the upper limit frequency H2 in the interlocked scene of the first period is higher than the lower limit frequency H1 in the interlocked scene of the second period, and the lower limit frequency H3 in the interlocked scene of the third period is the upper limit frequency H4 in the interlocked scene of the second period. Lower than.
Similarly, the upper limit frequency H6 in the interlocked scene of the third period is higher than the lower limit frequency H5 in the interlocked scene of the fourth period, and the lower limit frequency H7 in the interlocked scene of the fifth period is the upper limit frequency in the interlocked scene of the fourth period Lower than H8.

The marker data shown in FIG. 9C is added to specific imaging frame data. Specifically, the marker data is added to the imaging frame data that is the basis of the image displayed on the display device 4 at the timing when the user's wrist reaches the proximity position B or the separation position T. Therefore, at the timing when the user's wrist reaches the proximity position B or the separation position T, the reproduction of the video content is controlled so that the image is displayed based on the imaging frame data to which the marker data is added.
In the example shown in FIG. 9C, marker data B → T is added to the imaging frame data F (F1 M1), and marker data T → B is added to the imaging frame data F (F1 M2). Similarly, marker data B → T is added to the imaging frame data F (F1 M3), and marker data T → B is added to the imaging frame data F (F1 M4).
As described above, the shooting frame data F (F1) is a plurality of shooting frame data belonging to the interlocked scene of the first cycle. Here, the code M1 indicates the first marker data. Therefore, the imaging frame data F (F1M1) indicates imaging frame data to which the first marker data is added among the plurality of imaging frame data belonging to the interlocked scene of the first cycle. Similarly, shooting frame data F (F1M2) to shooting frame data F (F1 M4) are shooting frame data to which the second marker data is added among the plurality of shooting frame data belonging to the interlocked scene of the first cycle. 17 shows imaging frame data to which the second marker data is added.
The marker data B → T indicates that the wrist at the close position B moves toward the separation position T. Conversely, the marker data T → B indicates that the wrist at the separated position T moves toward the close position B. Therefore, the imaging frame data F (F1 M1) to the imaging frame data F (F1 M4) to which the respective marker data (B → T, T → B) are added are among the plurality of imaging frame data belonging to the interlocked scene of the first cycle. 14 shows imaging frame data selected when the user's wrist reaches the proximity position B or the separation position T.
As mentioned above, although imaging frame data of the interlocking scene of the 1st cycle was mentioned as an example and explained, marker data is similarly added to imaging frame data of the interlocking scene of other cycles.

<Relationship between marker data and shooting frame data>
FIG. 10 is a diagram showing the relationship between marker data and shooting frame data in each cycle (first to fifth cycles) of the interlocked scene.
As shown in FIG. 10, a plurality of shooting frame data belonging to the interlocked scene of the first cycle are shot at time intervals dt1, and the shooting frame data F (F1M1) to shooting frame data F (F1M5) are taken. Marker data (B → T, T → B) is added. A plurality of shooting frame data belonging to the interlocked scene of the second cycle are shot at time intervals dt1, and marker data (B → T,) for shooting frame data F (F2M1) to shooting frame data F (F2M5) T → B) is added.
Likewise, a plurality of shooting frame data belonging to the interlocked scene of the third cycle are shot at time intervals dt1, and marker data (frame data F (F3M1) to shooting frame data F (F3M5) are recorded. B → T, T → B) are added. A plurality of shooting frame data belonging to the interlocked scene of the fourth cycle are shot at time intervals dt1, and marker data (B → T,) for shooting frame data F (F4M1) to shooting frame data F (F4M5) T → B) is added. A plurality of shooting frame data belonging to the interlocked scene of the fifth period are shot at time intervals dt1, and marker data (B → T,) for shooting frame data F (F5M1) to shooting frame data F (F5M5) T → B) is added.
In the present embodiment, the time length TW in the interlocking scene of each cycle is common.

As described above, the linked scenes in each cycle have the same content. The marker data (B → T, T → B) is added to the imaging frame data at the timing when the user's wrist reaches the proximity position B or the separation position T. Therefore, if the ordinal numbers added after the code M are the same, it can be said that the contents of the photographed frame data are common even if the cycles are different.
For example, the imaging frame data F (F1 M1) belonging to the interlocked scene of the first cycle to which the first marker data (B to T) is added is the second to which the first marker data (B to T) is added. It can be said that the content is common to the shooting frame data F (F2M1) belonging to the linked scene of the cycle. It can be said that the content of the shooting frame data F (F2M1) is common to the shooting frame data F (F3M1) belonging to the interlocked scene of the third cycle.
Similarly, shooting frame data F (F1 M2) belonging to the linked scene of the first period to which the second marker data (T → B) is added is the same as the second to which the second marker data (T → B) is added. It can be said that the contents are common to the photographing frame data F (F3M2) belonging to the interlocked scene of three cycles and the photographing frame data F (F5M2) belonging to the interlocked scene of the fifth cycle.
In the present embodiment, switching of the cycle of the interlocked scene is performed between imaging frame data having the same ordinal number appended after the code M. By configuring in this manner, the sense of incongruity accompanying switching of the cycle of the interlocked scene is suppressed.

The linked scenes in each cycle have the same content in each cycle, but the traveling speed is different in each cycle. In addition, the time interval at the time of shooting between successive shooting frame data is common to the time interval dt1 in the interlocked scene of each cycle. Therefore, a shooting frame included between the shooting frame data to which the marker data (B → T or T → B) is added and the shooting frame data to which the next marker data (T → B or B → T) is added The number of data is different in each cycle of the interlocked scene.
For example, the number N 11 of shooting frame data F included between the shooting frame data F (F 1 M 1) to the shooting frame data F (F 1 M 2) in the linked scene of the first cycle is the shooting frame data F in the linked scene of the second cycle This number is larger than the number N21 of shooting frame data F included between F2M1) and shooting frame data F (F2M2).
Similarly, the number N32 of imaging frame data F included between imaging frame data F (F3M2) to imaging frame data F (F3 M3) in the interlocked scene of the third period is the imaging frame data F in the interlocked scene of the fourth period. This number is larger than the number N42 of shooting frame data F included between (F4M2) and shooting frame data F (F4M3).
Therefore, the imaging frame data in the interlocking cycle of each period is mutually including imaging frame data (one of the first imaging image information and the second imaging image information) to which the marker data (B → T) is added, and marker data (T It can be said that the number of other shooting frame data included between the shooting frame data (the first shooting image information and the other of the second shooting image information) to which B is added is different.

<Generation of audio data for playback>
Next, generation of audio data for reproduction in a linked scene of each cycle will be described.
As described above, in the interlocked scene, the reproduction of the video content is controlled in accordance with the cycle of the reciprocation of the user's wrist. That is, at the timing when the user's wrist reaches the proximity position B, the reproduction of the video content is controlled so that the image is displayed based on the imaging frame data corresponding to the marker data (B → T). Similarly, at timing when the user's wrist reaches the separation position T, reproduction of the video content is controlled so that an image is displayed based on the imaging frame data corresponding to the marker data (T → B).
When the above reproduction control is performed, if the audio data is reproduced as it is, there may be a disadvantage that a gap occurs between the image and the audio. In order to suppress this problem, the control unit 21 of the control device 2 performs reproduction control of sound based on the reproduction sound data.

FIG. 11A shows the type of audio data. FIG. 11 (b) is a diagram showing a generation procedure of reproduction audio data.
The storage unit 22 (see FIG. 4A) of the control device 2 stores audio data MF1-ALL to MF5-ALL of the interlocking scene of each cycle over the time length TW shown in FIG. 11A. There is.
The control unit 21 of the control device 2 refers to the combination of the audio file and the start time from the metadata shown in FIG. 9C, on condition that the elapsed time from the switching timing to the interlocked scene has reached the start time. , Generate audio data for reproduction. For example, when the interlocked scene of the first cycle is selected, the control unit 21 of the control device 2 for reproduction on condition that the elapsed time from the switching timing to the interlocked scene has reached the start time Mt1-001. Voice data MF1-001 is generated. Similarly, the control unit 21 of the control device 2 generates the audio data for reproduction MF1-002 on condition that the elapsed time from the switching timing to the interlocked scene has reached the start time Mt1-002.

The playback audio data MF1-XXX (XXX is a natural number of 3 digits, hereinafter the same) to be played back in the linked scene of the first cycle shown in FIG. 11 (b) to the playback voice played in the linked scene of the fifth cycle. The case of generating the data MF5-XXX will be described as an example.
When the interlocked scene of the first cycle is selected, the control unit 21 of the control device 2 sets the audio data MF1- on condition that the elapsed time from the switching timing to the interlocked scene has reached the start time Mt1-XXX. The voice data MF1-XXX 'is acquired by copying the portion of the prescribed time width Mtw from the start time Mt1-XXX in ALL. The prescribed time width Mtw may be, for example, 0.5 seconds to 1.5 seconds, but is not limited to these seconds. Next, the control unit 21 applies fade-in processing and fade-out processing to the copied audio data MF1-XXX ′ to generate reproduction audio data MF1-XXX.
The generation of the reproduction audio data MF1-XXX has been described above, but the reproduction audio data MF2-XXX to MF5-XXX related to other cycles can also be generated in the same procedure.

<Playback of audio data for playback>
The reproduction of the generated reproduction audio data MF1-XXX to MF5-XXX is started at the timing when the display based on the photographing frame data corresponding to the start time Mt1-XXX to Mt5-XXX is performed. For example, by dividing the start time Mt1-XXX by the time interval dt1, it is possible to calculate the number of imaging data frames F for starting the reproduction of the reproduction audio data MF1-XXX.
FIG. 12 (a) is a diagram showing the reproduction of reproduction audio data at standard time intervals. As shown in FIG. 12A, the linked scene in the first cycle is selected, and the playback audio data MF1-001 starts playback at the timing when display based on the shooting data frame F (F1L1) is performed, and playback is performed. It is assumed that the audio data MF1-002 starts to be reproduced at the timing when the display based on the shooting data frame F (F1L2) is performed. In the above case, since the fade-out of the audio data for reproduction MF1-001 and the fade-in of the audio data for reproduction MF1-002 overlap, the time for the audio data for reproduction MF1-001 and the audio data for reproduction MF1-002 is appropriate Played at intervals.

FIG. 12B is a diagram showing the reproduction of the audio data for reproduction at a time interval narrower than the standard time interval. In the case of FIG. 12B, although the reproduction start timing of the reproduction audio data MF1-002 starts earlier than the reference, since the reproduction audio data MF1-002 fades in, the reproduction audio data MF1 -Uncomfortable feeling with playback start can be suppressed. Similarly, since the reproduction audio data MF1-001 is faded out, it is possible to suppress the discomfort caused by the end of reproduction of the reproduction audio data MF1-001.
FIG. 12C is a diagram showing the reproduction of the audio data for reproduction at a time interval wider than the standard time interval. In the case of FIG. 12C, a silent section occurs between the end of reproduction of the reproduction audio data MF1-001 and the start of reproduction of the reproduction audio data MF1-002, but the reproduction audio data MF1-001 is faded out. After that, since the reproduction audio data MF1-002 is faded in, it is possible to suppress the discomfort when the reproduction is switched from the reproduction audio data MF1-001 to the reproduction audio data MF1-002.

<Playback control in interlocked scene>
As described above, at the timing when the user's wrist is in the proximity position B or the separation position T during the interlocking scene playback period, an image based on imaging frame data to which each marker data (B → T, T → B) is added Is displayed on the display device 4.
As described above, the timing at which the user's wrist reaches the proximity position B or the separation position T varies. Therefore, in order to smoothly reproduce the video content in conjunction with the reciprocating motion of the user's wrist, it is required to control the image display based on the photographing frame data in accordance with the reciprocating motion of the wrist. Therefore, in the present embodiment, the following control is performed in the interlocked scene.

The control unit 21 (prediction unit 21e) calculates, as a predicted time based on the sensor output, the movement time taken for the reciprocation of the specific part to interlock the reciprocation of the specific part and the captured image in the interlocked scene Do.
This predicted time may be calculated for the time of each reciprocation based on the sensor output applied to one direction in the reciprocation, or may be calculated in the moving direction based on the bi-directional sensor output in the reciprocation. It may be calculated. In this case, the prediction time is the average travel time of the reciprocating motion.
In this embodiment, the case of calculating the predicted time in one direction will be described.
The control unit 21 (prediction unit 21e) corresponds to the interlocked scene, and the movement time (predictive time) taken for the one-way movement of the specific part in the reciprocation is the same direction that was performed multiple times before this one-way movement. The movement is calculated based on the sensor output (operation information) output from the operation information output device 3 (operation information output unit) (described in detail below).
Also, the control unit 21 (time interval setting unit 21d) generates next marker data (B → T) from the imaging frame data (first imaging image information) to which the marker data (one of B → T and T → B) is added. The time interval up to the shooting frame data (second shooting image information) to which the other T → B) is added is set to the time interval based on the above-mentioned prediction time.
The control unit 21 (display image information generation unit 21 f) generates display image information by performing alpha blending processing on a pair of shooting frame data which are adjacent to each other in accordance with the timing defined by the display device 4.

<Procedure of generating display image information when moving the wrist from the separated position T to the close position B>
First, a procedure of generating display image information when the user's wrist moves from the separated position T to the close position B will be described. FIGS. 13 to 19 show a procedure of generating display image information corresponding to the first marker data F (F1 M1) in the interlocked scene of the first cycle. Although the first marker data F (F1M1) is described below, it is merely an example. The same procedure is performed for odd-numbered marker data such as the third marker data F (F1 M3) and the fifth marker data F (F1 M5).

<Calculation of predicted time>
FIG. 13 (a) is a diagram showing a calculation procedure of the prediction time.
The predicted time TS5 'shown in FIG. 13 is a predicted time until the sensor output corresponding to the imaging frame data F (F1 M1) is acquired from the latest sensor output acquisition timing ts5. In other words, the predicted time TS5 'is the movement time until the user's wrist moves from the remote position T to the near position B, which is predicted at the timing ts5.
In the example shown in FIG. 13A, the user's wrist is located at the separated position T at timings ts1, ts3 and ts5 and at the proximity position B at timings ts2 and ts4. The control unit 21 of the control device 2 recognizes the respective timings ts 1 to ts 5 based on the sensor output from the operation information output device 3.

As shown in FIG. 13A, a preparation period is set immediately after switching to the interlocked scene. During the preparation period, reproduction of the video content interlocked with the operation of the user is not performed, and the photographing frame data is displayed at predetermined intervals dt1. The preparation period continues, for example, over a predetermined time length or until predetermined imaging frame data is displayed.
In the preparation period, preparation for interlocking control is performed. For example, the control unit 21 acquires the moving time TS1 required to move from the separated position T to the close position B at timing ts2, and the moving time required to move from the close position B to the separated position T at timing ts3. Get TS2. Similarly, the control unit 21 acquires the moving time TS3 required for moving from the separated position T to the near position B at timing ts4, and the movement required for moving from the near position B to the separated position T at timing ts5. Get time TS4.

When the preparation period ends, it shifts to an interlocking period. First, the control unit 21 of the control device 2 obtains the predicted time TS5 '. The prediction time TS5 'relates to one-way movement in which the user moves from the separation position T to the near position B in one-way movement in which the user's wrist moves from one of the close position B to the other position T toward the other. It is predicted time.
The control unit 21 acquires the predicted time TS5 'based on the one-way motion from the separation position T to the proximity position B, which is performed before the timing ts5 at which the latest sensor output is acquired. Specifically, the control unit 21 obtains an average of the time TS1 required from the timing ts1 to the timing ts2 and the time TS3 required for the timing ts3 to the timing ts4 as the prediction time TS5 '.

<Variations in prediction time>
FIG. 13 (b) is a diagram showing the variation of the prediction time TS 5 ′. As shown in FIG. 13 (b), variations may occur in the predicted time TS5 '. This is because the respective timings ts1 to ts5 are acquired along with the reciprocating motion of the user's wrist. With the variation of the prediction time TS5 ', a variation may occur as in the timings ts6a' to 6c 'also for the timing ts6' when the prediction time TS5 'has passed from the timing ts5.
As shown in FIG. 13A, since the timing ts6 'corresponds to the timing at which the imaging frame data F (F1 M1) should be displayed, the imaging frame is generated from the imaging frame data F (F1 M0) displayed at the timing ts5. The time interval between the plurality of shooting frame data included in the data F (F1M1) may be deviated from the time interval dt1 between the plurality of shooting frame data in the preparation period.

In the case of timing ts6b 'shown in FIG. 13B, the time interval between shooting frame data from shooting frame data F (F1 M0) to shooting frame data F (F1 M1) is the time between multiple shooting frame data in the preparation period. It is equal to the interval dt1.
On the other hand, in the case of timing ts6a ', the time interval between the shooting frame data from the shooting frame data F (F1 M0) to the shooting frame data F (F1 M1) is shorter than the time interval dt1 between the plurality of shooting frame data in the preparation period Become.
Conversely, in the case of timing ts6c ', the time interval between the shooting frame data from the shooting frame data F (F1 M0) to the shooting frame data F (F1 M1) is greater than the time interval d1 between the plurality of shooting frame data in the preparation period. become longer.

<Adjustment of time interval between shooting frame data>
FIG. 14 is a diagram showing the time interval of the imaging frame data and the interval adjustment curve SP1 at the predicted time TS5 '. In the example shown in FIG. 14, the plurality of shooting frame data from the shooting frame data F (F1 M0) to the shooting frame data F (F1 M1) are defined at equal time intervals dt2. As a result, even if the time of reciprocation of the wrist fluctuates, it is possible to follow and reproduce the video content.
In the present embodiment, the time interval between the imaging frame data is adjusted by the interval adjustment curve SP1 so that the image is displayed smoothly at the control switching timing (for example, timing ts5). The interval adjustment curve SP1 acquires a frame rate (the number of frame data per unit time) at each of the timing at which the latest sensor output is acquired and the timing defined by the prediction time, and spline interpolation (for example, cubic spline interpolation) It is acquired by doing.

In the example illustrated in FIG. 14, the control unit 21 of the control device 2 calculates the effective frame rate V0 based on the number of imaging frame data actually displayed by the timing ts5 and the elapsed time until the timing ts5. . Further, the control unit 21 determines the frame rate based on the number of shooting frame data included in the shooting frame data F (F1 M0) to the shooting frame data F (F1 M1) and the predicted time from the timing ts5 to the timing ts6 '. Calculate V1 '. Furthermore, the control unit 21 spline-interpolates the effective frame rate V0 and the frame rate V1 'to acquire the interval adjustment curve SP1.

FIG. 15 is a diagram showing a time interval between imaging frame data adjusted by the interval adjustment curve SP1. By using the interval adjustment curve SP1 illustrated in FIG. 15, the interval between the imaging frame data F (F1M0 + 2) and the imaging frame data F (F1M0 + 3) is adjusted from the time interval dt2 to the time interval dt2a. Similarly, the interval between the imaging frame data F (F1 M0 + n) and the imaging frame data F (F 1 M 0 + m) is adjusted from the time interval dt2 to the time interval dt 2 b, and the imaging frame data F (F1 M1-3) and the imaging frame data F (F1 M1) Between -2), the time interval dt2 is adjusted to the time interval dt2c.
Here, the time interval dt2b is shorter than the time interval dt2a, and the time interval dt2c is shorter than the time interval dt2b. Therefore, it can be said that the interval adjustment curve SP1 adjusts the time interval between the imaging frame data in a stepwise manner toward the imaging frame data F (F1 M1).

FIG. 16 is a diagram showing a time interval between shooting frame data adjusted by another interval adjustment curve SP2. By using the interval adjustment curve SP2 illustrated in FIG. 16, the interval between the imaging frame data F (F1M0 + 2) and the imaging frame data F (F1M0 + 3) is adjusted from the time interval dt2 to the time interval dt2d. Similarly, the interval between the imaging frame data F (F1 M0 + n) and the imaging frame data F (F 1 M 0 + m) is adjusted from the time interval dt2 to the time interval dt2 e, and the imaging frame data F (F1 M1-3) and the imaging frame data F (F1 M1) Between -2), the time interval dt2 is adjusted to the time interval dt2f.
Here, the time interval dt2e is longer than the time interval dt2d, and the time interval dt2f is longer than the time interval dt2e. Therefore, it can be said that the interval adjustment curve SP2 adjusts the time interval between the imaging frame data in a stepwise manner toward the imaging frame data F (F1 M1).

<Generation of Composite Frame Data F (mix 1) to F (mix 4)>
When the time interval between the photographing frame data is adjusted by the above processing, synchronization with the refresh timing on the display device 4 can not be achieved, and as a result, there is a possibility that an image lacking in smoothness is displayed on the display device 4 is there.
In the present embodiment, in accordance with the refresh timing defined by the display device 4, the pair of shooting frame data which precede and follow each other is combined (alpha blend processing), and the combined frame obtained is written in the frame buffer 23. The combining process will be described below.

FIG. 17 is a diagram showing generation of combined frame data F (mix 1) to F (mix 4). FIG. 18A is a diagram showing the generation of combined frame data F (mix 1). FIG. 18B is a diagram showing generation of combined frame data F (mix 2). FIG. 18C is a diagram showing generation of combined frame data F (mix 3). FIG. 18D is a diagram showing generation of combined frame data F (mix 4).
As shown in FIG. 17, the display device 4 performs refresh at refresh timings Rf1 to Rf4. The refresh timings Rf1 to Rf4 arrive at fixed time intervals dRf. The time interval dRf is defined by the refresh rate.

As shown in FIG. 17, at the refresh timing Rf1, the control unit 21 of the control device 2 controls the shooting frame data F (F1 M0) before the refresh timing Rf1 and the shooting frame data F (F1 M0 + 1) after the refresh timing Rf1. To generate combined frame data F (mix 1). The composite frame data F (mix1) is composited by alpha blending processing using a division ratio of a time interval between shooting frame data at the refresh timing Rf1 as a coefficient.
As shown in FIG. 18A, in the combined frame data F (mix 1), the ratio of the shooting frame data F (F 1 M 0) is [(1−α 1) / dt 2], and the ratio of the shooting frame data F (F 1 M 0 + 1) is It is synthesized to be [α1 / dt2].

As shown in FIG. 17, at the refresh timing Rf2, the control unit 21 of the control device 2 sets the shooting frame data F (F1 M0) before the refresh timing Rf2 and the shooting frame data F (F1 M0 + 1) after the refresh timing Rf2. To generate combined frame data F (mix 2). The composite frame data F (mix 2) is composited by alpha blending processing using the division ratio of the time interval between shooting frame data at the refresh timing Rf 2 as a coefficient.
As shown in FIG. 18B, in the combined frame data F (mix 2), the ratio of the shooting frame data F (F 1 M 0) is [(1−β 1) / dt 2] and the ratio of the shooting frame data F (F 1 M 0 + 1) is It is synthesized to be [β1 / dt2].

As shown in FIG. 17, at the refresh timing Rf3, the control unit 21 of the control device 2 sets the shooting frame data F (F1M0 + 1) before the refresh timing Rf3 and the shooting frame data F (F1M0 + 2) after the refresh timing Rf3. To generate combined frame data F (mix 3). The composite frame data F (mix 3) is composited by alpha blending processing using a division ratio of a time interval between shooting frame data at the refresh timing Rf 3 as a coefficient.
As shown in FIG. 18C, in the combined frame data F (mix 3), the ratio of the shooting frame data F (F1M0 + 1) is [(1-γ1) / dt2], and the ratio of the shooting frame data F (F1M0 + 2) is It is synthesized to be [γ1 / dt2].

As shown in FIG. 17, at the refresh timing Rf4, the control unit 21 of the control device 2 sets the shooting frame data F (F1M0 + 1) before the refresh timing Rf4 and the shooting frame data F (F1M0 + 2) after the refresh timing Rf4. To generate combined frame data F (mix 4). The composite frame data F (mix 4) is composited by alpha blending processing using a division ratio of a time interval between shooting frame data at the refresh timing Rf 4 as a coefficient.
As shown in FIG. 18D, in the combined frame data F (mix 4), the ratio of the shooting frame data F (F1M0 + 1) is [(1-.delta.1) / dt2], and the ratio of the shooting frame data F (F1M0 + 2) is It is synthesized so as to be [δ1 / dt2].
By performing the above combining process, combined frame data F (mix 1) to F (mix 4) for each refresh timing can be generated, and a smooth image can be displayed on the display device 4.

<Procedure of generation of display image information at the time of movement of the wrist from the proximity position B to the separation position T>
Next, a procedure for generating display image information when the user's wrist moves from the proximity position B to the separation position T will be described. FIGS. 19 to 22 show a procedure of generating display image information corresponding to the second marker data F (F1M2) in the interlocked scene of the first cycle. Although the second marker data F (F1M2) is described below, it is merely an example. The same procedure is performed for even-numbered marker data such as the fourth marker data F (F1 M4) and the sixth marker data F (F1 M6).

<Calculation of predicted time>
FIG. 19A is a diagram showing a calculation procedure of the prediction time.
The predicted time TS6 ′ shown in FIG. 19A is a predicted time from when the latest sensor output acquisition timing ts5 until the sensor output corresponding to the imaging frame data F (F1M2) is acquired.
In the example shown in FIG. 19A, the wrist of the user is located at the proximity position B at timing ts6 after the example described in FIG. Along with this, the control unit 21 of the control device 2 recognizes the timing ts6 based on the sensor output from the operation information output device 3.

As illustrated in FIG. 19A, after recognizing the timing ts6, the control unit 21 of the control device 2 acquires the predicted time TS6 ′. The prediction time TS6 ′ relates to one-way movement in which the user moves from the proximity position B to the separation position T in one-way movement in which the user's wrist moves from one of the close position B to the separation position T toward the other. It is predicted time.
The control unit 21 acquires the predicted time TS6 ′ based on the one-way motion from the proximity position B to the separated position T, which is performed before the timing ts6 at which the latest sensor output is acquired. Specifically, the control unit 21 obtains an average of the time TS2 required from the timing ts2 to the timing ts3 and the time TS4 required from the timing ts4 to the timing ts5 as the prediction time TS6 '.

<Variations in prediction time>
FIG. 19 (b) is a diagram showing the variation of the prediction time. As shown in FIG. 19 (b), variations may occur in the prediction time TS6 '. With the variation of the prediction time TS6 ′, a variation may occur as in the timings ts7a ′ to 7c ′ also at the timing ts7 ′ where the prediction time TS6 ′ has elapsed from the timing ts6.

<Adjustment of time interval between shooting frame data>
FIG. 20 is a diagram showing the time interval of the shooting frame data and the interval adjustment curve SP3 at the prediction time TS6 '. In the example shown in FIG. 20, a plurality of pieces of shooting frame data from the shooting frame data F (F1 M1) to the shooting frame data F (F1 M2) are defined at equal time intervals dt3.
Also in this example, the time interval between the imaging frame data is adjusted by the interval adjustment curve SP3 so that the image is displayed smoothly at the control switching timing (for example, timing ts6). The interval adjustment curve SP3 is created by spline interpolation (for example, cubic spline interpolation) between the effective frame rate V1 and the frame rate V2 '. The space adjustment curve SP3 is equivalent to the space adjustment curve SP1 described above. The adjustment of the time interval between the imaging frame data using the interval adjustment curve SP3 is also performed similarly to the adjustment of the time interval between the imaging frame data using the interval adjustment curve SP1 described above. Therefore, the description is omitted.

<Generation of Combined Frame Data F (mix 11) to F (mix 14)>
FIG. 21 is a diagram showing the generation of combined frame data F (mix 11) to F (mix 14). FIG. 22A shows generation of combined frame data F (mix 11). FIG. 22B is a diagram showing generation of combined frame data F (mix 12). FIG. 22C shows the generation of combined frame data F (mix 13). FIG. 22D is a diagram showing the generation of combined frame data F (mix 14).
As shown in FIG. 21, the display device 4 performs refresh at refresh timings Rf11 to Rf14. The refresh timings Rf11 to Rf14 arrive at fixed time intervals dRf defined by the refresh rate.

At the refresh timing Rf11, the control unit 21 of the control device 2 uses the photographed frame data F (F1 M1) before the refresh timing Rf11 and the photographed frame data F (F1 M1 + 1) after the refresh timing Rf11 to synthesize frame data. Generate F (mix 11). The combined frame data F (mix 11) is combined by alpha blending processing using a division ratio of a time interval between shooting frame data at the refresh timing Rf 11 as a coefficient.
As shown in FIG. 22A, in the combined frame data F (mix 11), the ratio of the shooting frame data F (F 1 M 1) is [(1−α 2) / dt 3], and the ratio of the shooting frame data F (F 1 M 1 + 1) is It is synthesized to be [α 2 / dt 3].

As shown in FIG. 21, at the refresh timing Rf12, the control unit 21 of the control device 2 controls the imaging frame data F (F1 M1) before the refresh timing Rf12 and the imaging frame data F (F1 M1 + 1) after the refresh timing Rf12. To generate combined frame data F (mix 12). The composite frame data F (mix 12) is composited by alpha blending processing using the division ratio of the time interval between the captured frame data at the refresh timing Rf 12 as a coefficient.
As shown in FIG. 22B, in the combined frame data F (mix 12), the ratio of the shooting frame data F (F 1 M 1) is [(1−β 2) / dt 3], and the ratio of the shooting frame data F (F 1 M 1 + 1) is It is synthesized to be [β2 / dt3].

As shown in FIG. 21, at the refresh timing Rf13, the control unit 21 of the control device 2 controls the shooting frame data F (F1M1) before the refresh timing Rf13 and the shooting frame data F (F1M1 + 1) after the refresh timing Rf13. To generate combined frame data F (mix 13). The combined frame data F (mix 13) is combined by alpha blending processing using a division ratio of a time interval between shooting frame data at the refresh timing Rf 13 as a coefficient.
As shown in FIG. 22C, in the combined frame data F (mix 13), the ratio of the shooting frame data F (F 1 M 1) is [(1-γ 2) / dt 3], and the ratio of the shooting frame data F (F 1 M 1 + 1) is It is synthesized so as to be [γ2 / dt3].

As shown in FIG. 21, at the refresh timing Rf14, the control unit 21 of the control device 2 controls the shooting frame data F (F1M1 + 1) before the refresh timing Rf14 and the shooting frame data F (F1M1 + 2) after the refresh timing Rf14. To generate combined frame data F (mix 14). The combined frame data F (mix 14) is combined by alpha blending processing using a division ratio of a time interval between shooting frame data at the refresh timing Rf 14 as a coefficient.
As shown in FIG. 22D, in the combined frame data F (mix 14), the ratio of the shooting frame data F (F1M1 + 1) is [(1−δ2) / dt3], and the ratio of the shooting frame data F (F1M1 + 2) is It synthesizes so that it may become [δ2 / dt3].

By performing the above combining process, combined frame data F (mix 11) to F (mix 14) for each refresh timing can be generated, and a smooth image can be displayed on the display device 4.

<Flow of control by CPU 21a of control device 2>
Next, the flow of control by the CPU 21 a of the control device 2 will be described.
FIG. 23A is a flowchart showing a signal monitoring process performed by the CPU 21a. FIG. 23B is a flowchart showing main processing performed by the CPU 21a.
<Signal monitoring process>
In the signal monitoring process shown in FIG. 23A, the CPU 21a determines whether or not a predetermined data acquisition timing has come (step S1). If it is determined that the data acquisition timing has arrived, the CPU 21a acquires various data from the operation information output device 3 (step S2). For example, the sensor output from the operation information output device 3 is acquired, or the operation information for the reproduction related switch 36 is acquired. After acquiring various data in step S2, the CPU 21a returns to step S1 and determines whether the data acquisition timing has arrived.

<Main processing>
In the main processing shown in FIG. 23B, the CPU 21a determines whether or not the reproduction operation of the video content has been performed by the reproduction related switch 36 (step S11). This determination is performed based on the data acquired in step S2 of the signal monitoring process, specifically, the operation information for the reproduction related switch 36.
When there is a reproduction operation, the CPU 21a starts reproduction of the introduction scene in the video content (step S12). The CPU 21a reads the imaging frame data of the introduction scene described in FIG. 9A and stores the imaging frame data as display frame data in the frame data. The display frame data stored in the frame data is transmitted to the display device 4 each time the refresh timing comes. The display device 4 displays an image according to the display frame data.
Further, the CPU 21a reads the audio data of the introduction scene described in FIG. 9B, generates an acoustic signal based on the audio data, and outputs the acoustic signal to the sound output device 5. When the sound signal is input to the sound output device 5, the sound output device 5 outputs a sound according to the sound signal.

When it is determined in step S11 that the reproduction operation of the video content is not performed, or when the reproduction of the introduction scene is started in step S12, the CPU 21a determines whether or not the introduction scene is being reproduced (step S13). . If it is determined that the introduction scene is being reproduced, the CPU 21a determines whether or not the first mode switch 34 has been operated (step S14). This determination is performed based on the data acquired in step S2 of the signal monitoring process, specifically, the operation information for the first mode switch 34.
When the first mode switch 34 is operated, the CPU 21a performs interlocked scene reproduction processing (step S15). The interlocked scene reproduction processing will be described later.

If there is no operation on the first mode changeover switch 34, the CPU 21a determines whether or not the reproduction of the introduced scene has ended (step S16). If it is determined that the introduction scene has been reproduced, the CPU 21a performs interlocked scene reproduction processing (step S15).
If it is determined in step S13 that the introduction scene is not being reproduced, the CPU 21a determines whether or not the interlocked scene is being reproduced (step S17). If it is determined that the interlocked scene is being reproduced, the CPU 21a performs interlocked scene reproduction processing (step S15).
If it is determined in step S16 that the reproduction of the introduced scene is not completed, and if it is determined in step S17 that the interlocked scene is not being reproduced, the CPU 21a returns to the process of step S11.

When returning from the linked scene reproduction process of step S15, the CPU 21a determines whether or not the second mode switch 35 has been operated (step S18). This determination is performed based on the data acquired in step S2 of the signal monitoring process, specifically, the operation information for the second mode switch 35.
When the second mode switch 35 is operated, the CPU 21a performs a finish scene reproduction process (step S19). On the other hand, when there is no operation on the second mode changeover switch 35, the CPU 21a returns to the process of step S11.
In the finish scene reproduction process, the CPU 21a reads the photographing frame data of the finish scene described in FIG. 9A, and stores it in the frame data as display frame data. The display device 4 displays an image according to the display frame data. Further, the CPU 21a reads the sound data of the finish scene described in FIG. 9 (b), generates an acoustic signal based on the sound data, and outputs the sound signal to the sound output device 5. When the sound signal is input to the sound output device 5, the sound output device 5 outputs a sound according to the sound signal. This finish scene reproduction process is performed until the reproduction of a series of finish scenes is completed.

<Interlocking scene reproduction processing>
In the interlocked scene reproduction processing shown in FIG. 24, the CPU 21a determines whether the cycle of the interlocked scene is not set (step S21). In other words, the CPU 21a determines whether it is immediately after the transition from the introduction scene to the interlocking scene. When the cycle of the interlocked scene is not set (immediately after the transition from the introduction scene to the interlocked scene), the CPU 21a (the photographed image information selection unit 2e) sets an initial cycle (step S22). In the present embodiment, the CPU 21a sets a first period as an initial period. Along with this, reproduction of the interlocked scene of the first cycle is started.
Next, the CPU 21a determines whether the sensor output from the operation information output device 3 has the value "0" (step S22). In other words, the CPU 21a determines whether the wrist of the user is at either the close position B or the separated position T. If the CPU 21a determines that the sensor output is the value "0", the process proceeds to step S24. If the CPU 21a determines that the sensor output is not the value "0", the process proceeds to step S38.

In the process of step S24, the CPU 21a determines whether the sensor output has changed from a negative value ("-") to a value "0". In other words, the CPU 21a determines whether or not the wrist of the user who has moved in a direction approaching the chest of the user has stopped. If the sensor output changes from a negative value to a value “0”, the CPU 21a determines that the user's wrist is located at the close position (step S25). In other words, the CPU 21a (image selection unit 2a) determines that the display timing of the imaging frame data to which the marker data (B → T) is added has come.
If it is determined in the process of step S24 that the sensor output does not change from a negative value to the value "0", the CPU 21a determines whether the sensor output changes from a positive value ("+") to a value "0". Is determined (step S26). In other words, the CPU 21a determines whether or not the wrist of the user who has moved in a direction away from the user's chest has stopped. When the sensor output changes from a positive value to the value "0", the CPU 21a determines that the user's wrist is located at the separated position (step S27). In other words, the CPU 21a (image selection unit 2a) determines that the display timing of the imaging frame data to which the marker data (T → B) is added has come.
In the process of step S26, when it is determined that the sensor output does not become the value "0" from the positive value (when the sensor output of the value "0" was the previous process, too), the CPU 21a has the value "0". It is determined whether or not the sensor output of has continued a prescribed number of times (step S28). If it is determined that the sensor output of the value “0” has continued for a prescribed number of times, the CPU 21a determines that the wrist of the user is at rest (step S29). If it is determined that the sensor output of the value “0” is not continuous the prescribed number of times, the CPU 21a proceeds to the process of step S38.

After executing the processing of step S25, step S27 or step S29, the CPU 21a determines whether or not it is necessary to change the cycle in the interlocked scene (step S30). As described in FIG. 9D, the CPU 21a acquires the reciprocating frequency of the wrist based on the sensor signal from the operation information output device 3, and selects the interlocked scene of the cycle suitable for the acquired reciprocating frequency.
If the cycle of the selected linked scene is different from the current linked scene cycle, the CPU 21a determines that the linked scene cycle change is necessary, and if both cycles are the same, the linked scene cycle change is not necessary. judge.
The CPU 21a proceeds to the process of step S34 when determining that the cycle change of the interlocked scene is necessary, and proceeds to the process of step S31 when determining that the cycle change of the interlocked scene is not necessary.

In the process of step S31, the CPU 21a (prediction unit 2c) predicts the next sensor output timing. For example, as described in FIG. 13A and FIG. 19A, the CPU 21a acquires the prediction time based on the one-way movement of the wrist performed before the latest sensor output timing, and By adding the prediction time to the sensor output timing, the next sensor output timing is predicted.

After executing the processing of step S31, the CPU 21a (time interval setting unit 2b) sets the time interval between shooting frame data from the latest sensor output timing to the next sensor timing as the time interval based on the predicted time. (Step S32).
For example, as described in FIG. 14, the CPU 21a selects the imaging frame data F (F1 M0) at timing ts5, selects the imaging frame data F (F1 M1) at timing ts 6 ', and belongs between both imaging frame data A time interval dt2 is set between a plurality of shooting frame data.
Similarly, as described in FIG. 20, the CPU 21a selects the photographing frame data F (F1M1) at timing ts6, and selects the photographing frame data F (F1M2) at timing ts7 ′, and between the two photographing frame data A time interval dt3 is set between a plurality of pieces of shooting frame data to which the image belongs.

After executing the process of step S32, the CPU 21a (time interval setting unit 2b) sets an interval adjustment curve (step S33). For example, the interval adjustment curve SP1 described in FIG. 14, the interval adjustment curve SP2 described in FIG. 16, or the interval adjustment curve SP3 described in FIG. 20 is set. By setting the interval adjustment curve, the time interval between the imaging frame data is changed from the timing when the user's wrist is located at one of the close position B and the distant position T toward the timing when it reaches the other. Can.

In the process of step S34, the CPU 21a (captured image information selection unit 2e) predicts the next sensor output timing. For example, as described in FIG. 13A and FIG. 19A, the CPU 21a acquires the prediction time based on the one-way movement of the wrist performed before the latest sensor output timing, and By adding the prediction time to the sensor output timing, the next sensor output timing is predicted.
After executing the process of step S34, the CPU 21a (prediction unit 2c) predicts the next sensor output timing (step S35). After executing the processing of step S35, the CPU 21a (time interval setting unit 2b) sets the time interval between shooting frame data from the latest sensor output timing to the next sensor timing as the time interval based on the predicted time. (Step S36). The processes of step S35 and step S36 are the same as the processes of step S31 and step S32 described above, and thus the description thereof is omitted.

After executing the process of step S36, the CPU 21a sets a composite curve (step S37). The composite curve is a curve that defines a composite ratio of shooting frame data belonging to the linked scene of the current cycle and shooting frame data belonging to the linked scene of the next cycle. The CPU 21a combines both shooting frame data at the combining ratio defined by the combining curve in combining frame generation processing (step S39) described later, and generates combining frame data. As a result, when switching the cycle of the interlocked scene, the image can be changed smoothly.

When it is determined that the sensor output is not the value "0" in the process of step S23, the process of step S33 is performed when it is determined that the sensor output of the value "0" is not continuous the specified number of times in the process of step S28. After or after executing the process of step S37, the CPU 21a determines whether the refresh timing of the display device 4 has come (step S38). When the refresh timing arrives, the CPU 21a (display image information generation unit 21f) generates composite frame data by performing alpha blending processing on the pair of shooting frame data, and writes the composite frame data as display frame data in the frame buffer 23 (step S39). .
The display frame data (combined frame data) written in the frame buffer 23 is output to the display device 4 at the refresh timing. Thereby, the display device 4 displays an image based on the display frame data.
If it is determined in step S38 that the refresh timing is not reached, and after the process of step S39 is executed, the CPU 21a returns to the main process.

By performing the above processing, in the video display system 1 according to the present embodiment, it is possible to reproduce the interlocked scene in interlock with the reciprocation movement of the user's wrist. The types of exercise to be targeted can be expanded as compared with the conventional system that was directed to the exercise that continues traveling in one direction.
Further, in the image display system 1 according to the present embodiment, the first timing when the user's wrist is located at one of the close position B and the distant position T and the prediction time have elapsed from the first timing. The photographing frame data is selected at each of the second timing when the other reaches the other of the proximity position B and the separation position T, and the time interval between the plurality of photographing frame data belonging to the selected photographing frame data is Since the time interval is set based on the video content, it is possible to follow and reproduce the video content even if the time of reciprocating movement of the wrist varies.
Further, in the image display system 1 of the present embodiment, by using the interval adjustment curves SP1 to SP3, the time interval between the imaging frame data is changed along with going from the first timing to the second timing. Therefore, even when the selected set of shooting frame data is updated, the image can be smoothly changed. For example, from the set of shooting frame data F (F1 M0) and shooting frame data F (F1 M1) shown in FIG. 13 (a), shooting frame data F (F 1 M1) and shooting frame data F (F 1 M2) shown in FIG. Even in the case where the image is updated to the above, the image displayed on the display device 4 can be changed smoothly.
In addition, at each refresh timing of the display device 4, alpha-blend processing is performed on a pair of consecutive shooting frame data in succession to generate composite frame data, and the generated composite frame data is displayed on the display device 4 as display frame data. Because of this, the image displayed on the display device 4 can be smoothly changed also at this point.

<Modification>
In the above embodiment, after adjusting the time interval of the imaging frame data by the interval adjustment curves SP1 to SP3, according to the refresh timing, combining processing of a pair of imaging frame data which precedes and follows is performed, but it is limited to this configuration It is not something to be done.
For example, when using the display device 4 corresponding to the variable refresh rate, as shown in FIG. 25, the frame as the display frame data which causes the display device 4 to display shooting frame data after adjusting the time interval between shooting frame data It may be stored in the buffer 23.
In the example of FIG. 25, shooting frame data F (F1 M0) is stored in the frame buffer 23 at timing Rf20, shooting frame data F (F1 M0 + 1) is stored in the frame buffer 23 at timing Rf21, and shooting frame data F (F F1M0 + 2) is stored in the frame buffer 23.
Although the time interval from the timing Rf20 to the timing Rf21 is different from the time interval from the timing Rf21 to the timing Rf22, since the display device 4 corresponds to the variable refresh rate, the image is appropriately displayed. Can.

With regard to reproduction control of interlocked scenes, although the interlocked scene of the first period is reproduced first in the above embodiment, the interlocked scene to be reproduced first is not limited to the interlocked scene of the first period. For example, a linked scene of the second cycle may be used, or a linked scene of the third cycle may be used.

In the above embodiment, interlocking control is performed on a specific one-way motion related to the reciprocating motion. Specifically, the interlocking control is performed by the movement time of the wrist from the close position B to the separation position T and the movement time of the wrist from the separation position T to the close position B. However, control of interlocking is not limited to these. For example, control may be performed based on the movement time obtained by averaging the movement time of the wrist from the close position B to the separation position T and the movement time of the wrist from the separation position T to the close position B. Further, control of interlocking may be performed on the basis of reciprocating motion. Specifically, the interlock control may be performed based on the movement time of the wrist from the proximity position B through the separation position T to the proximity position B. Similarly, interlock control may be performed by the movement time of the wrist from the separated position T to the separated position T through the close position B.

Although the control apparatus 2 provided with CPU21a, RAM21b, and the frame buffer 23 was illustrated in the above-mentioned embodiment, it is not limited to this structure. For example, hardware logic may execute all or part of the processing executed by the CPU 21a. Further, regarding the control device 2, a plurality of CPUs 21a may be mounted, or a plurality of processors of different architectures such as the CPU 21a and a GPU (Graphics Processing Unit) may be mounted. Furthermore, the process performed using the frame buffer 23 may be performed using the RAM 21b, and conversely, the process performed using the RAM 21b may be performed using the frame buffer 23.
Although the control device 2, the operation information output device 3, the display device 4, and the sound output device 5 are separately provided in the above embodiment, the present invention is not limited to this configuration. For example, the control device 2 may be incorporated in the operation information output device 3 and integrated, or the control device 2 may be integrated in the display device 4. In addition, the control device 2, the display device 4, and the sound output device 5 may be integrated.
In the above-mentioned embodiment, although the composition which processes by one control device 2 was illustrated, it is not limited to this composition. For example, the configuration may be such that processing is performed by a plurality of control devices 2 communicably connected via a communication network.

Although the motion information output device 3 is illustrated as being worn on the wrist in the above-described embodiment, the motion information output device 3 is not limited to being worn on the wrist. The motion information output device 3 may be attached to a portion capable of reciprocating movement, such as the user's ankle, head or waist.
In the above-mentioned embodiment, although operation information output device 3 was provided with control part 31, it is not limited to this composition. For example, the processing performed by the control unit 31 may be performed by hardware logic. Further, the control unit 31 of the operation information output device 3 may be omitted, and the operation information output device 3 may be controlled by the control unit 21 of the control device 2.
In the above embodiment, the sensor output of the gyro sensor 32 included in the operation information output device 3 is exemplified by performing normalization processing by the control unit 31 included in the operation information output device 3. However, the present invention is not limited to this configuration. For example, the sensor output normalization process may be performed by the control unit 21 of the control device 2.
Also, instead of the motion information output device 3, a video camera for capturing the motion of the user may be used, and the motion of the user may be extracted as motion information from the video obtained by the video camera.

In the above-mentioned embodiment, although an image processing system suitable for exercise in the rowing machine 6 has been described, the present invention is not limited to this configuration. For example, the present invention may be applied to a sperm collection support system.
The sperm collection support system is a system that supports the collection of sperm when collecting male sperm for medical research and therapeutic needs. This sperm collection support system is used to collect sperm for investigating the cause of infertility between husband and wife, to treat sexual dysfunction, and to acquire sperm for artificial insemination. Furthermore, the sperm collection support system can meet various social needs such as prevention of sexual crimes by eliminating personal sexual desires, prevention of prostitution and reduction of the number of sexually transmitted infections.
When the present invention is applied to a sperm collection support system, for example, the motion information output device 3 is worn on a male's wrist using adult content obtained by photographing a sexual activity of a male and female as video content. Then, in response to the reciprocation of the wrist accompanying the male masturbating act, the adult content is interlocked and reproduced.
In this sperm collection support system, the sense of reality can be enhanced and the collection efficiency of sperm can be enhanced, as compared with a general configuration in which adult content is simply reproduced.

[Example of embodiment of the present invention and action, summary of effect]
First Embodiment
Usage between the first position (one of the proximity position B and the separation position T) and the second position (the other of the proximity position B and the separation position T) output from the movement information output unit (the movement information output device 3) An image processing apparatus that processes an image based on motion information (sensor output) that changes periodically along with the reciprocation of a specific part (wrist) of a person, and is a plurality of captured image information captured in time series ( An image storage unit (storage unit 22) for storing shooting frame data), an image selection unit (image selection unit 21c) for selecting shooting image information stored in the image storage unit, and a time interval between shooting image information Time interval setting unit (time interval setting unit 21d), and the photographed image information includes first photographed image information (photographing frame data F (F1 M0), F (F1 M1)) corresponding to the first position; Second shot corresponding to 2 positions A prediction unit that includes information (photographing frame data F (F1 M1), F (F1 M2)) and predicts the movement time of the specific part between the first position and the second position based on the motion information 21e), and the image selection unit selects the first captured image information at a first timing (timing ts5, ts6) at which the specific part is located at the first position for reciprocating motion, and the prediction unit predicts At the second timing (timing ts6 ', ts7') at which the specific part reaches the second position after the movement time (predicted time) has passed from the first timing, the second photographed image information is selected, and the time interval is The setting unit sets a time interval between the photographed image information from the first photographed image information to the second photographed image information to a time interval based on the movement time predicted by the prediction unit (time interval dt2, dt3) It is characterized by being set to
According to the video processing device of the present aspect, even if the movement time of the specific part between the first position and the second position varies, the first to the second photographed image information may be taken between the photographed image information Since the time interval is set according to the moving time, the display of the image can be made to follow the moving time of the specific part. As a result, it is possible to realize video processing suitable for reciprocating movement of a specific part by the user.

Second Embodiment
The prediction unit performs a moving time (time interval TS5 ') related to the one-way movement of the reciprocation for another one-way movement (at timing ts1 to ts2) performed a plurality of times before the one-way movement. The movement, movement of the wrist performed from timing ts3 to ts4) is predicted based on the operation information output from the operation information output unit.
According to the video processing device of this aspect, even if the moving time related to the one-way motion of the reciprocating motion varies, the display of the video can be made to follow the moving time of the specific part.

Third Embodiment
The time interval setting unit is characterized in that the time interval between the photographed image information is changed stepwise (dt2a to dt2c, dt2d to dt2f) as it goes from the first timing to the second timing.
According to the video processing device of this aspect, an image can be displayed smoothly at the switching timing of control.

Fourth Embodiment
A display image information generation unit (display image information generation unit 21f) that generates display image information (display frame data) to be displayed on the display device (display device 4) based on the captured image information is characterized.
According to the video processing device of the present aspect, an image suitable for the display device can be generated based on the photographed image information.

<Fifth embodiment>
The display image information generation unit generates display image information by performing alpha blend processing of a pair of successive captured image information, and the alpha blend processing is performed by dividing the time interval between the captured image information according to the refresh timing of the display device. Is a coefficient.
According to the video processing device of this aspect, an image can be displayed smoothly.

Sixth Embodiment
The display device is characterized in that it is a stereoscopic image display device that displays photographed image information by a stereoscopic image that can be viewed stereoscopically.
According to the video processing apparatus of this aspect, an image can be displayed as a stereoscopic image.

Seventh Embodiment
The photographed image information is another photographed image information included between the first photographed image information and the second photographed image information, and the number of the photographed image information is different. A plurality of sets (photographing of the interlocked scene of the first to fifth periods) The present invention is characterized by including a photographed image information selection unit (photographed image information selection unit 21g) that includes frame data and selects any one of a plurality of sets of photographed image information based on operation information.
According to the image processing apparatus of this aspect, it is possible to cope with fluctuations in the speed of the reciprocation of a specific part in a wide range.

DESCRIPTION OF SYMBOLS 1 ... Video display system, 2 ... Control apparatus (video processing apparatus), 3 ... Operation information output apparatus, 4 ... Display apparatus, 5 ... Sound output apparatus, 6 ... Rowing machine, 21 ... Control part, 21a ... CPU, 21b ... RAM, 21c: image selection unit, 21d: time interval setting unit, 21e: prediction unit, 21f: display image information generation unit, 21g: photographed image information selection unit, 22: storage unit (image storage unit), 23: frame buffer , 31 ... control unit, 32 ... gyro sensor, 33 ... acceleration sensor, 34 ... first mode changeover switch, 35 ... second mode changeover switch

Claims (10)

1. An image processing apparatus that processes an image based on operation information that periodically changes with reciprocating motion of a specific part of a user between a first position and a second position, which is output from an operation information output unit. ,
   An image storage unit storing a plurality of pieces of captured image information captured in time series;
   An image selection unit for selecting photographed image information stored in the image storage unit;
   A time interval setting unit configured to set a time interval between the photographed image information;
   Equipped with
   The captured image information includes first captured image information corresponding to the first position and second captured image information corresponding to the second position,
   And a prediction unit configured to predict movement time of the specific part between the first position and the second position based on the motion information.
   The image selection unit selects the first captured image information at a first timing at which the specific part is located at the first position, and the movement time predicted by the prediction unit is the first movement. Selecting the second captured image information at a second timing at which the specific part reaches the second position after a lapse of time from the timing of
   The time interval setting unit sets a time interval between the photographed image information from the first photographed image information to the second photographed image information to a time interval based on the movement time predicted by the prediction unit. An image processing apparatus characterized by
2. The prediction unit outputs the movement time related to the one-way movement of the reciprocating movement to the operation information output from the movement information output unit for another one-way movement performed a plurality of times before the one-way movement. The image processing apparatus according to claim 1, wherein the image processing apparatus predicts based on the image.
3. 3. The apparatus according to claim 1, wherein the time interval setting unit changes the time interval between the captured image information in a stepwise manner as the first timing moves to the second timing. Image processing apparatus as described.
4. The video processing apparatus according to any one of claims 1 to 3, further comprising a display image information generation unit that generates display image information to be displayed on a display device based on the photographed image information.
5. The display image information generation unit generates the display image information by performing alpha blend processing on a pair of the photographed image information which are in succession.
   5. The video processing apparatus according to claim 4, wherein the alpha blending process uses a division ratio of a time interval between the photographed image information according to a refresh timing of the display device as a coefficient.
6. The video processing apparatus according to claim 4 or 5, wherein the display device is a stereoscopic image display device that displays the photographed image information by a stereoscopic image which can be viewed stereoscopically.
7. The photographed image information includes a plurality of sets of other photographed image information which is included between the first photographed image information and the second photographed image information, the number of which differs.
   The video processing according to any one of claims 1 to 6, further comprising: a photographed image information selection unit which selects any one of the plurality of sets of photographed image information based on the operation information. apparatus.
8. A video processing method for processing a video based on motion information periodically changed with reciprocating motion of a specific part of the user between the first position and the second position, which is output from the motion information output unit. ,
   Storing a plurality of pieces of photographed image information which are photographed in time series and which include first photographed image information corresponding to the first position and second photographed image information corresponding to the second position;
   Predicting the movement time of the specific part between the first position and the second position based on the motion information;
   Regarding the reciprocation, the first captured image information is selected at a first timing when the specific part is located at the first position, and the identification after the predicted movement time has elapsed from the first timing Selecting the second captured image information at a second timing when the part reaches the second position;
   Setting a time interval between the photographed image information from the first photographed image information to the second photographed image information to a time interval based on the predicted movement time. Method.
9. A computer program causing a computer to execute each step in the video processing method according to claim 8.
10. An operation information output unit that outputs operation information that periodically changes with reciprocating motion of a specific part of the user between the first position and the second position, and image processing that processes an image based on the operation information A video processing system comprising:
    The video processing device is
    An image storage unit storing a plurality of pieces of captured image information captured in time series;
    An image selection unit for selecting photographed image information stored in the image storage unit;
    A time interval setting unit configured to set a time interval between the photographed image information;
    Equipped with
    The captured image information includes first captured image information corresponding to the first position and second captured image information corresponding to the second position,
    A prediction unit that predicts the movement time of the specific part between the first position and the second position based on the motion information;
    The image selection unit selects the first captured image information at a first timing at which the specific part is located at the first position, and the movement time predicted by the prediction unit is the first movement. Selecting the second captured image information at a second timing at which the specific part reaches the second position after a lapse of time from the timing of
    The time interval setting unit sets a time interval between the photographed image information from the first photographed image information to the second photographed image information to a time interval based on the movement time predicted by the prediction unit. Video processing system characterized by

Images