JP7397282B2 - Stationary determination system and computer program - Google Patents

Description

The present invention relates to a stationary determination system and a computer program.

Patent Document 1 discloses a method of acquiring an image of a user performing an exercise movement, generating a human movement screening score based on the image, and generating a personalized exercise program based on the score. .

Japanese Patent Application Publication No. 2016-73789

In order to evaluate such user movements, it has become necessary to stably determine whether the user's body is still.

The present invention has been made in view of these circumstances, and provides a stillness determination system and a computer program that make it possible to stably determine whether a user's motion is still.

According to the present invention, there is provided a system for determining whether or not a user's body is stationary, including an imaging section, a region specifying section, an image area setting section, and a stillness determination section, the imaging section comprising: The user's movements are photographed to obtain photographed data including a plurality of frames, the region specifying section identifies a predetermined region of the user's body in the acquired photographic data, and the image area setting section , a system in which an image area is set to be associated with the identified predetermined part, and the stillness determination unit determines whether or not the predetermined part is still based on a difference between frames of the image area. is provided.

With such a configuration, it is possible to stably determine whether a specific part of the user's body is still.

FIG. 1 is a configuration diagram of a determination system 1 according to a first embodiment of the present invention. 1 is a diagram showing the appearance of a mobile terminal 10. FIG. 1 is a block diagram showing the hardware configuration of a mobile terminal 10. FIG. 1 is a functional block diagram of a determination system 1. FIG. 7 is a flowchart showing the process flow of the determination method. 3 is an explanatory diagram illustrating identification of a region by a region identification unit 32. FIG. 3 is a diagram illustrating an example of area setting by an image area setting unit 33. FIG. FIG. 7 is a diagram showing another example of area setting by the image area setting unit 33; FIG. 3 is a diagram illustrating a stillness determination by a stillness determination unit 34. FIG. It is a block diagram of the determination system 1 of 2nd Embodiment of this invention.

Embodiments of the present invention will be described below with reference to the drawings. Various features shown in the embodiments described below can be combined with each other. Further, the invention is established independently for each characteristic matter.

<1. First embodiment>
(1-1. Overall configuration of judgment system 1)
As shown in FIG. 1, a determination system 1 according to a first embodiment of the present invention includes a mobile terminal 10. The mobile terminal 10 can be a smartphone, a tablet terminal, or the like, on which a plurality of application programs can be installed.

In this embodiment, the mobile terminal 10 includes a camera 20 and is arranged to photograph the user U from the front side. Note that the configuration shown in FIG. 1 is an example, and the mobile terminal 10 may be placed at a position other than the front side of the user U.

(1-2. Configuration of mobile terminal 10)
As shown in FIG. 2, the housing 12 of the mobile terminal 10 is equipped with a touch panel display 14, which is an image display means for displaying an image on a screen, on the front surface 12A. The touch panel display 14 includes, for example, an LCD (Liquid Crystal Display) and a touch sensor. The LCD displays various images, and the touch sensor accepts various input operations performed using a pointing object such as a finger, stylus, or pen.

Further, the front surface 12A of the mobile terminal 10 is equipped with a microphone 16 for inputting sound, a speaker 18 for outputting sound, and a camera 20 for capturing an image of a subject. The camera 20 is provided not only on the front side 12A of the housing 12 but also on the back side of the housing 12. Furthermore, the side surface 12B of the housing 12 is provided with operation buttons 22 such as a power button for starting or stopping the mobile terminal 10 and a volume adjustment button for the sound output from the speaker 18.

Furthermore, the housing 12 of the mobile terminal 10 is provided with a slot into which a memory card is inserted, a USB (Universal Serial Bus) terminal, and the like.

FIG. 3 is a functional block diagram showing the electrical configuration of the mobile terminal 10.

In addition to the above configuration, the mobile terminal 10 includes a control section 24, a storage section 26, and a communication section 28.

The control unit 24 is, for example, a CPU (Central Processing Unit), a microprocessor, a DSP (Digital Signal Processor), etc., and controls the overall operation of the mobile terminal 10.

The storage unit 26 includes, for example, a RAM (Random Access Memory), a DRAM (Dynamic Random Access Memory), and the like, and is used as a work area or the like when the control unit 24 executes processing based on various programs.

The storage unit 26 further includes a nonvolatile memory such as a flash memory, and stores various data such as images and programs used for processing by the control unit 24. The programs stored in the storage unit 26 include, for example, an OS (Operating System) for realizing the basic functions of the mobile terminal 10, drivers for controlling various hardware, e-mail and web browsing, and various other functions. These are application programs, etc. to realize this. Further, the storage unit 26 stores in advance a program for executing a determination method according to the present embodiment, the details of which will be described later.

The communication unit 28 is, for example, a NIC (Network Interface Controller), and has a function of connecting to a communication network such as a mobile phone network. Note that the communication unit 28 has a function of connecting to a wireless LAN (Local Area Network), a function of connecting to a wireless WAN (Wide Area Network), instead of or together with the NIC, and has a short-range communication function such as Bluetooth (registered trademark). It may also have functions that enable wireless communication, infrared communication, and the like.

These control section 24, storage section 26, communication section 28, touch panel display 14, microphone 16, speaker 18, camera 20, and operation button 22 are electrically connected to each other via a system bus 30. Therefore, the control unit 24 accesses the storage unit 26, displays images on the touch panel display 14, grasps the state of operation of the touch panel display 14 and operation buttons 22 by the user, inputs sound to the microphone 16, and outputs sounds from the speaker 18. output, control over the camera 20, and access to various communication networks and other information processing devices via the communication unit 28.

(1-3. Functional block diagram)
FIG. 4 is a functional block diagram regarding the determination function according to this embodiment.

As shown in FIG. 4, the control section 24 includes an imaging section 31, a region specifying section 32, an image area setting section 33, and a stillness determining section 34.

The photographing unit 31 photographs the motion of the user U and obtains photographed data including a plurality of frames. The region identifying unit 32 identifies a predetermined region of the user U that is a target of stillness determination in the acquired imaging data.

The image area setting unit 33 sets an image area associated with the specified predetermined part. The stillness determination unit 34 determines whether the specified predetermined region is still or not based on the difference between frames of the image area.

Each functional configuration of the control section 24 described above is realized by a computer program stored in the storage section 26. Further, the computer program may be stored in a computer-readable non-transitory recording medium and read from the recording medium as necessary.

(1-4. Flow of judgment processing)
Referring to FIG. 5, the flow of the stillness determination process using the determination system 1 will be described in detail.

In step S110, the photographing unit 31 photographs the user U's movements. Here, if the photographing is performed for T seconds at N frames/second, the number M of frames included in the photographic data is calculated as N×T. N and T are positive real numbers, and M is a natural number. Each frame contains a still image. N is, for example, 60, and T is, for example, 5. In this case, the photographic data includes 300 still images (M=60×5=300).

In step S120, the region specifying unit 32 specifies the region of the user U that is the object of the stillness determination in the acquired imaging data. This identification can be performed, for example, based on a posture model based on two-dimensional posture estimation for each still image that constitutes the photographic data.

In two-dimensional posture estimation, the posture of the user U can be estimated two-dimensionally by estimating the coordinates of a plurality of feature points of the user U in a still image using, for example, an algorithm using deep learning. Furthermore, by estimating the two-dimensional posture of a plurality of still images, it is also possible to measure the movement of the user's U skeleton.

FIG. 6 shows an example of arrangement of a plurality of feature points. FIG. 6 illustrates an example of an image in which the user U is standing on one leg with his right foot. Here, feature points 0 to 14 are head (0), neck (1), right shoulder (2), right elbow (3), right hand (4), left shoulder (5), left elbow (6), They correspond to the left hand (7), waist (8), right hip joint (9), right knee (10), right ankle (11), left hip joint (12), left knee (13), and left ankle (14).

In this way, when the coordinates of the feature points 0 to 14 of the user U are obtained in each still image by two-dimensional pose estimation, the user's pose can be estimated based on these coordinates. Note that the number and arrangement of feature points are not limited to those in this embodiment, and can be changed as appropriate.

In this way, the coordinates of a plurality of feature points obtained by two-dimensional pose estimation for each still image constituting each photographic data can be acquired as measurement data. Here, when the photographic data is composed of M frames, the measurement data corresponding to the photographic data is also composed of M frames.

The part specifying unit 32 specifies a predetermined part in the body of the user U based on the acquired measurement data (that is, the coordinates of the plurality of feature points). Here, the region identification unit 32 may change the region to be identified depending on the user's U motion. In this case, the region identifying unit 32 does not need to acquire measurement data for all feature points, and may acquire measurement data only for the region that should be identified for each action performed by the user U.

In step S130, the image area setting unit 33 sets an image area D associated with the specified predetermined part. When setting the image area D, feature points corresponding to each movement of the user U are stored in the storage unit 26. As an example, for the action of standing on one leg with the right foot shown in FIG. 7, feature points 1 to 7, which are upper body parts, are associated as predetermined parts. Therefore, the image area setting unit 33 refers to the storage unit 26 and sets the image including the feature points 1 to 7 as the image area D. Further, the image area setting unit 33 may set the image above the feature point 8 (waist) as the image area D, for example, when the region to be specified is the upper body.

In another example shown in FIG. 8, a case is illustrated in which the user U extends his left arm. In such a case, an image including the feature points 5 to 7 may be set as the image region D in order to estimate the position of the left arm from the coordinate positions of the feature points 5 to 7.

In step S140, the stillness determination unit 34 determines whether the specified predetermined region is still or not based on the difference between frames of the image area. Specifically, the stillness determination unit 34 calculates the difference in gradation values for each pixel included in the image area for three consecutive frames in the acquired imaging data, and determines whether the specified predetermined region is still. Determine whether or not the

A detailed explanation will be given below with reference to FIG. 9. FIG. 9 shows three consecutive images G1 to G3 selected from the acquired photographic data. It is assumed that the photographic data is composed of 300 still images (M=300), and is composed of a starting point frame F1 to an ending point frame F300. Images G1 to G3 correspond to frames Fn to Fn+2 (1≦n≦298), respectively.

Images G1 to G3 each include image areas D1 to D3 set by image area setting section 33. Here, the image regions D1 to D3 have tone values ranging from 0 to 255 for each of a plurality of pixels. That is, it can be expressed by the following (Formula 1) to (Formula 3).

Here, D1(i,j) means each pixel included in the image area D1. The operator "∋" means possible values. The stillness determination unit 34 calculates the absolute value of the difference between each image area where frames are continuous as difference images F12 and F23.

Next, the stillness determination unit 34 calculates a logical product image Fa by calculating a logical product of the difference images F12 and F23.

Here, the operator "∩" means an intersection set. Next, the stillness determination unit 34 performs a binarization process by comparing it with an arbitrary threshold value T, and calculates a binarized difference image Ft.

Through the above processing, in the image regions D1 to D3, portions (pixels) determined to be stationary have a gradation value of 255, and pixels determined not to be stationary have a gradation value of 0. Note that the threshold value T may be appropriately set according to the accuracy required in the stationary determination.

Next, a total sum F of pixel values in the difference image Ft is calculated, and a determination value Fv for determining the stationary state is calculated.

As described above, a plurality of (298 in this embodiment) determination values Fv are determined for frames Fn to Fn+2 (1≦n≦298). A stationary determination is made by performing predetermined statistical processing on this determination value Fv and comparing it with a predetermined threshold. The statistical processing here is, for example, processing for calculating a statistically determined value such as a simple sum, a weighted sum, or a value based on this (simple average, weighted average).

As described above, the determination system 1 according to the present embodiment includes the imaging section 31, the region specifying section 32, the image area setting section 33, and the stillness determining section 34. The photographing unit 31 photographs the user's movements and acquires photographic data including a plurality of frames. The region identification unit 32 identifies a predetermined region of the user's body based on the acquired imaging data.

The image area setting unit 33 sets an image area associated with the specified predetermined part. The stillness determination unit 34 determines whether a predetermined region is still or not based on the difference between frames in the image area.

With such a configuration, it is possible to suppress the effects of blurring and the like upon detection of a specific part of the user's body, and it is possible to stably perform a stationary determination.

<2. Second embodiment>
Referring to FIG. 10, a determination system 1 according to the second embodiment will be described. The determination system 1 according to the second embodiment differs from the first embodiment in that it includes a base device 10a and a slave device 10b as the mobile terminal 10. Hereinafter, differences from the first embodiment will be mainly explained.

As shown in FIG. 10, in this embodiment, the handset 10b is arranged on the right side, left side, and rear side of the user U. The slave device 10b is configured with a mobile terminal 10, similarly to the base device 10a. The base unit 10a and the slave unit 10b are configured to be able to communicate through the communication unit 28. The handset 10b functions as a photographing unit 31, photographs the movements of the user U from a plurality of directions, and acquires a plurality of photographic data.

The photographic data acquired by the slave unit 10b is transmitted to the base unit 10a. The region identifying unit 32 of the base device 10a identifies a predetermined region for each of the plurality of acquired photographic data.

The stillness determination unit 34 sets weighting based on the shooting direction for the user U, for shooting data acquired from a plurality of directions. The stillness determination unit 34 determines whether a predetermined part of the user is stationary based on the difference between frames of the image area set by the image area setting unit 33 and the weighting.

With such a configuration, whether or not the user's body is stationary is analyzed based on a plurality of photographic data, and it is possible to more stably determine whether the user's body is stationary.

Here, preferably, the stillness determining unit 34 may set weighting based on the photographing direction according to the motion of the user U who photographs. Weighting based on the imaging direction is set in association with the identified predetermined region. That is, in the movement of the user U, it is conceivable to set a large weight for photographic data taken from a direction in which movement is likely to be detected for each predetermined region.

With such a configuration, it is possible to change the weighting based on the shooting direction according to the user's motion, and it is possible to more stably perform a stillness determination.

<3. Other embodiments>
Although the embodiments have been described above, the technical scope of the present application is not limited to the above embodiments.

For example, the stillness determination unit 34 may be configured to select three frames selected based on the determination value of the stillness determination as determination target frames from among consecutive L frames included in the acquired photographic data.

In this case, the stillness determination unit 34 calculates the determination value for the stillness determination for P consecutive three frames (L=3) having different starting points, for example. Here, P is a natural number. The stillness determination unit 34 may select frames whose ranking is within Q rank when the P determination values are arranged in descending order of the movement of the user U as the determination target frame. Note that L can also be set to a value other than 3.

For example, if P=90 and Q=10, 10 determination values Fv with the smallest values are selected from 90 determination values Fv. Furthermore, if P=90, Q=1, and the judgment value Fv of three consecutive frames starting from frame F121 (that is, frames F121 to F123) is the minimum, frames F121 to F123 become the judgment target frames. . Note that the determination value Fv may be calculated for three consecutive frames starting from frame F0, but the reliability of data immediately after the start of imaging is low. That is, the reliability of the determination may be increased by excluding the first plurality of frames from the measurement data and specifying the determination target frame.

In addition, when selecting evaluation target frames when photographing from multiple directions as in the second embodiment, a determination value Fv for still determination is calculated from photographic data from the photographing direction with the highest weighting, and the determination target frame is may be selected. In this case, the determination value Fv of the frame in the other photographing direction corresponding to the selected determination target frame is calculated, and the stillness determination is performed by considering weighting based on the photographing direction to the determination value Fv in each direction.

Further, in the above embodiment, the determination system 1 is configured using a mobile terminal 10 such as a smartphone or a tablet, but at least one of the devices configuring the determination system 1 is capable of communication and has a shooting function. It may be any device (e.g. a web camera) that has a

Furthermore, the present invention can also be realized as a program that causes a mobile terminal to function in order to realize the above-described determination system.

Furthermore, the present invention can also be realized as a computer-readable non-transitory recording medium that stores the above-mentioned program.

<4. Features of embodiment>
The features of the embodiments of the present invention are summarized below.

The determination system 1 includes an imaging section 31 , a region specifying section 32 , an image area setting section 33 , and a stillness determining section 34 . The photographing unit 31 photographs the user's movements and acquires photographic data including a plurality of frames. The region identification unit 32 identifies a predetermined region of the user's body based on the acquired imaging data. The image area setting unit 33 sets an image area associated with the specified predetermined part. The stillness determination unit 34 determines whether a predetermined region is still or not based on the difference between frames in the image area.

With such a configuration, it is possible to suppress the effects of blurring and the like upon detection of a specific part of the user's body, and it is possible to stably perform a stationary determination.

The region identification unit 32 included in the determination system 1 may identify a predetermined region of the user U's body based on the posture model estimated from the imaging data.

With such a configuration, it is possible to stably identify a part of the user's body.

The region identification unit 32 may identify a predetermined region based on user U's motion.

With such a configuration, a region to be specified is set for each movement, and the region can be specified more stably.

The plurality of frames of photographic data is composed of M frames, and the stillness determination unit 34 determines a predetermined value based on the difference between the frames for P consecutive L frames having different starting points from among the photographic data. The determination value Fv regarding the degree of stillness of the part is calculated, and the specified predetermined value is determined based on the L frames that rank within Q when the P different determination values Fv are arranged in descending order of movement of the predetermined part. It may also be determined whether the region is stationary.

With such a configuration, the targets for the stillness determination can be narrowed down, and the stillness determination can be performed more stably.

The photographing unit 31 acquires a plurality of pieces of photographic data by photographing the user's movements from a plurality of directions, and the part specifying unit 32 specifies each of the predetermined parts from the plurality of acquired photographic data. Good too.

With such a configuration, a plurality of pieces of photographic data are analyzed, and a stillness determination can be made more stably.

The stillness determining unit 34 sets weighting for the plurality of pieces of photographic data based on the photographing direction with respect to the user U, and determines whether a predetermined region is stationary based on the difference between frames in the image area and the weighting of the photographic data. It may also be determined whether the

With such a configuration, a plurality of pieces of data are weighted in the photographing direction, and a stillness determination can be performed more stably.

The stillness determination unit 34 may set weighting based on the shooting direction depending on the user's U movement.

With such a configuration, the weighting of the photographing direction is changed according to the movement, and it is possible to more stably determine whether the object is still.

Although various embodiments of the invention have been described, these are presented as examples and are not intended to limit the scope of the invention. The new embodiment can be implemented in various other forms, and various omissions, substitutions, and changes can be made without departing from the gist of the invention. The embodiment and its modifications are included within the scope and gist of the invention, and are included within the scope of the invention described in the claims and its equivalents.

1: Judgment system 10: Mobile terminal 10a: Base unit 10b: Child unit 11: Feature points 12: Housing 12A: Front 12B: Side 13: Feature points 14: Touch panel display 16: Microphone 18: Speaker 20: Camera 22: Operation Button 24: Control unit 26: Storage unit 28: Communication unit 30: System bus 31: Imaging unit 32: Part identification unit 33: Image area setting unit 34: Stillness determination unit U: User

Claims (8)

A system that determines whether a user's body is stationary,
Comprising an imaging section, a region specifying section, an image area setting section, and a stillness determination section,
The photographing unit photographs the user's movements and obtains photographic data including a plurality of frames;
The part specifying unit is configured to determine, with respect to the acquired imaging data, a predetermined portion of the user's body that is associated with a plurality of feature points based on the imaging data, among a plurality of feature points including a hand and a part other than the hand. identify the part of
The image area setting unit sets an image area associated with the identified predetermined part,
The stillness determination unit determines whether or not the predetermined region is still based on a difference between frames of the image area. A system that determines whether a user's body is stationary,
Comprising an imaging section, a region specifying section, an image area setting section, and a stillness determination section,
The photographing unit photographs the user's movements and obtains photographic data including a plurality of frames;
The part identifying unit identifies a predetermined part of the user's body with respect to the acquired imaging data,
The image area setting unit sets an image area associated with the identified predetermined part,
The stationary determination unit determines whether the predetermined part is stationary based on a difference between frames of the image area,
The system, wherein the region identifying unit identifies the predetermined region based on a posture model estimated from the imaging data. A system that determines whether a user's body is stationary,
Comprising an imaging section, a region specifying section, an image area setting section, and a stillness determination section,
The photographing unit photographs the user's movements and obtains photographic data including a plurality of frames;
The part identifying unit identifies a predetermined part of the user's body with respect to the acquired imaging data,
The image area setting unit sets an image area associated with the identified predetermined part,
The stationary determination unit determines whether the predetermined part is stationary based on a difference between frames of the image area,
The plurality of frames of the photographic data is composed of M frames,
The stationary determination unit includes:
Calculating a determination value regarding the degree of stillness of the predetermined region based on the difference between frames for P consecutive L frames having different starting points from the imaging data;
It is determined whether or not the specified predetermined part is stationary based on L frames whose ranking is within Q rank when the P judgment values are arranged in descending order of movement of the predetermined part. ,system. A system that determines whether a user's body is stationary,
Comprising an imaging section, a region specifying section, an image area setting section, and a stillness determination section,
The photographing unit photographs the user's movements from a plurality of directions to obtain photographic data including a plurality of frames;
The part identifying unit identifies a predetermined part of the user's body with respect to the acquired imaging data,
The image area setting unit sets an image area associated with the identified predetermined part,
The stillness determination unit determines whether or not the predetermined region is still based on a difference between frames of the image area. A computer program for causing a computer to execute a method for determining whether a user's body is at rest, the method comprising: a photographing step, a region specifying step, an image area setting step, and a stillness determination step, the method comprising:
In the photographing step, photographing the user's movements and obtaining photographic data including a plurality of frames;
In the region identification step, for the acquired imaging data, a predetermined portion of the user's body that is associated with a plurality of feature points based on the imaging data, among a plurality of feature points including hands and areas other than the hands, is determined. identify the part of
In the image area setting step, an image area that is associated with the identified predetermined part is set;
In the stillness determining step, the computer program determines whether or not the predetermined region is still based on a difference between frames of the image area. A computer program for causing a computer to execute a method for determining whether a user's body is at rest, the method comprising: a photographing step, a region specifying step, an image area setting step, and a stillness determination step, the method comprising:
In the photographing step, photographing the user's movements and obtaining photographic data including a plurality of frames;
In the region identifying step, a predetermined region on the user's body is identified with respect to the acquired imaging data,
In the image area setting step, an image area that is associated with the identified predetermined part is set;
In the stationary determination step, it is determined whether the predetermined part is stationary based on a difference between frames of the image area,
In the region specifying step, the computer program specifies the predetermined region based on a posture model estimated from the imaging data. A computer program for causing a computer to execute a method for determining whether a user's body is at rest, the method comprising: a photographing step, a region specifying step, an image area setting step, and a stillness determination step, the method comprising:
In the photographing step, photographing the user's movements and obtaining photographic data including a plurality of frames;
In the region identifying step, a predetermined region on the user's body is identified with respect to the acquired imaging data,
In the image area setting step, an image area that is associated with the identified predetermined part is set;
In the stationary determination step, it is determined whether the predetermined part is stationary based on a difference between frames of the image area,
The plurality of frames of the photographic data is composed of M frames,
In the stationary determination step,
Calculating a determination value regarding the degree of stillness of the predetermined region based on the difference between frames for P consecutive L frames having different starting points from the imaging data;
It is determined whether or not the specified predetermined part is stationary based on L frames whose ranking is within Q rank when the P judgment values are arranged in descending order of movement of the predetermined part. , computer program. A computer program for causing a computer to execute a method for determining whether a user's body is at rest, the method comprising: a photographing step, a region specifying step, an image area setting step, and a stillness determination step, the method comprising:
In the photographing step, the user's movements are photographed from a plurality of directions to obtain photographic data including a plurality of frames;
In the region identification step, a predetermined region of the user's body is identified based on the acquired imaging data,
In the image area setting step, an image area that is associated with the identified predetermined part is set;
In the stillness determining step, the computer program determines whether or not the predetermined region is still based on a difference between frames of the image area.