JP2013201714A - Moving-object image discrimination device and moving-object image discrimination method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To automatically discriminate a measurement target on the basis of an image and to accurately discriminate the target using a moving image of insufficient resolution.SOLUTION: A moving-object image discrimination device comprises: a moving image camera 1 for imaging an identical measurement region from an identical direction and a still image camera 2 with a higher image resolution than that of the moving image camera 1; moving image side image extraction means 3 which extracts an image of a discrimination target in a moving image frame using information on movement of the discrimination target, from the moving image imaged by the moving image camera 1; still image side image extraction means 4 which, on the basis of the positional information of the extracted image extracted by the moving image side image extraction means 3, extracts an image of the discrimination target from a still image at an identical time; and means which discriminates the type of the discrimination target on the basis of the extracted image extracted by the still image side image extraction means 4.

Description

  The present invention relates to a moving body image discrimination apparatus and a moving body image discrimination method. More specifically, the present invention relates to a moving body image discriminating apparatus and a moving body image discriminating method for automatically discriminating a measurement target by performing pattern discrimination based on an image photographed by a camera.

  In recent years, the temperature of the Japanese coast has risen due to the effects of climate change, which is said to indicate a warming trend, and tropical and subtropical fish that have died in the winter even if they have drifted to the coast of Japan have overwintered. The possibility of being established is shown. In addition, there are concerns that naruto ray from the tropics goes north on the coast of Japan and preys on clams and clams, threatening useful marine resources. In order to elucidate the impact of the concerned climate change and future global warming on the diversity of such coastal ecosystems and marine resources, fish habitat surveys have been conducted. The fish habitat survey is also an important research activity for environmental assessment around the power plant, general environmental conservation, and fishery promotion.

  Currently, such fish surveys include methods for quantifying the species and amount of fish that have been captured by installing research nets and stabbed nets, methods for diving a diver and estimating the amount of live by visual and recording, There is a method in which an operator discriminates a fish shadow of a moving image taken underwater and quantifies the type and amount of the fish.

  However, surveys using bottom nets and gill nets have the advantage of being able to acquire a variety of information such as accurate species identification, weight and total length of the fishes to be investigated, sex and sexual maturity, etc. However, it has the disadvantages that it causes a decrease in biological resources and that it is difficult to quantify large fish species that do not penetrate the gill net. In addition, the diver survey method that has been carried out in recent years due to the spread of scuba tanks has the advantage of being able to visually check large fish and observe and record the actual behavior of fish, but the survey time is limited to a short time, In addition, it has the disadvantage of changing fish habitat and behavior under the influence of observers.

  On the other hand, installing video cameras underwater and recording and recording fish inhabitants and behavior for a long time or periodically is a new method that covers the shortcomings of stabbed net and diver surveys that have been conducted so far. Although it is expected to be a technique, the resolution of video images (moving images) taken with a video camera is insufficient particularly in the sea, so it is difficult to accurately discriminate the fish species from the extracted fish shadow.

  A technique for detecting fish based on a video image is disclosed in, for example, Japanese Patent Application Laid-Open No. 2005-309485.

  Such problems are not limited to measurement of fish in the sea. For example, there are cases in which video is taken and measured for long-distance animals such as birds and insects, small animals, fast-moving animals, etc. It is the same.

JP 2005-309485 A

  The present invention has been made to solve the above-described problems, and enables automatic determination of a measurement target based on an image, and enables accurate determination using a video with insufficient resolution. An object of the present invention is to provide a movable body image discrimination apparatus and a mobile body image discrimination method.

  In order to achieve this object, a moving body image discrimination device according to claim 1 is a moving image camera that captures the same measurement area from the same direction, a still image camera having a higher image resolution than the moving image camera, and a moving image camera. The moving image-side image extracting means for detecting the position of the image to be determined from the moving image captured in step (b), and the image to be determined from the still image at the same time based on the position information of the image detected by the moving image-side image extracting means Still image side image extraction means and means for discriminating the type of the discrimination target based on the image extracted by the still image side image extraction means.

  According to another aspect of the present invention, there is provided a moving object image discriminating apparatus in which a synchronization mark for time synchronization is imprinted on at least a part of a frame of a moving image and at least one still image.

  The moving object image discriminating apparatus according to claim 3, wherein the still image side image extracting means is a coordinate converting means for converting the coordinates of the image detected by the moving image side image extracting means to coordinates on the still image at the same time; Image search means for searching for an image to be discriminated in a still image with reference to conversion coordinates by the conversion means.

  The moving body image discriminating apparatus according to claim 4, when the main moving direction of the discrimination target is classified into the horizontal direction and the vertical direction, the moving image camera and the still image camera are arranged up and down when classified in the horizontal direction, When classified in the vertical direction, the moving picture camera and the still picture camera are arranged on the left and right.

  According to a fifth aspect of the present invention, there is provided a moving body image discriminating apparatus including a database in which a discrimination target image is associated with a type, and the means for discriminating the discrimination target type performs pattern discrimination by referring to the database. Is what you do.

  According to a sixth aspect of the present invention, the means for determining the type of the discrimination target counts the discrimination target.

  According to a seventh aspect of the present invention, there is provided a moving body image discrimination method that captures the same measurement area from the same direction by a moving image camera and a still image camera having a higher image resolution than the moving image camera and corresponds to the moving image and the moving image. A large number of still images in chronological order are acquired, the position of the image to be determined is detected from the moving image captured by the video camera, and the image of the determination target is determined from the still image at the same time based on the detected position information. And the type of the discrimination target is discriminated.

  According to another aspect of the present invention, there is provided a moving object image discrimination method in which a synchronization mark is imprinted on at least a part of a frame of a moving image and at least one still image, and a moving image and a still image are shot based on the synchronization mark. Synchronize time.

  Further, in the moving body image discrimination method according to claim 9, the discrimination target image is extracted from the still image by converting the coordinates of the discrimination target image detected from the moving image into coordinates on the still image. This is performed by searching for an image to be discriminated in a still image with reference to the converted coordinates.

  The moving body image discrimination method according to claim 10 classifies the main moving direction of the discrimination target into a horizontal direction and a vertical direction, and when moving in the horizontal direction, a moving image camera and a still image camera are arranged vertically. However, when classified in the vertical direction, a moving image camera and a still image camera are arranged on the left and right.

  According to another aspect of the present invention, there is provided a method for discriminating a type of a discrimination target by performing pattern discrimination while referring to a database in which the type of discrimination target image is associated with a type.

  A moving body image discrimination method according to a twelfth aspect counts discriminated discrimination targets.

  According to the moving body image discriminating apparatus according to claim 1 and the moving body image discriminating method according to claim 7, since detection of the discrimination target is performed based on the moving image, the detection accuracy can be improved and the discrimination can be performed. Since the image to be used is extracted based on the still image, the image to be used becomes clear and the discrimination accuracy can be improved. In other words, since the characteristics of the moving image and the still image are used, it is possible to efficiently and accurately perform the discrimination. In addition, it is an automatic discrimination by image processing, which can suppress labor by manpower, and can obtain a highly objective result by preventing variation in discrimination results due to the knowledge amount and proficiency level of the operator. Furthermore, the measurement area in the measurement time zone is unmanned, and it is possible to prevent the worker from affecting the behavior to be discriminated and to measure the normal situation as it is. Further, since the discrimination target is not captured, the resource is not reduced.

  Further, according to the moving body image discriminating apparatus according to claim 2 and the moving body image discriminating method according to claim 8, since the synchronization mark for time synchronization is used, the shooting time of the moving image and the still image can be synchronized. It can be done easily and accurately.

  In addition, according to the moving body image discrimination device according to claim 3 and the moving body image discrimination method according to claim 9, it is possible to improve the accuracy of the association between the moving image and the still image, and the image is captured in the still image. The extraction accuracy of the discrimination target image can be improved.

  Further, according to the moving body image discrimination device according to claim 4 and the moving body image discrimination method according to claim 10, the arrangement of the moving image camera and the still image camera suppresses the influence of the parallax shift as much as possible. It is possible to improve the extraction accuracy of the image to be discriminated in the still image.

  Further, according to the moving object image discrimination apparatus according to claim 5 and the moving object image discrimination method according to claim 11, automatic discrimination by the support vector machine can be performed.

  Furthermore, according to the moving body image discrimination apparatus of claim 6 and the moving body image discrimination method of claim 12, it is easy to know the number of objects to be measured.

It is a block diagram which shows an example of embodiment of the image discrimination device of the moving body of this invention. It is a flowchart which shows an example of embodiment of the image discrimination method of the moving body of this invention. It is a flowchart which shows the procedure in which the still image side image extraction means extracts the image of discrimination | determination object. It is a flowchart which shows other embodiment of the image discrimination method of the moving body of this invention. It is a flowchart which shows the procedure of the pre-processing phase of FIG. It is a perspective view which shows the equipment for underwater photography. It is a figure for demonstrating the difference in the appearance of a fish shadow by parallax shift. It is a graph for demonstrating fish shadow extraction using a sigma band. It is a figure for demonstrating the concept of the coordinate transformation from a moving image to a still image. It is a figure for demonstrating the concept of the fish shadow search by template collation. It is a figure which shows the example of the frame image of the moving image used for evaluation experiment. FIG. 4 shows an image used for time synchronization (time synchronization) of a moving image and a still image, (a) shows an example of a frame image of the moving image, and (b) shows an example of a still image. FIG. 4A is a diagram showing reference points on a moving image frame, and FIG. 4B is a diagram showing reference points on a still image. It is a figure which shows the relationship between the position in a screen, and an area | region capture rate. It is a figure which shows the positional relationship between the height in a screen, and a still image. It is a figure which shows a Haar wavelet characteristic.

  Hereinafter, the configuration of the present invention will be described in detail based on the form shown in the drawings. In the present embodiment, fish in the water is described as an example of discrimination, but the present invention is not limited to this. For example, the present invention may be applied to discriminating animals that swim around in water other than fish, birds such as birds and insects flying in the air, or the like, or may be applied to discriminating animals moving in a plane on the ground. Furthermore, the present invention may be applied to discrimination of moving objects other than animals. For example, when crows, pigeons, rabbits, sparrows and other birds fly, only crows for bird damage control purposes are identified and counted, or specific vehicle types are identified and counted from a group of vehicles traveling on the road, etc. It is applicable to. Furthermore, since a fine image to be discriminated can be obtained from a still image based on position information in a moving image, it can be applied to security measures (such as monitoring a thief) and marketing (such as grasping a visitor).

  FIG. 1 shows an example of an embodiment of a moving object image discrimination apparatus of the present invention. A moving body image discriminating apparatus (hereinafter simply referred to as an image discriminating apparatus) includes a moving image camera 1 that captures the same measurement region from the same direction, a still image camera 2 that has a higher image resolution than the moving image camera 1, and a moving image camera 1. The moving image side image extracting unit 3 that detects the position of the discrimination target from the moving image captured in step S3, and the discrimination target image is extracted from the still image at the same time based on the position information of the image detected by the moving image side image extracting unit 3. Still image side image extraction means 4 and type discrimination means 5 for discriminating the type of the discrimination target based on the image extracted by the still image side image extraction means 4 are provided. The moving image side image extracting unit 3, the still image side image extracting unit 4 and the type discriminating unit 5 are realized by installing and executing a dedicated program on the computer 6.

  The computer 6 constituting the image discrimination device includes a calculation unit 7, a storage unit 8, an input unit 9, a display unit 10, and a transmission / reception unit 11, which are connected to each other by a signal circuit 12 such as a bus.

  The arithmetic unit 7 performs arithmetic operations related to control of the entire image discriminating apparatus by a control program stored in the storage unit 8, and is, for example, a CPU (Central Processing Unit). The storage unit 8 is a device capable of storing at least data and programs, and is, for example, a memory or a hard disk. The input unit 9 is an interface for inputting at least an operator's command and data, and is, for example, a keyboard and a mouse. The display unit 10 performs drawing / display of characters, graphics, and the like under the control of the calculation unit 7 and is, for example, a display, a printer, or the like. The storage unit 8 stores a moving image and a still image to be processed, and a fish species database 20 for discriminating fish species.

  The arithmetic unit 7 executes the control program to realize the moving image side image extracting unit 3, the still image side image extracting unit 4, and the type determining unit 5.

  The moving image side image extracting means 3 extracts fish shadows from all frames of the moving image in the measurement time zone. In the extracted information, the same fish shadows are grouped according to the positional relationship between the previous and next frames, and are organized for each fish shadow. For example, when fish shadow A appears at time t1, moves within the screen, and disappears from the screen at time t2, the trajectory of fish shadow A from time t1 to t2 and the shape of each frame are managed together. . This is in order to avoid duplicate counting of fish shadows extracted from still images.

  The still image side image extracting unit 4 of the present embodiment includes a coordinate converting unit 14 that converts the coordinates of the image detected by the moving image side image extracting unit 3 into coordinates on the still image at the same time, and a converted coordinate by the coordinate converting unit 14. And image search means 15 for searching for a discrimination target image in the still image.

  The type discriminating means 5 discriminates an image by pattern recognition using a machine learning technique, and a support vector machine (SVM) is used in this embodiment. The type discriminating means 5 calculates the feature amount from the fish shadow image registered in the fish species database 20, creates a discrimination criterion for automatic discrimination by machine learning, and the fish searched by the image search means 15 The fish image is automatically discriminated by comparing the shadow image with this discrimination criterion. The fish species database 20 is a database that stores a fish shadow image and its label (a result of discrimination of fish type, classification of fish type or unknown fish) in association with each other, and is created and stored in advance in this embodiment. Stored in the unit 8.

  Moreover, the classification determination means 5 of this embodiment arranges the determination results, and counts and sums up for each fish type. First, the number of appearances during the measurement time for the fish shadows for which the fish species were found is tabulated. The number of appearances is one for one trajectory of fish shadow obtained from the video. As a result, total information such as the average size: S1 pixel and X appearances for fish species A and the average size: S2 pixel and Y appearances for fish species B is obtained.

  In addition, about the fish shadow which could not discriminate | determine a fish type by collation with the fish species database 20, a process is performed as an unknown fish shadow. Also, when the fish type discrimination result is different in different still images corresponding to the same one trajectory, the process is performed as an unknown fish shadow. These fish shadows are subsequently subjected to fish type discrimination by humans, and the fish shadow database 20 is improved by additionally registering them in the fish shadow database 20. In addition, fish shadows that appear and disappear during the still image capture interval even though fish shadows are detected in the video are counted as unknown fish shadow trajectories. .

  In the present embodiment, the type discriminating means 5 totals the image size (number of pixels), that is, the apparent size in the screen, as the fish shadow size.

  The moving image camera 1 is a camera capable of shooting moving images, for example, a video camera. The still image camera 2 is a camera capable of taking a still image, for example, a digital camera. In the present embodiment, a waterproof camera or a camera housed in a waterproof case is used as the moving image camera 1 and the still image camera 2 in order to take an underwater image.

  The present invention is characterized in that fish shadows are detected and discriminated using both moving images and still images. A moving image has a large number of images in time units, and is suitable for extracting a moving object by image processing. However, the resolution is low (for example, in a widespread video camera, the high-definition, that is, 1920 × 1080 pixels is the maximum), and the fish shadow shot at a position far away from the video camera 1 is small and unclear. Not suitable for species discrimination. On the other hand, a still camera (still image camera 2) for taking still images has a high resolution (for example, an inexpensive compact type that achieves 4500 × 3400 pixels or more, and the resolution increases year by year), and fish. Suitable for species discrimination. However, the still camera has a limited number of shots per time unit, and a video recording apparatus that achieves both high-speed long-time shooting and high-resolution shooting is not widespread. Therefore, in the present invention, the position of the fish shadow is detected with a moving image, the type determination of the fish shadow is performed with a still image, and automatic determination by image processing is performed by cooperating while sharing roles.

  Here, it is preferable that the still image camera 2 has a function of performing automatic shooting at fixed time intervals called so-called interval shooting or time-lapse shooting. This function is a function of taking images at regular time intervals such as every 5 seconds or every 10 seconds. However, it is not always necessary to have such an automatic photographing function. In this case, photographing may be repeatedly performed at regular time intervals by an operation from an external timer or a manual operation.

  In the present embodiment, underwater photography is performed, so a fixing jig 16 for underwater photography is used. An example of the fixing jig 16 is shown in FIG. The fixing jig 16 is a box-shaped frame formed of a metal net, and the moving image camera 1 and the still image camera 2 are fixed to the frame. A weight 17 is attached to the frame. The fixing jig 16 is submerged in water by a diver and installed at the photographing position. The reason for using a net as the fixing jig 16 is mainly to reduce resistance to the flow of water in the sea and prevent movement due to the water flow. It also has the purpose of facilitating installation work while reducing environmental impact. However, the moving image camera 1 and the still image camera 2 may be installed at the photographing position using a tripod or the like without using the fixing jig 16.

  The video camera 1 and the still image camera 2 capture the same measurement area from the same direction, and are preferably arranged adjacent to each other. Of course, “same” includes physical identity, but “same” here does not necessarily need to be physically identical, and the still image-side image extraction means 4 has the same time as the moving image. If a moving image and a still image can be taken from the same direction to the extent that a position can be associated with a still image, it is included in “same”. That is, if the parallax shift between the moving image camera 1 and the still image camera 2 can be suppressed within a processable range, “same” is used.

  When two cameras 1 and 2 are installed side by side, a parallax shift occurs. The point of reducing the parallax deviation between the two cameras 1 and 2 is to bring the two cameras 1 and 2 as close as possible. In addition, it is desirable that the optical axes of the two cameras 1 and 2 be parallel without crossing each other. This is because, when the optical axes are crossed, the way the images are reflected varies greatly between short distance and long distance even if the distance between the two cameras 1 and 2 is short. The distance between the two cameras 1 and 2 is ideally zero. The zeroing method requires the production of a special optical system that splits the same image into two parts using a half mirror (for example, Japanese Patent No. 4052382 “Non-contact image measuring device”). Becomes expensive, and the image becomes dark. As another means, there is a digital camera that can shoot video images, so it is possible to use a camera that has a mode in which high-resolution still images are shot with a digital camera at the same time while shooting video images. Conceivable.

  When the distance between the two cameras 1 and 2 cannot be reduced to zero, there are two methods for juxtaposing the cameras 1 and 2: a method of arranging them in the horizontal direction (horizontal direction) and a method of stacking them in the vertical direction (vertical direction). Here, when the distance between the lenses of the cameras 1 and 2 does not change by either arrangement, it is desirable to stack the two cameras 1 and 2 in the vertical direction. This is because fish mostly move in the horizontal direction, not in the vertical direction. Even if the position of the fish shadow extracted from the video is calculated as a slightly different position in the still image due to the parallax shift between the two cameras 1 and 2, if the shift is a vertical shift, the fish The effect is small because it does not move so much. On the other hand, if the shift is in the horizontal direction, the fish moves from moment to moment, so that it becomes a place where no fish shadow exists in the still image, and the possibility that the fish shadow cannot be extracted from the still image increases.

  That is, when the main moving directions to be discriminated are classified into the horizontal direction and the vertical direction, it is desirable that the moving image camera 1 and the still image camera 2 be arranged vertically, and the vertical movement is classified. Sometimes it is desirable to arrange the video camera 1 and the still image camera 2 on the left and right. By arranging in this way, the influence of the parallax shift between the two cameras 1 and 2 can be suppressed, and the extraction accuracy of the image to be discriminated in the still image can be improved.

  The shooting position where the cameras 1 and 2 are installed is selected so that a synchronization mark for time synchronization is imprinted on at least a part of the frames of the moving image and at least one still image. Here, as the synchronization mark, for example, it is preferable to select an image that has a motion and can recognize a change in a frame unit of a moving image, such as a swimming diver or a conspicuous fish shadow. For example, by using a frame that changes in units of frames, such as a diver's hand movement, as a synchronization mark, a corresponding frame can be found based on the difference of each frame. In addition, when a synchronization mark is used for a thing that does not exist in the measurement area in the natural state, such as a diver, it is preferable to copy the synchronization mark in the image immediately before the start of the measurement (observation). For example, when the object to be used is used as a synchronization mark, the synchronization mark may be imprinted on the image after the start of measurement.

  Data of images taken by the video camera 1 and the still camera 2 are stored in the built-in memory, and are taken out after the camera is collected after the shooting is finished. However, extraction of image data is not limited to after collection of the camera. For example, image data may be taken out before collection of the camera by providing an image data transmission / reception device to perform communication. The image data can be obtained.

  The moving image camera 1 and the still image camera 2 are operated underwater at the time of installation and collection by a diver, for example. However, for example, it may be operated remotely from the water. The video camera 1 continues shooting in the measurement time zone. The still image camera 2 repeatedly shoots at predetermined time intervals in the measurement time zone. As a result, a moving image over the measurement time zone and a large number of time-series still images corresponding to the moving image are acquired.

  Next, a moving body image discrimination method (hereinafter simply referred to as an image discrimination method) of the present invention will be described. 2 and 3 show an image discrimination method. In the image discrimination method, the moving image 1 and the still image camera 2 having a higher image resolution than the moving image camera 1 are photographed in the same direction, and the moving image and a number of time-series still images corresponding to the moving image are captured. The image is acquired (step S41), the position of the image to be determined is detected from the moving image taken by the moving camera 1 (step S43), and the image to be determined from the still image at the same time based on the detected position information. Are extracted (step S44), and the type of the discrimination target is discriminated (step S45). The position detection of the image to be discriminated from the moving image is automatically performed by the moving image side image extracting unit 3, and the image to be discriminated from the still image is automatically extracted by the still image side image extracting unit 4. The type determination is automatically performed by the type determination unit 5.

  First, a moving image and a still image are acquired (step S41). The moving image camera 1 and the still image camera 2 are installed at a shooting position where the measurement area can be shot, and shooting is started. At that time, in order to make time synchronization between the captured moving image and the still image as easy as possible, it is preferable that the built-in clock of the moving image camera 1 and the built-in clock of the still image camera 2 are matched as much as possible. Further, in this embodiment, shooting is started with the video camera 1 and the still image camera 2 early before the measurement is started, and the diver crosses in front of the video camera 1 and the still image camera 2 before the measurement is started. The synchronization mark is copied.

  After the elapse of the measurement time, the moving image camera 1 and the still image camera 2 are collected from the water, and the image data is taken out, for example, temporarily stored in the storage unit 8 of the computer 6. However, it may be stored in another storage device, or may be directly processed without being stored in the storage unit 8. The image data is taken into the computer 6 via the transmission / reception unit 11.

  In the present embodiment, a moving image and a still image group to be processed by actually performing on-site shooting are acquired. However, if a moving image and a still image group to be processed are already prepared, the image data is stored in a computer. 6 to capture.

  Next, the shooting times of the moving image and the still image are synchronized (step S42). Time synchronization is performed as follows. The point is to match the still image. First, in the still image, the first image (i) having a synchronization mark that can be used for time adjustment is searched. When an image i with a synchronization mark is found, the time t of the image i is recorded. For example, it is assumed that “August 17, 2010, 10:46:35”. Next, a video image (moving image) is reproduced, and a frame (frame) at the same moment as the image i is searched for using a synchronization mark as a clue. Search for the same sync mark. Eventually, the frame search is performed, the closest frame in the video image is found, and the time of that frame is reset to t. In the above example, “August 17, 2010 10: 46: 35.000 milliseconds” is assumed. That is, the time stamp of the video image is ignored, and the time of the video image is corrected based on the time t. Time synchronization is performed by an operator, for example. It should be noted that the time synchronization may not be performed when the shooting times of the moving image and the still image can be matched without performing the time synchronization.

  Next, the moving image side image extraction means 3 detects the position of the fish shadow from the moving image (step S43). A known method can be used as a method for extracting a moving object from a moving image. In this embodiment, the background difference method is used. However, it is not limited to the background difference method. An example of the background subtraction method is disclosed in Japanese Patent Application Laid-Open No. 2005-309485. In the background subtraction method, data representing the background other than the detection target (background data) is created from the moving image, and the moving object is extracted based on the difference between the data and the frame image at a certain time. Although it is easy to create background data in an experimental environment where there is no change at all, there are some changes in the actual environment, so the point is how to create appropriate background data.

  In the present embodiment, for the purpose of functioning effectively in an environment where the light fluctuation is large, such as the water surface or underwater, background data considering the dispersion of color values in the time direction is created. Specifically, for each coordinate in the screen, the average and standard deviation of the pixel values in the time direction are calculated within a certain time range. That is, when the screen size is horizontal w pixels and vertical h pixels, the time average μ and standard deviation σ of w × h pixel values are obtained. In the case of an RGB color system used in a computer, an average and a standard deviation are obtained for each of R, G, and B, and these are used as background data. This average and standard deviation represent the variation of the pixel value within the corresponding range.

  The presence / absence determination of the difference will be described with reference to FIG. FIG. 8 is a graph showing a change in luminance at a certain coordinate. The vertical axis represents luminance and the horizontal axis represents time. Although the luminance at the coordinates changes, most of them fall within a certain range (the shaded portion in the figure). If a fish that is brighter than the background passes through these coordinates, it deviates from the range for that time period. Conversely, if a fish that is darker than the background passes, it will deviate from the range. In this way, the presence or absence of a difference is determined.

  Specifically, assuming that only the luminance is handled in gray scale, a difference occurs when the luminance g at a certain time is outside the section (sigma band) of [μ−βσ, μ + ασ]. judge. Here, α and β are parameters for adjusting the determination (in the case of an underwater moving image in Example 1 described later, α = β = 4.0). In the case of a color in the RGB color system, it is determined that a difference has occurred when it is outside one of the r, g, and b sigma bands.

  In the present embodiment, since a moving image that has already been shot is targeted, the average and standard deviation are calculated for each time interval, and the difference is detected in the same time interval. Note that if the time interval is lengthened, it will not be possible to cope with underwater lighting fluctuations, but if it is too short, fluctuations in the light underwater will not be absorbed and false detection will increase. It is preferable (in Example 1 described later, the time interval is 5 minutes).

  The above processing is performed for all the coordinates on the screen, but it can be partially distinguished from the fish shadow on the screen, for example, when the brightness fluctuates extremely or when the black fish shadow passes through a dark place. If not, the difference cannot be extracted. Therefore, the pixel to be extracted is not the entire fish shadow but a part thereof. Therefore, a plurality of areas may be extracted from one fish. Thus, in the present embodiment, processing is performed to reconstruct a region representing the entire fish shadow by combining regions having close coordinates at a constant distance. Furthermore, in order to track the movement of the fish, the areas in the preceding and following moving image frames are associated. In the present embodiment, the fish shadow is finally traced for a while, and the fish movements that are too fast are excluded and finally determined as fish shadows.

  The moving image side image extraction means 3 automatically detects the position of the fish shadow. The moving image-side image extracting means 3 performs fish shadow position detection on all moving images in the measurement time zone, and then proceeds to step S44.

  Next, the still image side image extracting means 4 extracts a fish shadow from the still image (step S44). The extraction of fish shadows from the still image by the still image side image extracting means 4 is performed for all fish shadows detected by the moving image side image extracting means 3. Here, for example, fish shadow A that appears in the moving image frame at time t1, moves within the screen, and disappears in the moving image frame after the moving image frame at time t2 (shown in the moving image frame between time t1 and time t2). As for fish shadow A), a still image taken between time t1 and time t2 becomes a still image at the same time as a moving image frame in which fish shadow A is reflected. At this time, if there is one still image taken between time t1 and time t2, that still image becomes a still image at the same time, and if the shooting time of the still image is time ta, at time ta Coordinate conversion between a captured moving image frame (moving image frame captured at the time closest to time ta when there is no moving image frame accurately captured at time ta) and the still image captured at time ta And the subsequent processing is performed. In addition, when a plurality of still images are taken between time t1 and time t2, these still images become still images at the same time, and the subsequent processing is performed on each still image. However, when a plurality of still images are taken between time t1 and time t2, one still image may be selected from the still images, coordinate conversion may be performed, and the subsequent processing may be performed.

  In the present embodiment, the coordinate conversion step S48 (FIG. 3) for converting the coordinates of the fish shadow detected from the moving image into the coordinates on the still image, and the fish shadow for searching for the fish shadow in the still image based on the converted coordinates. The fish shadow is extracted from the still image by executing the search step S49. The coordinate conversion step S48 is performed by the coordinate conversion means 14 constituting the still image side image extraction means 4, and the fish shadow search step S49 is performed by the image search means 15 constituting the still image side image extraction means 4. However, it is not restricted to such a method.

  In the coordinate conversion step S48 of the present embodiment, the simplest linear conversion method is used as a method for converting a plurality of camera coordinates, but the present invention is not limited to this. The coordinate conversion means 14 automatically converts the coordinates of the fish shadow position. The linear conversion method will be described with reference to FIG. Three points that can be associated between a moving image and a still image are designated in advance by the user as reference points. The reference point is designated by an operation from the input unit 9 while viewing the image displayed on the display unit 10. This designation is performed only at the first coordinate conversion, and is not performed at the second and subsequent coordinate conversions, and the same reference point is used. As the reference point, it is preferable to use, for example, a coral on the seabed, a diver's limb, a fish with a characteristic color or pattern, but is not limited thereto. It is also possible to display a plate on which a lattice pattern is drawn in front of the camera and use it as a mark (reference point).

  Now, the reference points on the moving image are p1, p2, and p3, respectively, and the corresponding reference points on the still image are p1 ', p2', and p3 '. An oblique coordinate system Av is set with an arbitrary one of these three points, for example, p1 as an origin, and a vector from p1 to p2 and a vector from p1 to p3, respectively, as unit vectors. Similarly, an oblique coordinate system As is set with a vector from p1 'to p2' and a vector from p1 'to p3' as a unit vector, with p1 'corresponding to the point as the origin on the moving image as the origin. The fish shadow position is converted by using the coordinates on the oblique coordinate system Av as they are in the oblique coordinate system As. Three points as reference points define one plane in the underwater space, and conversion means matching in this plane. The P1 plane in FIG. 7 corresponds to this conversion plane. Therefore, in principle, the association is performed accurately in the range in which the fish moves in this plane. However, when the fish shadow moves away from this plane due to the effect of the parallax shift, the fish shadow that appears at the same location on the moving image camera 1 appears at a different position on the still image camera 2. This shift increases as the fish shadow moves away from the plane P1. Since the actual lens is distorted, the length and direction of the unit vector slightly change between the central portion and the peripheral portion. To accurately reproduce this, it is possible to divide and finely divide the inside of the screen using a checkered pattern covering the entire screen as a mark. However, this shift is slight compared with the influence of the parallax shift, and can be ignored.

  In fish shadow search step S49, a fish shadow is searched for on the still image. This is because even if the conversion coordinates to the oblique coordinate system As do not indicate the actual fish shadow coordinates due to the parallax shift, the fish shadows can be found in the vicinity of the conversion coordinates if they are images at the same time. Based on assumptions.

  The search method will be described with reference to FIG. First, a fixed search range is set around the coordinates on the still image associated in the coordinate conversion step S48. Within that range, similar images are searched for using the image of the fish shadow area in the moving image frame as a template. The specific procedure is as follows.

  First, a rectangular fish shadow region image 18 including a fish shadow on a moving image frame detected by the moving image side image extracting means 3 is expanded or contracted by linear conversion in accordance with the size on the still image at the same time, and this is converted into a template 19. And When the horizontal width of the template 19 is w and the vertical length is h, a search range 22 having a horizontal width of aw and a vertical length of bh is set around the corresponding point on the still image obtained in the coordinate conversion step S48. Set. Here, a and b are parameters for adjusting the width of the search range (a = 5 and b = 3 in Example 1 described later). The search range 22 is scanned, and an image having the highest similarity with the template 19 is used as a fish shadow. In the present embodiment, a rectangular image including a portion where a fish is shown is a fish shadow extracted by the still image side image extraction means 4.

  The reason for changing the search range according to the template size is to consider that the closer the camera is to the parallax, the greater the parallax.

  The reason why the search range is not wide, such as the entire screen, is as follows. Due to differences in viewpoints between moving images and still images, the way they look is quite different. For this reason, when there are a plurality of fish shadows on the screen, it is often difficult to distinguish them by the degree of similarity with the template 19, and the possibility of collation errors tends to increase as the search range is expanded. This is because it is necessary to narrow down the search range as much as possible.

  In the present embodiment, normalized cross-correlation in gray scale is used for similarity determination. The reason for the gray scale is that the color differs depending on the white balance between the moving image and the still image. The reason for using normalized cross-correlation is the most typical and practical template matching method.

  The fish search is automatically performed by the coordinate search means 15.

  The still image side image extracting unit 4 repeats steps S48 and S49 for all fish shadows detected by the moving image side image extracting unit 3. Then, it progresses to step S45 of FIG.

  Next, the type discrimination means 5 discriminates the fish type (step S45). In this embodiment, the size of the fish is determined together with the determination of the fish species, but the size determination need not be performed. The type discriminating unit 5 performs pattern discrimination by comparing the fish shadow image extracted by the still image side image extracting unit 4 with the discriminating criterion created based on the fish species database 20 stored in the storage unit 8 in advance. The fish type is discriminated and the size is discriminated from the number of pixels of the fish shadow image. Further, the type discriminating means 5 adds and discriminates the discriminated fish shadow and fish species to the fish species database 20 to improve the accuracy of the next discrimination. The type discrimination means 5 of this embodiment uses a support vector machine as a discrimination means.

  The type discrimination means 5 outputs the discrimination result to the display unit 10. The type discriminating means 5 of the present embodiment totals the number and size of fish for each fish type and outputs them to the display unit 10, but is not necessarily limited thereto. By counting the results and displaying them on the display unit 10, the measurement results can be easily understood.

  In addition, the aggregation of fish types may target all fish types or main fish types, or may target one or more specific types. For example, one or more fish species to be surveyed are determined in advance, the fish species to be surveyed are counted for each fish species, and the fish species other than the survey targets are aggregated as other fish species. You may make it do.

  Note that fish shadows for which the fish type could not be determined by collation with the fish species database 20 are processed as unknown fish shadows. In addition, the same fish shadow detected from the moving image is also processed as an unknown fish shadow even when the corresponding different still images have different fish species discrimination results. These fish shadows are subsequently subjected to fish type discrimination by humans, and the fish shadow database 20 is improved by additionally registering them in the fish shadow database 20. In addition, fish shadows that appear and disappear during the still image capture interval even though fish shadows are detected in the video are counted as unknown fish shadow trajectories. .

  The discrimination of the fish type and size, the totalization, and the display of the results are automatically performed by the type discrimination means 5.

  According to the present invention, since the fish shadow is detected based on the moving image, the detection accuracy can be improved, and the image determination is performed based on the still image, so that the determination accuracy can be improved. In other words, since the characteristics of the moving image and the still image are used, it is possible to efficiently and accurately perform the discrimination. In addition, automatic discrimination by image processing can reduce labor by manpower, and it is possible to obtain highly objective results by preventing variations in discrimination results due to the amount of knowledge and proficiency of an operator.

  In addition, since the measurement area in the measurement time zone is unmanned and divers do not affect the behavior of fish, etc., it is possible to measure the normal situation as it is, and only to photograph the fish and capture the biological resources by capture It does not cause a decrease.

  In the present invention, two cameras 1 and 2 are used. Usually, when two cameras are used, there are the following problems, but the present invention solves these problems. That is, when the moving image camera 1 and the still image camera 2 are used, it is necessary to associate the moving image and the still image. There are (1) temporal shift and (2) spatial shift (parallax shift) between moving images and still images shot with different imaging systems, and correction is necessary.

(1) Time lag Since the fish moves, if the time of the moving image and the still image do not match, the position of the photographed fish will be different, and cooperation will be difficult. Note that the built-in clocks of the video camera 1 and the still camera 2 are set within a possible range before shooting, but it is difficult to ensure synchronization accuracy within one second at this point. This error is mainly caused by the still camera mechanism and the accuracy of the time stamp (shooting time information).

  For example, the interval shooting of a still camera set to shoot every 10 seconds is not accurately shot every 10 seconds and 0 milliseconds. First, a slight shift occurs depending on the physical mechanism such as focusing and the time required for processing of an electronic circuit such as image compression / storage. Furthermore, since the time stamp has only information in seconds, the deviation in milliseconds is not recorded. Therefore, an image recorded with a time stamp of 10 seconds, 20 seconds, and 30 seconds is recorded underwater conditions of about 10 seconds, about 20 seconds, and about 30 seconds. In other words, in the linkage between a moving image and a still image, it is necessary to associate them with the assumption of this time lag.

  In contrast, in the present invention, the synchronization mark is used to enable time synchronization between the moving image and the still image (step S42), thereby enabling processing.

(2) Disparity shift Next, even if a moving image and a still image are taken at exactly the same time, a difference occurs in the taken images. This is due to the difference in “parallax” that results from the spatial separation of the video camera and the still camera.

  The parallax shift is a phenomenon in which an image shifts because there is a distance between the two cameras 1 and 2 (between the optical axes of the lenses), such as a human right eye and left eye. This deviation increases as the distance between the two cameras 1 and 2 increases. In addition, the shift is larger near the camera and small in the distance.

  The influence of the parallax deviation will be described based on FIG. There are a video camera and a still camera on the left side of FIG. 7, and the right direction of FIG. 7 is taken. There is a separation between the two cameras. Here, for explanation, the P0 plane on the right side of FIG. 7 is used as the imaging plane of the camera (actually on the back side of the lens). In this example, the three fish at the positions of f1, f2, and f3 are all seen at the position of F0 on the imaging surface when viewed from the video camera. This is because it is on a straight line connecting the video camera and F0. On the other hand, when viewed from a still camera, these fish appear at different positions F1, F2, and F3.

  As described above, the present invention takes a means of estimating the position on the still image from the fish shadow position obtained from the moving image. However, this is a problem that the position information of F1, F2, and F3 is estimated from only the position information of F0, and this is a defect setting problem that lacks information necessary to obtain a solution.

  On the other hand, in the present invention, the coordinate conversion means 14 converts the fish shadow coordinates extracted from the moving image frame to the coordinates on the still image, and the image search means 15 uses the converted coordinates as a reference for the fish shadow in the still image. Since the search is performed, the accuracy in associating the positions of the moving image and the still image is improved (steps S48 and S49), and the extraction accuracy of the image to be discriminated in the still image can be improved.

  The above-described embodiment is an example of a preferred embodiment of the present invention, but is not limited thereto, and various modifications can be made without departing from the scope of the present invention.

  For example, in the above description, the type discriminating means 5 discriminates the fish species using the already created fish species database 20, but the processing may be performed while creating the fish species database 20. .

  The procedure in this case is shown in FIGS. In addition, the same number is attached | subjected to the step which performs the same process as the above-mentioned step, and those description is abbreviate | omitted. As described above, the pre-processing phase S51 is performed to create the fish species database 20 before performing the execution phase S50 for performing fish species discrimination using the fish species database 20. A pre-processing phase S51 is performed on a part of the acquired moving image and still image, and an execution phase S50 is performed on the remaining moving image and still image. In the execution phase S50, as described above, the type discrimination means 5 automatically discriminates the fish species using the fish species database 20, but in the preprocessing phase S51, the fish species are discriminated and inputted by a human, The fish species discrimination and creation of the fish species database 20 are performed simultaneously.

  In the preprocessing phase S51, the fish shadow position is detected by the moving image side image extracting means 3 (step S3) and the fish shadow is extracted by the still image side image extracting means 4 (step S44), and then the fish type input step S52. Is executed.

  In the fish type input step S52, the database creation assisting unit 21 displays the fish shadow extracted by the still image side image extracting unit 4 on the display unit 10, and waits for the input of the fish type (label) from the input unit 9. When a label is input by the operator, the database creation assisting means 21 creates a fish species database 20 by registering the displayed fish shadow and the label in association with each other. Label entry and registration in the database is performed for each fish shadow.

  When the label input and the database registration are performed for all the fish types processed in the preprocessing phase S51, the execution phase S50 is executed, and the fish types are automatically determined for the remaining fish shadows. In the execution phase S50, the fish species are automatically identified using the fish species database 20 created in the preprocessing phase S51.

  The database creation assisting means 21 is realized in the computing unit 7 when the computer 6 executes a control program.

  Note that a labeling support tool may be used for inputting a label for creating the fish shadow database 20. The labeling support tool is software having a user interface for labeling, and displays a frame indicating the search range 22 to be searched by the image search means 15 overlaid on a still image, enlarged display of fish shadows, Movement, shape change, and labeling (input) can be performed. The labeling support tool is installed in the computer 6 and executed.

  In the above description, the size of the fish is determined based on the number of fish shadow pixels in step S45. However, the actual fish size may be estimated instead of the apparent size. For example, the distance between the cameras 1 and 2 and the fish may be measured at the time of photographing, and the actual fish size may be calculated based on the measured distance and the photographed fish shadow size (number of pixels). For example, a commercially available multi-view video camera or a laser range finder can be used as a measuring device for measuring the distance to the fish, but is not limited thereto. Further, in addition to the two cameras 1 and 2, another moving camera may be installed, and the distance may be calculated from the parallax of the same fish shadow acquired by the two moving cameras. Furthermore, the distance may be calculated from the parallax of a still image taken simultaneously using a 3D digital camera (digital camera with two lenses) as a digital camera (still image camera 2) for recording a still image. Since the 3D digital camera has a small parallax, the distance resolution is low. However, since the optical systems of the two lenses are accurately aligned, it is easy to calculate the distance by image processing.

  Further, since the fish moving speed is related to the fish size, it is also possible to estimate the fish size from the average moving speed of the trajectory (the number of moving pixels per unit time).

  Further, in the above description, regarding the association between the moving image and the still image, the image search unit 15 searches the image of the search range 22 around the coordinates on the still image associated by the coordinate conversion unit 14 ( When the coordinates on the moving image have a one-to-one correspondence with the coordinates on the still image, step S49), for example, when the movement of an animal (including a human) or the like moving around the ground is planar. Since the fish shadow exists at the coordinates on the still image associated by the coordinate conversion means 14, the image search in the search range 22 may be omitted.

  In order to evaluate the present invention, an evaluation experiment using moving images and still images taken underwater was performed. The evaluation was 26 minutes (14:41 to 15:17) taken on August 20, 2011 with a video camera and a still camera installed at a depth of 1m on the Majanohama coast in Akajima, Zamami-mura, Tokashiki-gun, Okinawa. (DV format) and 157 still images taken simultaneously (photographed every 10 seconds, 4000 × 3000 pixels) were used. The distance between the video camera and the still camera is approximately 20 cm. The coral reef is developed at this point, and fish are abundant, which is suitable for the performance evaluation of the present invention.

(1. Evaluation of fish shadow detection from video)
The first frame image of the underwater video is shown in FIG. There is a coral in front of the camera installation position, which occupies about the lower third of the screen. The moving image is a DV format, a widened image of 720 pixels wide and 360 pixels long, and an interlace.

  The evaluation target was 157 frame images at the same time as when the still image was recorded. The fish shadows used as the evaluation standard are all 157 frames displayed, frame-by-frame forward and backward, and all fish shadows found visually (fish fish shadows of 50 pixels or less are Not related to size parameter)

  In contrast, as a result of fish shadow detection using the present invention, the number of correct answers: 1050, the number of oversights: 146, the number of errors: 20. Therefore, the precision (Precision) = 1050 / (1050 + 20) = 98.1%, and the recall (Recall) = 1050 / (1050 + 146) = 87.8%.

  The parameter settings were as follows. Sigma band width adjustment coefficient: α = β = 4.0, time width for obtaining average and standard deviation: 300 seconds, maximum fish shadow speed: 20 pixels / frame, fish shadow size upper limit: 5000 pixels, lower limit: 50 pixels The minimum number of tracking times until the fish shadow is determined is 3. In the software in which the present invention is implemented, these parameters can be adjusted while performing image processing in real time. Therefore, suitable parameters were adjusted while viewing the processing status for 300 seconds from the beginning of the moving image.

  The parameter that has a large effect on processing is the sigma band width adjustment coefficient, and if the setting value is reduced, it will cause erroneous measurement due to light fluctuations, such as light flickering on the surface of the coral. . As described above, since the number of overlooked results is somewhat large, the optimal parameter may be a little smaller.

  An oversight occurred when: First, on the surface of coral where the light fluctuates, the sigma band grows, and the appearance of bright fish is sometimes overlooked. On the other hand, black fish could be overlooked in dark places that are always shadows of corals. These were difficult to find visually and were only found by playing the video. Also, when fish shadows overlap or touch on the screen, the fish shadows are considered as one and may be overlooked.

  On the other hand, most of the mistakes were when a large single fish was split into two or more. There was almost no mistake in background brightness changes such as light fluctuations with fish shadows.

(2. Evaluation of fish shadow extraction from still images)
(1) Time adjustment of video and still images The time stamps of video and still images do not match when shot. Therefore, the time of the movie was adjusted to the still image. Specifically, it was performed as follows. First, in the still images taken before the measurement time zone, images such as swimming divers and conspicuous fish shadows that have movement and can be distinguished from change in units of frames (still images with synchronous landmarks) ) Is selected. Assume that the time of the still image is H hour M minute S second. In a moving image, the same frame time as that still image is set to H hour M minute S second 0 millisecond. Each frame time of the moving image is recalculated from the reset time in terms of the number of frames. Since the video used this time is NTSC, it is 29.97 frames per second. FIG. 12 shows a frame image for time adjustment used this time.

(2) Setting of Reference Point for Association In the present invention, some mark is required as three reference points for associating the coordinates of the moving image and the still image. This time, as shown in FIG. 13, the reference point is a fish shadow that is distinguishable from other places that swim up, as shown in FIG. As described above, when moving away from the plane including these three reference points, the result of coordinate conversion and the actual fish shadow position deviate. The reference point selection policy is to set the width as long as possible in the vertical and horizontal directions at a middle distance in the depth direction from the camera. Since the field of view of the camera is narrow in the vicinity of the camera and wide in the distance, the number of fish shadows that appear is large in the distance. On the other hand, as the distance from the camera increases, the fish shadow becomes smaller and unclear and the fish shadow cannot be determined even on a still image. For this reason, it is important to measure in the middle, so we decided to provide a reference point at an intermediate distance. Also, the distance between the reference points is increased in order to reduce the influence of errors in setting the reference points of the moving image and the still image. When the reference points are close to each other, a difference of several pixels appears as a big difference at the end away from the reference point.

(3) Accuracy of matching In order to evaluate fish shadow matching, a correct fish shadow position on a still image is required. A labeling support tool was developed and used to input the correct fish shadow position. The labeling support tool is software having a user interface for labeling, and displays a frame indicating the search range 22 to be searched by the image search means 15 overlaid on a still image, enlarged display of fish shadows, Movement, shape change, and labeling (input) can be performed.

  As the evaluation index, the area capture rate was defined and used (Formula 1).

<Equation 1>
Area capture rate = area where extraction area and fish shadow area overlap ÷ area of fish shadow area

  Here, the extraction area is a fish shadow extracted by the still image side image extraction means 4 (that is, a rectangular image area including a part where the fish is reflected), and the fish shadow area is a part of the fish actually reflected. Is the smallest rectangle that encloses That is, the index is 100% when the area automatically extracted by the still image side image extraction means 4 correctly covers the fish shadow, and 0% when there is no overlap. With respect to all 817 areas corresponding to fish shadows, the average was 47.3% when only coordinate conversion was performed, and the average was 70.8% when fish shadow search was performed.

  The search range coefficients, which are parameters for adjusting the width of the search range of the image search means 15, are a = 5.0 and b = 3.0. The reason why the width is wide (a> b) is that the two cameras 1 and 2 used are juxtaposed on the left and right, and the parallax deviation is large in the horizontal direction.

  The reason why the number of parameters 817 used for the evaluation is smaller than the above-described extracted fish shadow region 1050 (1050 fish) is that there is a fish shadow that appears in the moving image but does not appear in the still image. Such a region is associated with a region having no fish shadow such as the seabed or seawater. The reason for not appearing is that there is a fish outside the screen that cannot be photographed with a still camera, or there is a case where the fish shadow has moved to a position different from that of the moving image due to the time shift.

  Even when the fish shadow search was performed, there were 114 regions with a region capture rate of 0%. These occur mainly at the bottom of the screen. FIG. 14 shows the area capture rate according to the number of pixels in the vertical direction from the upper end of the screen. The vertical axis represents the number of pixels in the vertical direction of a still image reduced according to the size of the moving image, and the vertical width of the still image is 600 pixels (FIG. 15). From this, it can be seen that the capture rate in the bottom area of the screen is as low as 36.8%.

  As a result of analyzing an example in which the association failed, it was found that there are the following three causes. First, the coordinates given as the initial position of the fish shadow search by the coordinate conversion are greatly different due to the effect of the parallax shift, and therefore there is no fish shadow in the search range. The closer to the camera, the greater the parallax shift. Next, if you are in a background that is difficult to distinguish from fish shadows, such as coral, you may have selected coral instead of fish. This is also affected by the difference in color and resolution between moving images and still images. Finally, due to the effect of the time lag between the moving image and the still image, the direction and angle of the fish differ greatly between the moving image and the still image, and the images are not similar. Since both conditions occur frequently at the bottom of the screen, the area capture rate is low at the bottom of the screen.

(3. Evaluation of fish species discrimination)
(1) Creation of an image database In order to evaluate fish species discrimination, it is necessary to create an image database corresponding to the fish shadow region extracted by the still image side image extraction means 4. Therefore, all the 1021 fish shadow regions extracted by the still image side image extraction means 4 are labeled by experts. The labeling was performed using a labeling support tool after correcting the correct fish shadow position so that the area capture rate of Equation 1 was 100%. Note that the extraction region includes not only fish shadows but also fish shadows such as corals, the seabed, and seawater. The reason why it is smaller than the extracted fish shadow area 1050 (1050 fish) is that the area overlapping the same area is excluded. There were four types of labels in Table 1.

  The number of each area | region was set to 332: Other fishes: 42, Unknown fish: 485, Other than fish: 162. In addition, the average size of each area (the number of pixels when the still image is 800 × 600 pixels) is: damselfish: 2292, other fish: 3711, unknown fish: 980, non-fish: 3465, and small fish Shadows were difficult to determine. The number of each area is tabulated in Table 2 for each distance range from the upper end of the screen. From this, it can be seen that there are more unknown fish at the top of the screen. Since the upper part of the screen also captures fish shadows far from the camera, many fish shadows that cannot be identified appear. The reason why the number of areas other than fish is large at the upper end is that the still image shooting time is delayed and the fish swims off the screen.

(2) Fish shadow and other judgment performance As a fish species discrimination function, it is first necessary to distinguish between fish and non-fish. Therefore, as a rudimentary trial of automatic judgment, we tried to automatically distinguish fish shadows including unknown fish from those other than fish. The target area for automatic determination is an area where the fish shadow position is corrected and labeled by an expert. As the feature quantity, we adopt the pattern feature quantity (Yoshito Nakajima: “High-speed intruder detection method using existing surveillance camera”, Research Report of Central Research Institute of Electric Power, R04007, 2005) that showed good performance in human detection did. Specifically, the region image was reduced to 64 pixels in the vertical direction and 64 pixels in the horizontal direction, and a total of 512 LH components and HL components obtained by the two-dimensional Haar wavelet were used (FIG. 16). An SVM was used as the learning device, and an RBF kernel that was considered suitable for natural images was used as the kernel. Table 3 shows the results of cross validation using 3/4 of the data randomly selected and used for learning and the remaining 1/4 for evaluation while changing γ, which is a parameter of the RBF kernel, 50 times. It was shown that fish shadow and other determinations are possible at about 94%.

  Since only the damselfish was photographed at this time, the judgment performance of the fish species was not evaluated.

  The results of the evaluation experiment are summarized below. Regarding the detection of fish shadows from moving images, high detection performance was shown for moving images used for evaluation, with a precision of 98.1% and a recall of 87.8%. In the extraction of fish shadows from still images, the area capture rate was evaluated, and the average area capture rate was 47.3% only by coordinate conversion, but by adding fish shadow search using template matching, The fish shadow capture rate was 70.8% on average, confirming the effect of the present invention. Regarding the discrimination of fish species, since the fish shadow database is not yet established, it was determined as an elementary trial whether or not it was a fish shadow, and it was determined with an accuracy of 93.8%. It was confirmed that it was possible to distinguish fish species automatically.

DESCRIPTION OF SYMBOLS 1 Movie camera 2 Still image camera 3 Movie image side image extraction means 4 Still image side image extraction means 5 Type discrimination means 14 Coordinate conversion means 15 Image search means 20 Fish species database

Claims (12)

  A video camera that captures the same measurement area from the same direction, a still image camera that has a higher image resolution than the video camera, and a video-side image extraction unit that detects the position of the image to be determined from the video captured by the video camera A still image side image extracting unit that extracts the image to be discriminated from the still image at the same time based on the position information of the image detected by the moving image side image extracting unit, and the still image side image extracting unit extracts And a means for discriminating the type of the discrimination target based on the obtained image.   The mobile image determination device according to claim 1, wherein a synchronization mark for time synchronization is imprinted on at least a part of the frames of the moving image and at least one of the still images.   The still image side image extraction means is a coordinate conversion means for converting the coordinates of the image detected by the moving image side image extraction means to coordinates on the still image at the same time, and the conversion coordinates by the coordinate conversion means as a reference. 2. The moving object image discriminating apparatus according to claim 1, further comprising image search means for searching for an image to be discriminated in a still image.   When the main moving direction of the discrimination target is classified into the horizontal direction and the vertical direction, the video camera and the still image camera are arranged vertically and are classified in the vertical direction when classified in the horizontal direction. 2. The moving body image discriminating apparatus according to claim 1, wherein the moving image camera and the still image camera are sometimes arranged on the left and right.   A database in which the classification target image and the type are associated with each other; and the means for determining the classification target type is a support vector machine, wherein pattern discrimination is performed with reference to the database to perform determination. The moving body image discrimination apparatus according to claim 1.   2. The moving object image discriminating apparatus according to claim 1, wherein the means for discriminating the type of discrimination object counts the discrimination object.   The moving image camera and a still image camera having a higher image resolution than the moving image camera capture the same measurement area from the same direction to obtain a moving image and a number of time-series still images corresponding to the moving image. And detecting the position of the image to be determined from the moving image taken in step, and extracting the image of the determination target from the still image at the same time based on the detected position information to determine the type of the determination target. A moving body image discrimination method.   The synchronization mark is imprinted on at least a part of the frames of the moving image and at least one of the still images, and the shooting times of the moving image and the still image are synchronized based on the synchronization mark. Item 8. The moving object image discrimination method according to Item 7.   The extraction of the image to be discriminated from the still image is performed by converting the coordinates of the image to be discriminated detected from the moving image into coordinates on the still image and then using the converted coordinates as a reference in the still image. 8. The moving object image discrimination apparatus according to claim 7, wherein the image discrimination apparatus is performed by searching for an image to be discriminated.   The main moving direction of the discrimination target is classified into a horizontal direction and a vertical direction, and when moving in the horizontal direction, the moving image camera and the still image camera are arranged vertically, and when moving in the vertical direction, the moving image 8. The moving object image discrimination method according to claim 7, wherein the camera and the still image camera are arranged on the left and right.   8. The moving object image discrimination method according to claim 7, wherein pattern discrimination is performed while referring to a database in which the discrimination target image and type are associated with each other to discriminate the discrimination target type.   8. The moving object image discrimination method according to claim 7, wherein the discriminated discrimination objects are counted.

Images